Rufflewind's Scratchpad

Linking Rust crates to native libraries

Sun, 19 Nov 2017 00:00:00 -0800

As far as I know, there’s two ways to link native libraries in a Rust package:

Attributes: #[link(name = "…")]
Build scripts: cargo:rustc-link-lib=dylib=…

You can also pass the linker flags directly to rustc, but that’s a bit too low level for packages.

The attribute approach is the easiest, but it’s also fairly inflexible. Once the attribute is set in the upstream FFI library, dependents downstream have no control over it. This is usually OK for most libraries because the name is fixed and known ahead of time.

However, in cases where the name is not known ahead of time (e.g. platform-dependent libraries), or cases where you want to offer the choice of library to the final application, build scripts offer far more flexibility.

One might be tempted to omit the #[link(…)] attribute in the upstream FFI library, and let the downstream application set the attribute, but this doesn’t work as one might expect. This is due to rust-lang/rust#28605, which causes the linker flags to be ordered like this:

cc downstreamapp -l somenativelib upstreamrustlib

Because the native library appears before the upstream library, symbols in the native library are not available for the upstream library.

Fortunately, in build scripts the custom linking flags are always placed at the end. For convenience, one would often place this in a separate crate that contains nothing but a build script like this:

fn main() {
    println!("cargo:rustc-link-lib=dylib=somenativelib");
}

Cheatsheet for Futures

Mon, 15 May 2017 00:00:00 -0700

A simplified listing of the various Future combinators in futures-rs (expanded version):

// Constructing leaf futures
fn empty ()             -> Future<T, E>
fn ok    (T)            -> Future<T, E>
fn err   (E)            -> Future<T, E>
fn result(Result<T, E>) -> Future<T, E>

// General future constructor
fn poll_fn(FnMut(thread_local!(Task)) -> Poll<T, E>) -> Future<T, E>

// Mapping futures
fn Future::map     (Future<T, E>, FnOnce(T) -> U) -> Future<U, E>
fn Future::map_err (Future<T, E>, FnOnce(E) -> F) -> Future<T, F>
fn Future::from_err(Future<T, Into<E>>)           -> Future<T, E>

// Chaining (sequencing) futures
fn Future::then    (Future<T, E>, FnOnce(Result<T, E>) -> IntoFuture<U, F>) -> Future<U, F>
fn Future::and_then(Future<T, E>, FnOnce(T)            -> IntoFuture<U, E>) -> Future<U, E>
fn Future::or_else (Future<T, E>, FnOnce(E)            -> IntoFuture<T, F>) -> Future<T, F>
fn Future::flatten (Future<Future<T, E>, Into<E>>)                          -> Future<T, E>

// Joining (waiting) futures
fn Future::join (Future<T, E>, IntoFuture<U, E>)                                                       -> Future<(T, U),          E>
fn Future::join3(Future<T, E>, IntoFuture<U, E>, IntoFuture<V, E>)                                     -> Future<(T, U, V),       E>
fn Future::join4(Future<T, E>, IntoFuture<U, E>, IntoFuture<V, E>, IntoFuture<W, E>)                   -> Future<(T, U, V, W),    E>
fn Future::join5(Future<T, E>, IntoFuture<U, E>, IntoFuture<V, E>, IntoFuture<W, E>, IntoFuture<X, E>) -> Future<(T, U, V, W, X), E>
fn join_all     (IntoIterator<IntoFuture<T, E>>)                                                       -> Future<Vec<T>,          E>

// Selecting (racing) futures
fn Future::select (Future<T, E>, IntoFuture<T, E>) -> Future<(T, Future<T, E>), (E, Future<T, E>)>
fn Future::select2(Future<T, E>, IntoFuture<U, F>) -> Future<Either<(T, Future<U, F>), (U, Future<T, E>)>, Either<(E, Future<U, F>), (F, Future<T, E>)>>
fn select_all     (IntoIterator<IntoFuture<T, E>>) -> Future<(T, usize, Vec<Future<T, E>>), (E, usize, Vec<Future<T, E>>)>
fn select_ok      (IntoIterator<IntoFuture<T, E>>) -> Future<(T, Vec<Future<T, E>>), E>

// Utility
fn lazy         (FnOnce() -> IntoFuture<T, E>)             -> Future<T, E>
fn loop_fn      (S, FnMut(S) -> IntoFuture<Loop<T, S>, E>) -> Future<T, E>
fn Future::boxed(Future<T, E>+Send+'static)                -> Future<T, E>+Send+'static

// Miscellaneous
fn Future::into_stream   (Future<T, E>)            -> Stream<T, E>
fn Future::flatten_stream(Future<Stream<T, E>, E>) -> Stream<T, E>
fn Future::fuse          (Future<T, E>)            -> Future<T, E>
fn Future::catch_unwind  (Future<T, E>+UnwindSafe) -> Future<Result<T, E>, Any+Send>
fn Future::shared        (Future<T, E>)            -> Future<SharedItem<T>, SharedError<E>>+Clone
fn Future::wait          (Future<T, E>)            -> Result<T, E>

Using libraries

Sat, 25 Feb 2017 00:00:00 -0800

Note: This article is somewhat biased toward Linux-like environments.

Native vs non-native

Native libraries: these are the ones you get through native compilation. This applies to languages such as C, C++, Fortran. These libraries are interoperable.

This category also applies to modern natively-compiled languages such as Haskell or Rust, but I’m not as familiar with those so I won’t discuss them here.
Non-native libraries: these are made of source code or bytecode that isn’t executed directly on the CPU. Examples include C#, Java, Perl, Python, Ruby, etc. Non-native libraries are usually highly language-specific and not interoperable with each other.

This article is only concerned with native libraries, specifically of the older family (C, C++, Fortran).

Parts of a library

Libraries are generally divided into two components:

The library file, which contains mostly machine code. This is the essential library implementation.

The file extensions are usually:
- .a/.so (ELF OSes like Linux),
- .a/.dylib (OS X),
- .lib/.dll (Windows).
The former of each pair are file extensions for static libraries, whereas the latter of each pair are for shared libraries a.k.a. dynamically-linked libraries (DLLs). The distinction is explained in the next section.
The library interface, which is a protocol that users of the library must abide by in order to use the library correctly. Failing to do so will often lead to crashes (segfaults) and brokenness.

For C and C++, this is usually distributed as a header file (.h, .hpp, etc). There are ways to use a library even without a header file, but it is usually discouraged as it is error-prone. There is one exception though: non-C/C++ programs (e.g. a Rust program) that want to use a C/C++ library often don’t require header files.

For Fortran, the library interface is usually just the source code itself, or interchangeably the .mod files.

Static vs shared libraries

In static libraries, the library’s own code is copied and merged with the user’s code, creating one combined file. This means after compilation, the static library file itself is no longer needed nor used. (Of course, you probably want to keep it in case you want to compile more programs with that same library.)

Note that static libraries are quite dumb: you cannot link more than one copy of the same static library into the same program. You might wonder “Why would anyone do that anyway?” Well, it’s often unintentional: say you use library A and library B, both of which use library C. If this whole thing is linked statically, then both A and B will bring in their own copy of C!
In shared libraries, the library’s own code remains a separate file. This means later when the program runs, it must load the library as a separate step. If the shared library can’t be found the program will fail to launch. If the shared library is upgraded or changed, then the behavior of any program that depends on that library may change (this can be both useful and annoying).

Even though shared libraries are never incorporated into the final program, they are still needed during compilation because the compiler still needs to be aware of the library’s interface. On non-Windows systems, the compiler simply reads the shared library itself (.so or .dylib) during compilation. On Windows, compilation requires a so-called import library, which has the extension .lib (not to be confused with static libraries on Windows, which also have the same extension).

Installing a library

The first step to use a library is to install it, obviously. The process varies a lot. It’s best to read the documentation for that specific library. One of the most crucial things to figure out is where the library is going to be installed.

(Another crucial thing is to find out if the library has optional features, because they might be off by default, or turned off if the configuration step fails to find its dependencies!)

Prefixes

On Unix-like OSes, libraries are usually installed to the /usr/local prefix by default. This prefix means that all of the library’s relevant files will be installed to /usr/local/lib (library files), /usr/local/include (header files), /usr/local/bin (executable programs), etc. To change the prefix, the process varies depending on the build system used by the library:

For Autoconf-like build systems, this can often be changed through the ./configure script. The command would be something like
```
./configure --prefix=/my_custom_prefix
```
For the CMake build system, this can be changed using
```
cmake -DCMAKE_INSTALL_PREFIX=/my_custom_prefix .
```

It is considered bad practice to install libraries to the /usr prefix as that lies within the territory of the system package manager.

Including a library

To use the library in your C or C++ code, you’d likely have added

#include <some_library_header.h>

to various places in your source code to inform the compiler about the library interface (by forward declaring various types, functions, and variables). These header files are found under the include directory of wherever the library was installed, usually.

If the library header files exist in a standardized location such as /usr/include, then the compiler will find them just fine. But if you installed them in an unusual location like /my_custom_prefix/include, then you’ll have to give a hint to your compiler so it knows where to look. This can be accomplished via the -I flag:

cc -c -I/my_custom_prefix/include my_program.c

(Replace cc with whatever compiler you use.) An alternative approach is to use the C_INCLUDE_PATH (C) and/or CPLUS_INCLUDE_PATH (C++) environment variables:

export C_INCLUDE_PATH=/my_custom_prefix/include
cc -c my_program.c

You can have multiple paths in the variable using colon (:), analogous to PATH. These environment variables may not be recognized by all compilers, however. I know it works for clang and gcc at least.

Linking with a library

When all the source files have been compiled with -c, you now need to link everything together, including any libraries you use. This is done using the -l flag:

cc my_program.o my_blah.o my_foo.o -lalpha -lbeta

The word that follows -l is the name of the library. If the library file is libalpha.so, then its name is just alpha.

If alpha and/or beta are at an unconventional location /my_custom_prefix/lib, then you have to pass in a flag -L to tell the compiler where they could be found:

cc -L/my_custom_prefix/lib my_program.o my_blah.o my_foo.o -lalpha -lbeta

You can also do this using another colon-separated environment variable LIBRARY_PATH:

export LIBRARY_PATH=/my_custom_prefix/lib
cc my_program.o my_blah.o my_foo.o -lalpha -lbeta

Again, I think only some compilers recognize this variable.

Loading a shared library

The last step is to make sure the program can actually load the shared library. Usually this is automatic, but if the library is in an unconventional place like /my_custom_prefix/lib, then you gotta give it another hint using yet-another colon-separated environment variable LD_LIBRARY_PATH (or DYLIB_LIBRARY_PATH if you’re using OS X):

export LD_LIBRARY_PATH=/my_custom_prefix/lib
./a.out

Alternatively, you can bake the library path directly into the program so you don’t need an environment variable to run the program. This is the -rpath flag:

cc -Wl,-rpath=/my_custom_prefix/lib my_program.o my_blah.o my_foo.o -lalpha -lbeta

You can specify a path relative to the location of the program itself using the ${ORIGIN} placeholder. Note that this is not a shell variable, so be sure to single-quote it in the shell:

cc -Wl,-rpath='${ORIGIN}/lib' …

Two common problems with Git submodules

Tue, 21 Feb 2017 00:00:00 -0800

Git submodules are useful, but their UX is a bit intrusive for users who aren’t even interacting with the submodules. Here’s a list of the two common ones I often run into. (Let me know if there’s anything else that folks often run into!)

I cloned a repo containing submodules, but there’s nothing in them!

This happens if you clone a submodule without the --recursive flag. The solution is to run

git submodule update --init --recursive

which will initialize everything.

I figure the main reason this is not the default is that cloning submodules can waste a lot of time, disk space, and/or bandwidth if you don’t actually need the submodules.

Why are submodules showing up in `git status` even though I never touched them?

This can happen after git pull, git checkout, or git reset, which change the working tree but do not update the submodules, causing them to lag behind. In git status, the problem manifests as:

modified:   mysubmodule (new commits)

If you are sure you didn’t change any file within the submodules, you can update them using

git submodule update --recursive

I’m not sure why this isn’t the default. Perhaps it’s to avoid accidentally losing changes within the submodules?

Graphical depiction of ownership and borrowing in Rust

Wed, 15 Feb 2017 00:00:00 -0800

Below is a graphical depiction of moving, copying, and borrowing in the Rust language. Most of these concepts are fairly specific to Rust and are therefore a common stumbling block for many learners.

To avoid clutter in the graphics, I have tried to keep the text to a minimum. It isn’t meant to be a replacement for the various tutorials out there but more of a different perspective for programmers who prefer to grok concepts visually. If you are learning Rust and find these graphics helpful, I would recommend annotating your own code with such diagrams to help solidify the concepts :)

You can zoom in by clicking the image. You can also get it as an SVG or PDF.

The upper two figures depict the two main kinds of semantics for data that you own: either move semantics or copy semantics.

The picture on move semantics (⤳) looks almost too simple. There is no deception here: move semantics are strange only because most languages allow variables to be used as many times as the programmers please. This stands in contrast to much of the real world: I can’t just give someone my pen and still use it for writing! In Rust, any variable whose type does not implement the Copy trait has move semantics and would behave as shown.
Copy semantics (⎘) are reserved for types that do implement the Copy trait. In this case, every use of the object would result in a copy, as shown by the bifurcation.

The central two figures depict the two ways in which you can borrow an object you own, and what each one offers.

For mutable borrowing, I used a lock symbol (🔒) to signify that the original object is effectively locked for the duration of the borrow, rendering it unusable.
In contrast, for non-mutable borrowing I used a snowflake symbol (❄) to indicate that the original object is only frozen: you can still take more non-mutable references, but you cannot move or take mutable references of it.

In both figures, ’ρ is a name I have chosen for the lifetime of the references. I used a Greek letter on purpose because there is no syntax for concrete lifetimes in Rust, currently.

The last two figures summarize the key differences and similarities between the two kinds of references, both pictorally and in text form. The “exteriorly” qualifier is important, since you can still have interior mutability through Cell-like things.

Reverse-mode automatic differentiation: a tutorial

Fri, 30 Dec 2016 00:00:00 -0800

In this post, I’ll walk through the mathematical formalism of reverse-mode automatic differentiation (AD) and try to explain some simple implementation strategies for reverse-mode AD. Demo programs in Python and Rust are included.

A simple example

Suppose we want to calculate the expression:

\[z = x \cdot y + \sin(x)\]

To do this using a program, we’d just translate it directly to code:

z = x * y + sin(x)

What if we’re also interested in the derivatives of $z$? The “obvious” approach is to just find the expression by hand (or using a computer algebra system) and then punch it into the computer as we did before. But that assumes we have an explicit form for $z$. What if all we had was a program?

The important realization that leads to automatic differentiation is the fact that even biggest, most complicated program must be built from a small set of primitive operations such as addition, multiplication, or trigonometric functions. The chain rule allows us to take full advantage of this property.

Forward-mode automatic differentiation

First, we need to think about how a computer would evaluate $z$ via a sequence of primitive operations (multiplication, sine, and addition):

# Program A
x = ?
y = ?
a = x * y
b = sin(x)
z = a + b

The question marks indicate that x and y are to be supplied by the user.

I was careful to avoid reassigning to the same variable: this way we can treat each assignment as a plain old math equation:

\[\begin{align} x &= {?} \\ y &= {?} \\ a &= x \cdot y \tag{A} \\ b &= \sin(x) \\ z &= a + b \end{align}\]

Let’s try to differentiate each equation with respect to some yet-to-be-given variable $t$:

\[\begin{align} \frac{\partial x}{\partial t} &= {?} \tag{F1} \\ \frac{\partial y}{\partial t} &= {?} \\ \frac{\partial a}{\partial t} &= y \cdot \frac{\partial x}{\partial t} + x \cdot \frac{\partial y}{\partial t} \\ \frac{\partial b}{\partial t} &= \cos(x) \cdot \frac{\partial x}{\partial t} \\ \frac{\partial z}{\partial t} &= \frac{\partial a}{\partial t} + \frac{\partial b}{\partial t} \end{align}\]

To obtain this, I have made liberal use of the chain rule:

\[\begin{align} \frac{\partial w}{\partial t} &= \sum_i \left(\frac{\partial w}{\partial u_i} \cdot \frac{\partial u_i}{\partial t}\right) \tag{C1} \\ &= \frac{\partial w}{\partial u_1} \cdot \frac{\partial u_1}{\partial t} + \frac{\partial w}{\partial u_2} \cdot \frac{\partial u_2}{\partial t} + \cdots \end{align}\]

where $w$ denotes some output variable and $u_i$ denotes each of the input variables that $w$ depends on.

If we substitute $t = x$ into equations (F1), we’d have an algorithm for calculating $\partial z / \partial x$. Alternatively, to get $\partial z / \partial y$, we could just plug in $t = y$ instead.

Now, let’s translate the equations (F1) back into an ordinary program involving the differential variables {dx, dy, …}, which stand for $\{\partial x / \partial t, \partial y / \partial t, \ldots\}$ respectively:

# Program B
dx = ?
dy = ?
da = y * dx + x * dy
db = cos(x) * dx
dz = da + db

If we substitute $t = x$ into the mathematical equations, what happens to this program? The effect is remarkably simple: we just need to initialize dx = 1 and dy = 0 as the seed values for the algorithm. Hence, by choosing the seeds dx = 1 and dy = 0, the variable dz will contain the value of the derivative $\partial z / \partial x$ upon completion of the program. Similarly, if we want $\partial z / \partial y$, we would use the seed dx = 0 and dy = 1 and the variable dz would contain the value of $\partial z / \partial y$.

So far we have shown how the derivative can be calculated for a specific function like our example. To make the process fully automatic, we prescribe a set of rules for translating a program that evaluates an expression (like Program A) into a program that evaluates its derivatives (like Program B). We have already discovered 3 of these rules, in fact:

c = a + b     =>    dc = da + db
c = a * b     =>    dc = b * da + a * db
c = sin(a)    =>    dc = cos(a) * da

This can be extended further for subtraction, division, powers, other trigonometric functions, etc using multivariable calculus:

c = a - b     =>    dc = da - db
c = a / b     =>    dc = da / b - a * db / b ** 2
c = a ** b    =>    dc = b * a ** (b - 1) * da + log(a) * a ** b * db
c = cos(a)    =>    dc = -sin(a) * da
c = tan(a)    =>    dc = da / cos(a) ** 2

To translate using the rules, we simply replace each primitive operation in the original program by its differential analog. The order of the program remains unchanged: if a statement K is evaluated before another statement L, then the differential analog of statement K is still evaluated before the differential analog of statement L. This is forward-mode automatic differentiation.

A careful inspection of Program A and Program B reveals that it is actually possible to interleave the differential calculations with the original calculations:

x  = ?
dx = ?

y  = ?
dy = ?

a  = x * y
da = y * dx + x * dy

b  = sin(x)
db = cos(x) * dx

z  = a + b
dz = da + db

This demonstrates the two main advantages of forward-mode AD:

The differential variables usually depend on the intermediate variables, so if we do them together there’s no need to hold on to the the intermediate variables until later, saving memory.
This enables an implementation using dual numbers. In languages with operator overloading, this translates to a very simple, direct implementation of forward-mode AD.

For an example in Rust, see the rust-ad library.

Reverse-mode automatic differentiation

The implementation simplicity of forward-mode AD comes with a big disadvantage, which becomes evident when we want to calculate both $\partial z/\partial x$ and $\partial z/\partial y$. In forward-mode AD, doing so requires seeding with dx = 1 and dy = 0, running the program, then seeding with dx = 0 and dy = 1 and running the program again. In effect, the cost of the method scales linearly as O(n) where n is the number of input variables. This would be very costly if we wanted to calculate the gradient of a large complicated function of many variables, which happens surprisingly often in practice.

Let’s take a second look at the chain rule (C1) we used to derive forward-mode AD:

To calculate the gradient using forward-mode AD, we had to perform two substitutions: one with $t = x$ and another with $t = y$. This meant we had to run the entire program twice.

However, the chain rule is symmetric: it doesn’t care what’s in the “numerator” or the “denominator”. So let’s rewrite the chain rule but turn the derivatives upside down:

\[\begin{align} \frac{\partial s}{\partial u} &= \sum_i \left(\frac{\partial w_i}{\partial u} \cdot \frac{\partial s}{\partial w_i}\right) \tag{C2} \\ &= \frac{\partial w_1}{\partial u} \cdot \frac{\partial s}{\partial w_1} + \frac{\partial w_2}{\partial u} \cdot \frac{\partial s}{\partial w_2} + \cdots \end{align}\]

In doing so, we have inverted the input-output roles of the variables. The same naming convention is used here: $u$ for some input variable and $w_i$ for each of the output variables that depend on $u$. The yet-to-given variable is now called $s$ to highlight the change in position.

In this form, the chain rule could be applied repeatedly to every input variable $u$, akin to how in forward-mode AD we applied the chain rule repeatedly to every output variable $w$ to get equation (F1). Therefore, given some $t$, we expect a program that uses chain rule (C2) to be able to compute both $\partial s / \partial x$ and $\partial s / \partial y$ in one go!

So far, this is just a hunch. Let’s try it on the example problem (A).

\[\begin{align} \frac{\partial s}{\partial z} &= {?} \tag{R1} \\ \frac{\partial s}{\partial b} &= \frac{\partial s}{\partial z} \\ \frac{\partial s}{\partial a} &= \frac{\partial s}{\partial z} \\ \frac{\partial s}{\partial y} &= x \cdot \frac{\partial s}{\partial a} \\ \frac{\partial s}{\partial x} &= y \cdot \frac{\partial s}{\partial a} + \cos(x) \cdot \frac{\partial s}{\partial b} \end{align}\]

If you haven’t done this before, I suggest taking the time to actually derive these equations using (C2). It can be quite mind-bending because everything seems “backwards”: instead of asking what input variables a given output variable depends on, we have to ask what output variables a given input variable can affect. The easiest way to see this visually is by drawing a dependency graph of the expression:

Graph of the expression

The graph shows that

the variable a directly depends on x and y,
the variable b directly depends on x, and
the variable z directly depends on a and b.

Or, equivalently:

the variable b can directly affect z,
the variable a can directly affect z,
the variable y can directly affect a, and
the variable x can directly affect a and b.

Let’s now translate the equations (R1) into code. As before, we replace the derivatives $\{\partial s / \partial z, \partial s / \partial b, \ldots\}$ by variables {gz, gb, …}, which we call the adjoint variables. This results in:

gz = ?
gb = gz
ga = gz
gy = x * ga
gx = y * ga + cos(x) * gb

Going back to the equations (R1), we see that if we substitute $s = z$, we would obtain the gradient in the last two equations. In the program, this is equivalent to setting gz = 1 since gz is just $\partial s / \partial z$. We no longer need to run the program twice! This is reverse-mode automatic differentiation.

There is a trade-off, of course. If we want to calculate the derivative of a different output variable, then we would have to re-run the program again with different seeds, so the cost of reverse-mode AD is O(m) where m is the number of output variables. If we had a different example such as:

\[\begin{cases} z = 2 x + \sin(x) \\ v = 4 x + \cos(x) \end{cases}\]

in reverse-mode AD we would have to run the program with gz = 1 and gv = 0 (i.e. $s = z$) to get $\partial z / \partial x$, and then rerun the program with gz = 0 and gv = 1 (i.e. $s = v$) to get $\partial v / \partial x$. In contrast, in forward-mode AD, we’d just set dx = 1 and get both $\partial z / \partial x$ and $\partial v / \partial x$ in one run.

There is a more subtle problem with reverse-mode AD, however: we can’t just interleave the derivative calculations with the evaluations of the original expression anymore, since all the derivative calculations appear to be going in reverse to the original program. Moreover, it’s not obvious how one would even arrive at this point in using a simple rule-based algorithm – is operator overloading even a valid strategy here? How do we put the “automatic” back into reverse-mode AD?

A simple implementation in Python

One way is to parse the original program and then generate an adjoint program that calculates the derivatives. This is usually quite complicated to implement, and its difficulty varies significantly depending on the complexity of the host language. Nonetheless, this may be worthwhile if efficient is critical, as there are more opportunities to perform optimizations in this static approach.

A simpler way is to do this dynamically: construct a full graph that represents our original expression as as the program runs. The goal is to get something akin to the dependency graph we drew earlier:

Graph of the expression

The “roots” of the graph are the independent variables x and y, which could also be thought of as nullary operations. Constructing these nodes is a simple matter of creating an object on the heap:

class Var:
    def __init__(self, value):
        self.value = value
        self.children = []
        …
    …

# define the Vars for the example problem
# initialize x = 0.5 and y = 4.2
x = Var(0.5)
y = Var(4.2)

What does each Var node store? Each node can have several children, which are the other nodes that directly depend on that node. In the example, x has both a and b as its children. Cycles are not allowed in this graph.

By default, a node is created without any children. However, whenever a new expression $u$ is built out of existing nodes $w_i$, the new expression $u$ registers itself as a child of each of its dependencies $w_i$. During the child registration, it will also save its contributing weight

\[\frac{\partial w_i}{\partial u}\]

which will be used later to compute the gradients. As an example, here is how we would do this for multiplication:

class Var:
    …
    def __mul__(self, other):
        z = Var(self.value * other.value)
        self.children.append((other.value, z)) # weight = ∂z/∂self = other.value
        other.children.append((self.value, z)) # weight = ∂z/∂other = self.value
        return z
    …

…
# “a” is a new Var that is a child of both x and y
a = x * y

As you can see, this method, like most dynamic approaches for reverse-mode AD, requires doing a bunch of mutation under the hood.

Finally, to get the derivatives, we need to propagate the derivatives. This can be done using recursion, starting at the roots x and y. To avoid unnecessarily traversing the tree multiple times, we cache the value of in an attribute called grad_value.

class Var:
    def __init__(self):
        …
        # initialize to None, which means it’s not yet evaluated
        self.grad_value = None

    def grad(self):
        # recurse only if the value is not yet cached
        if self.grad_value is None:
            # calculate derivative using chain rule
            self.grad_value = sum(weight * var.grad()
                                  for weight, var in self.children)
        return self.grad_value
    …

…
a.grad_value = 1.0
print("∂a/∂x = {}".format(x.grad())) # ∂a/∂x = 4.2

Here is the complete demonstration of this approach in Python.

Note that because we are mutating the grad_value attribute of the nodes, we can’t reuse the tree to calculate the derivative of a different output variable without traversing the entire tree and resetting every grad_value attribute to None.

A tape-based implementation in Rust

The approach described is not very efficient: a complicated expression can contain lots of primitive operations, leading to lots of nodes being allocated on the heap.

A more space-efficient way to do this is to create nodes by appending them to an existing, growable array. Then, we could just refer to each node by its index in this growable array. Note that we do not use pointers here! If the vector’s capacity changes, pointers to its elements would become invalid.

Using a vector to store nodes does a great job at reducing the number of allocations, but, like any arena allocation method, we won’t be able to deallocate portions of the graph. It’s all or nothing.

Also, we need to somehow fit each node into a fixed amount of space. But then how would we store its list of children?

Turns out, we don’t actually need to store the children. Instead, each node could just store indices to their parent nodes. Conceptually, it would look like this for our example problem:

Concrete representation of the graph

Note the similarity with the graph earlier.

In Rust, we can describe each node using a structure containing two weights and two parent indices:

struct Node {
    weights: [f64; 2],
    deps: [usize; 2], // parent (“dependency”) indices
}

You might wonder why we picked two. This is because we are assuming all primitive operations are binary. For example, the node for the variable a = x * y would look like:

Node {
    weights: [
        y.value, // ∂a/∂x
        x.value, // ∂a/∂y
    ],
    deps: [x.index, y.index],
}

But there’s unary and nullary operations too − how will we deal with those? Quite easy actually, we just set the weights to zero. For example, the node for the variable b = sin(x) would look like:

Node {
    weights: [
        x.value.cos(), // ∂b/∂x
        0.0,
    ]
    deps: [x.index, /* whatever */],
}

As a convention, we will put the index of the node itself into /* whatever */. It really doesn’t matter what we put in there as long as the index is not out of bounds.

The nodes themselves are stored in a common array (Vec<Node>) that is shared by the entire expression graph, which also acts as the allocation arena. In AD literature, this shared array is often called a tape (or Wengert list). The tape can be thought of as a record of all the operations performed during the evaluation of the expression, which in turn contains all the information required to compute its gradient when read in reverse.

In the Python implementation, the nodes were identified with expressions: nodes can be directly combined via arithmetic operations to form new nodes. In Rust, we treat the nodes and expressions as separate entities. Nodes exist solely on the tape, while expressions are just thin wrappers over node indices. Here is what the expression type looks like:

#[derive(Clone, Copy)]
pub struct Var<'t> {
    tape: &'t Tape,
    index: usize,
    value: f64,
}

The expression type contains a pointer to the tape, an index to the node, and an associated value. Note that the expression satisfies Copy, which allows it to be duplicated freely without regard. This is necessary to maintain the illusion that the expression acts like an ordinary floating-point number.

Also, note that the tape is an immutable pointer. We need to modify the tape as we build the expression, but we are going to have lots of expressions holding a pointer to the same tape. This won’t work with a mutable pointer, since they are exclusive, so we must “cheat” Rust’s read-write-lock system using a RefCell:

pub struct Tape { nodes: RefCell<Vec<Node>> }

The bulk of the implementation work lies in coding up the primitive operations. Here’s what the unary sin function looks like:

impl<'t> Var<'t> {
    pub fn sin(self) -> Self {
        Var {
            tape: self.tape,
            value: self.value.sin(),
            index: self.tape.push1(
                self.index, self.value.cos(),
            ),
        }
    }
}

Any unary function can be implemented just like this. Here, push1 is a helper function that constructs the node, pushes it onto the tape, and then returns the index of this new node:

impl Tape {
    fn push1(&self, dep0: usize, weight0: f64) -> usize {
        let mut nodes = self.nodes.borrow_mut();
        let len = nodes.len();
        nodes.push(Node {
            weights: [weight0, 0.0],
            deps: [dep0, len],
        });
        len
    }
}

Finally, when it’s time to do the derivative calculation, we need to traverse the entire tape in reverse and accumulate the derivatives using the chain rule. This is done by the grad function associated with the Var object:

impl<'t> Var<'t> {
    pub fn grad(&self) -> Grad {
        let len = self.tape.len();
        let nodes = self.tape.nodes.borrow();

        // allocate the array of derivatives (specifically: adjoints)
        let mut derivs = vec![0.0; len];

        // seed
        derivs[self.index] = 1.0;

        // traverse the tape in reverse
        for i in (0 .. len).rev() {
            let node = nodes[i];
            let deriv = derivs[i];

            // update the adjoints for its parent nodes
            for j in 0 .. 2 {
                derivs[node.deps[j]] += node.weights[j] * deriv;
            }
        }

        Grad { derivs: derivs }
    }
}

The crucial part lies in the loop. Here, we do not sum over all the derivatives at the same time like in chain rule (C2) or like in the Python program. Rather, we break chain rule up into a sequence of addition-assignments:

\[\frac{\partial s}{\partial u} \leftarrow \frac{\partial s}{\partial u} + \frac{\partial w_i}{\partial u} \frac{\partial s}{\partial w_i}\]

The reason we’re doing this is because we don’t keep track of children anymore. So rather than accumulating all the derivatives contributed by each child all at once, we let each node make its contributions to their parents at their own pace.

Another major difference from the Python program is that the derivatives are now stored on a separate array derivs, which is then disguised as a Grad object:

pub struct Grad { derivs: Vec<f64> }

This means that, unlike the Python program, where all the derivatives are stored in the grad_value attribute of each node, we have decoupled the tape from the storage of the derivatives, allowing the tape / expression graph to be re-used for multiple reverse-mode AD calculations (in case we have multiple output variables).

The derivs array contains all the adjoints / derivatives at the same index as its associated node. Hence, to get the adjoint of a variable whose node is located at index 3, we just need to grab the element the derivs array at index 3. This is implemented by the wrt (“with respect to”) function of the Grad object:

impl Grad {
    pub fn wrt<'t>(&self, var: Var<'t>) -> f64 {
        self.derivs[var.index]
    }
}

Here’s the full demonstration of reverse-mode AD in Rust. To use this rudimentary AD library, you would write:

let t = Tape::new();

let x = t.var(0.5);
let y = t.var(4.2);

let z = x * y + x.sin();

let grad = z.grad();

println!("z = {}", z.value);         // z = 2.579425538604203
println!("∂z/∂x = {}", grad.wrt(x)); // ∂z/∂x = 5.077582561890373
println!("∂z/∂y = {}", grad.wrt(y)); // ∂z/∂y = 0.5

Total differentials and differential operators

Notice that if we throw away the $\partial t$ from the denominators of equation (F1), we end up with a set of equations relating the total differentials of each variable:

\[\begin{align} \mathrm{d} x &= {?} \\ \mathrm{d} y &= {?} \\ \mathrm{d} a &= y \cdot \mathrm{d} x + x \cdot \mathrm{d} y \\ \mathrm{d} b &= \cos(x) \cdot \mathrm{d} x \\ \mathrm{d} z &= \mathrm{d} a + \mathrm{d} b \end{align}\]

This is why variables such as dx are called “differentials”. We can also write chain rule (C1) in a similar form:

\[\mathrm{d} w = \sum_i \left(\frac{\partial w}{\partial u_i} \cdot \mathrm{d} u_i\right)\]

Similarly, we can throw away the $s$ in equation (R1), which then becomes an equation of differential operators:

\[\begin{align} \frac{\partial}{\partial z} &= {?} \\ \frac{\partial}{\partial b} &= \frac{\partial}{\partial z} \\ \frac{\partial}{\partial a} &= \frac{\partial}{\partial z} \\ \frac{\partial}{\partial y} &= x \cdot \frac{\partial}{\partial a} \\ \frac{\partial}{\partial x} &= y \cdot \frac{\partial}{\partial a} + \cos(x) \cdot \frac{\partial}{\partial b} \end{align}\]

We can do the same for (C2):

\[\frac{\partial}{\partial u} = \sum_i \left(\frac{\partial w_i}{\partial u} \cdot \frac{\partial}{\partial w_i}\right) \tag{C4}\]

Saving memory via a CTZ-based strategy

OK, this section is not really part of the tutorial, but more of a discussion regarding a particular optimization strategy that I felt was interesting enough to deserve some elaboration (it was briefly explained on in a paper by Griewank).

So far, we have resigned ourselves to the fact that reverse-mode AD requires storage proportional to the number of intermediate variables.

However, this is not entirely true. If we’re willing to repeat some intermediate calculations, we can make do with quite a bit less storage.

Suppose we have an expression graph that is more or less a straight line from input to output, with N intermediate variables lying in between. So this is not so much an expression graph anymore, but a chain. In the naive solution, we would require O(N) storage space for this very long expression chain.

Now, instead of caching all the intermediate variables, we construct a hierarchy of caches and maintain this hierachy throughout the reverse sweep:

cache_0 stores the initial value
cache_1 stores the result halfway down the chain
cache_2 stores the result 3/4 of the way down the chain
cache_3 stores the result 7/8 of the way down the chain
cache_4 stores the result 15/16 of the way down the chain
…

Notice that the storage requirement is reduced to O(log(N)) because we never have more than log2(N) + 1 values cached.

During the forward sweep, maintaining such a hierarchy would require evicting older cache entries at an index determined by a formula that involves the count-trailing-zeros function.

The easiest way to understand the CTZ-based strategy is to look at an example. Let’s say we have a chain of 16 operations, where 0 is the initial input and f is the final output:

 0 1 2 3 4 5 6 7 8 9 a b c d e f

Suppose we have already finished the forward sweep from 0 to f. In doing so, we have cached 0, 8, c, e, and f:

 0 1 2 3 4 5 6 7 8 9 a b c d e f
                                ^
 X---------------X-------X---X-X

The X symbol indicates that the result is cached, while ^ indicates the status of our reverse sweep. Now let’s start moving backward. Both e and f are available so we can move past e without issue:

 0 1 2 3 4 5 6 7 8 9 a b c d e f
                            ^
 X---------------X-------X---X-X

Now we hit the first problem: we are missing d. So we recompute d from c:

 0 1 2 3 4 5 6 7 8 9 a b c d e f
                            ^
 X---------------X-------X---X-X
                         |
                         +-X

We then march on past c.

 0 1 2 3 4 5 6 7 8 9 a b c d e f
                        ^
 X---------------X-------X---X-X
                         |
                         +-X

Now we’re missing b. So we recompute starting at 8, but in doing so we also cache a:

 0 1 2 3 4 5 6 7 8 9 a b c d e f
                        ^
 X---------------X-------X---X-X
                 |       |
                 +---X-X +-X

We continue on past a:

 0 1 2 3 4 5 6 7 8 9 a b c d e f
                    ^
 X---------------X-------X---X-X
                 |       |
                 +---X-X +-X

Now 9 is missing, so recompute it from 8:

 0 1 2 3 4 5 6 7 8 9 a b c d e f
                    ^
 X---------------X-------X---X-X
                 |       |
                 +---X-X +-X
                 |
                 +-X

Then we move past 8:

 0 1 2 3 4 5 6 7 8 9 a b c d e f
                ^
 X---------------X-------X---X-X
                 |       |
                 +---X-X +-X
                 |
                 +-X

To get 7, we recompute starting from 0, but in doing so we also keep 4 and 6:

 0 1 2 3 4 5 6 7 8 9 a b c d e f
                ^
 X---------------X-------X---X-X
 |               |       |
 +-------X---X-X +---X-X +-X
                 |
                 +-X

By now you can probably see the pattern. Here are the next couple steps:

 0 1 2 3 4 5 6 7 8 9 a b c d e f
            ^
 X---------------X-------X---X-X
 |               |       |
 +-------X---X-X +---X-X +-X
         |       |
         +-X     +-X

 0 1 2 3 4 5 6 7 8 9 a b c d e f
        ^
 X---------------X-------X---X-X
 |               |       |
 +-------X---X-X +---X-X +-X
         |       |
         +-X     +-X

 0 1 2 3 4 5 6 7 8 9 a b c d e f
        ^
 X---------------X-------X---X-X
 |               |       |
 +-------X---X-X +---X-X +-X
 |       |       |
 +---X-X +-X     +-X

 0 1 2 3 4 5 6 7 8 9 a b c d e f
    ^
 X---------------X-------X---X-X
 |               |       |
 +-------X---X-X +---X-X +-X
 |       |       |
 +---X-X +-X     +-X

 0 1 2 3 4 5 6 7 8 9 a b c d e f
    ^
 X---------------X-------X---X-X
 |               |       |
 +-------X---X-X +---X-X +-X
 |       |       |
 +---X-X +-X     +-X
 |
 +-X

 0 1 2 3 4 5 6 7 8 9 a b c d e f
^
 X---------------X-------X---X-X
 |               |       |
 +-------X---X-X +---X-X +-X
 |       |       |
 +---X-X +-X     +-X
 |
 +-X

From here it’s fairly evident that the number of times the calculations get repeated is bounded by O(log(N)), since the diagrams above are just flattened binary trees and their height is bounded logarithmically.

Here is a demonstration of the CTZ-based chaining strategy.

As Griewank noted, this strategy is not the most optimal one, but it does have the advantage of being quite simple to implement, especially when the number of calculation steps is not known a priori. There are other strategies that you might find interesting in his paper.

For a more advanced review of automatic differentiation, see Bartholomew-Biggs et al “Automatic differentiation of algorithms”

Making Jekyll play nice with MathJax: switching from Redcarpet to Pandoc

Wed, 14 Dec 2016 00:00:00 -0800

In the process of writing a math-heavy blog post, I ran into several problems with the existing Jekyll configuration. I had set up Jekyll to use Redcarpet as the Markdown renderer, but it simply does not play well with MathJax: it will screw up \ and & inside the MathJax code.

It is said that Kramdown does better with MathJax, but it doesn’t support syntax highlighting on fenced code blocks and its MathJax syntax is very non-standard: $$ for inline math is just weird.

As a last resort, I decided to integrate Jekyll with Pandoc, which is arguably the Swiss army knife of markup formats. It is unopinionated and has a lot of flexibility in what it can do.

Fortunately, there is already a jekyll-pandoc plugin to do this. It was as simple as running gem install jekyll-pandoc and then tweaking the configuration file:

markdown: Pandoc
gems:
- …
- jekyll-pandoc
pandoc:
  extensions:
  - mathjax

I had wanted to integrate with Pygments as well, but Pandoc’s highlight-kate isn’t too bad and I didn’t feel like adding another complication to the process, so I will just stick with it unless I run into something I don’t like.

To make the colors show up, I just needed to write some CSS akin to the default themes (see Kate docs for details on the categories):

code.sourceCode span.co { color: …; } /* comment */
code.sourceCode span.al { color: …; } /* alert (warning) */
code.sourceCode span.er { color: …; } /* error (wrong_syntax) */
code.sourceCode span.bn { color: …; } /* base_n (nondecimal) */
code.sourceCode span.dv { color: …; } /* decimal_value */
code.sourceCode span.fl { color: …; } /* float */
code.sourceCode span.ch { color: …; } /* character */
code.sourceCode span.st { color: …; } /* string */
code.sourceCode span.dt { color: …; } /* data_type */
code.sourceCode span.fu { color: …; } /* function */
code.sourceCode span.im { color: …; } /* import */
code.sourceCode span.cf { color: …; } /* control_flow */
code.sourceCode span.kw { color: …; } /* keyword */
code.sourceCode span.pp { color: …; } /* preprocessor */
code.sourceCode span.re { color: …; } /* region_marker */
code.sourceCode span.ot { color: …; } /* other */

Hopefully none of the older posts broke. If so, let me know!

Computational interpretation of double negation elimination

Sun, 11 Dec 2016 00:00:00 -0800

Today’s shower thought is: Is there a way to interpret double negation elimination as a program?

((a -> Void) -> Void) -> a

(Here, Void denotes the empty type.)

At first glance, it seems preposterous: how does one expect to produce a return value of a completely arbitrary type a merely from a function of type (a -> Void) -> Void, which doesn’t even return a value of type a?

A hint comes from Peirce’s law, also known as call-with-current-continuation, which has a similar form:

callCC :: ((a -> Cont r) -> Cont a) -> Cont a

(We use Cont to allow the code to be tested in a Haskell program.)

Peirce’s law gives the program full control over its own state (i.e. stack frame). The application of callCC to a callee g :: (a -> Cont r) -> Cont a bestows upon it the power of a nonlocal jump:

example :: Cont String
example =
  callCC $ \ yield -> do
    yield "escaped"
    pure "stayed"

This would return "escaped" rather than "stayed". It’s as if the inner function performed a goto/return/break/longjmp to the outer scope, passing the result along the way.

Turns out doubleNegationElimination can be expressed in terms of callCC:

doubleNegationElimination :: ((a -> Cont Void) -> Cont Void) -> Cont a
doubleNegationElimination doubleNegation =
  callCC $ \ yield -> do
    void <- doubleNegation $ \ x ->
      yield x -- goodbye!
    absurd void

From this, we can see the loophole: by passing in a continuation a -> Cont Void into the doubleNegation :: (a -> Cont Void) -> Cont Void, the program can capture and send the result to the caller through a non-local jump. The remaining part involving absurd is just to satisfy the type checker.

For example, if the doubleNegation was constructed like this:

doubleNegation :: (String -> Cont Void) -> Cont Void
doubleNegation negation = negation "strings do exist"

then doubleNegationElimination would be able to manifest the evidence "strings do exist" by capturing the value and then immediately jumping to the outer scope, avoiding the crash that would have arisen from evaluating a Void.

On a sidenote, callCC is crazy powerful. You can implement Python-style generators with them!

import Control.Monad.Cont (MonadCont, MonadIO, (<=<),
                           callCC, forever, liftIO, runContT)
import Data.Foldable (for_)
import Data.IORef (IORef)
import Data.Void (Void, absurd)
import qualified Data.IORef as IO

main :: IO ()
main = runContT generatorExample pure

generatorExample :: (MonadCont m, MonadIO m) => m ()
generatorExample = callCC $ \ stop -> do
  g <- newGenerator [0 .. 9 :: Integer]
  forever $ do
    result <- g
    case result of
      Nothing -> stop ()
      Just x  -> liftIO (print x)

newGenerator :: (MonadCont m, MonadIO m) => [a] -> m (m (Maybe a))
newGenerator xs = callCC' $ \ ret -> do
  -- rNext stores the function used to jump back into the generator
  rNext <- newIORef ($ Nothing)
  -- rYield stores the function used to jump out of the generator
  rYield <- newIORef <=< callCC' $ \ next -> do
    writeIORef rNext next
    ret $ do
      -- the generator itself is just a function that saves the current
      -- continuation and jumps back into the generator via rNext
      callCC' $ \ yield -> do
        readIORef rNext >>= ($ yield)
  for_ xs $ \ i -> do
    -- save the current continuation then jump out of the generator
    writeIORef rYield <=< callCC' $ \ next -> do
      writeIORef rNext next
      readIORef rYield >>= ($ Just i)
  readIORef rYield >>= ($ Nothing)

-- | This is just double negation elimination under a different name, which
-- turns out to be a little more convenient than 'callCC' here.
callCC' :: MonadCont m => ((a -> m Void) -> m Void) -> m a
callCC' action = callCC ((absurd <$>) . action)

newIORef :: MonadIO m => a -> m (IORef a)
newIORef value = liftIO (IO.newIORef value)

readIORef :: MonadIO m => IORef a -> m a
readIORef ref = liftIO (IO.readIORef ref)

writeIORef :: MonadIO m => IORef a -> a -> m ()
writeIORef ref value = liftIO (IO.writeIORef ref value)

The need to associate function pointers with environments

Wed, 21 Sep 2016 00:00:00 -0700

A friend once asked me a question like this:

void register_event_handler(void *f_ctx, void (*f)(void *ctx));
I’m a little confused about the purpose of f_ctx. Here, f is a handler function that gets called when the event is triggered, and f_ctx is − according to the documentation – some pointer argument that gets passed to f whenever it gets called. Why do we need f_ctx? Wouldn’t f alone suffice?

This is a trick for low-level languages like C where functions are represented using a raw function pointer, which does not store an enclosing environment (sometimes called a context). It is not needed in higher-level languages with support for first-class functions, such as Python, as these languages allow functions to be nested inside other functions and will automatically store the enclosing environment within the function objects in a combination called a closure.

The need for an environment pointer f_ctx arises when you want to write a function that depends on external parameters not known at compile time. The f_ctx parameter allows you to smuggle these external parameters into f however you like.

It might be best to illustrate this with an example. Consider a 1-dimensional numerical integrator like this:

double integrate_1(
    double (*f)(double x), /* function to be integrated */
    double x1,
    double x2
);

This works fine if you know the complete form of the function f ahead of time. But what if this is not the case – what if the function requires parameters? Say we want to calculate the gamma function using an integral:

double integrand(double x)
{
    double t = /* where do we get "t" from?? */;
    return pow(x, t - 1.0) * exp(-x);
}

double gamma_function(double t)
{
    /* how do we send the value of "t" into "integrand"? */
    return integrate_1(&integrand, 0.0, INFINITY) / M_PI;
}

Using integrand_1 there are only three ways to do this:

Store t into a global variable, sacrificing thread safety. It would be bad to simultaneously call gamma_function from different threads as they will both attempt to use the same global variable.
Use a thread-local variable, a feature not available until C11. At least it is thread-safe now, but it is still not reentrant.
Write raw machine code to create an integrand on the fly. This can be implemented in a thread-safe and reentrant manner, but it is both inefficient, unportable, and inhibits compiler optimizations.

However, if the numerical integrator were to be re-designed like this:

double integrate_2(
    double (*f)(void *f_ctx, double x),
    void *f_ctx, /* passed into every invocation of "f" */
    double x1,
    double x2
);

Then there is a much simpler solution that avoids all of these problems:

double integrand(void *ctx, double x)
{
    double t = *(double *)ctx;
    return pow(x, t - 1.0) * exp(-x);
}

double gamma_function(double t)
{
    return integrate_2(&integrand, &t, 0.0, INFINITY) / M_PI;
}

This is thread-safe, reentrant, efficient, and portable.

As mentioned earlier, this problem does not exist in languages like Python where functions can be nested inside other functions (or rather, it is automatically taken care of by the language itself):

def gamma_function(t):
    def integrand(x):
        return x ** (t - 1.0) * math.exp(-x)
    return integrate(integrand, 0.0, math.inf) / math.pi

Sorting with a random comparator

Fri, 16 Sep 2016 00:00:00 -0700

Most sorting algorithms rely on the correct implementation of a comparison function that returns the ordering between two elements – that is, a function that takes two elements and returns whether the first is less than, equal to, or greater than the second:

function compare(x, y) {
    if (x < y) {
        return LESS_THAN;
    } else if (x > y) {
        return GREATER_THAN;
    } else {
        return EQUAL;
    }
}

The comparison function is required to define a valid total ordering on the elements in the sequence in order for the sorting algorithm to make any sense.

It is tempting to ask whether using a random comparison function would offer a clever way to shuffle a sequence. The short answer is no, generally speaking. It’s really not a good idea to do that even if it does somehow magically work. The correct solution is to use the Fisher-Yates shuffling algorithm.

From an engineering perspective, sorting functions are designed to sort things. Abusing it to shuffle an array is kind of just asking for trouble, because you never know if a change in the sorting algorithm down the road might break the “shuffling” algorithm. Worse, the breakage could be subtle, creating biases in the output that might be hard to detect unless the tests rigorously verify the statistics. Moreover, some sorting algorithms can run unusually slow with a random comparator, or take a very long time if the randomness was not in their favor.

From a mathematical perspective, bounded-time deterministic comparison sorts just can’t give you a high quality distribution of random permutations! In fact, if you use a random comparator with an equal probability of returning LESS_THAN or GREATER_THAN (a “uniformly random comparator”), it’s provably impossible for a such an algorithm to shuffle correctly.

Suppose you have an array of size N. The probability of any element (say, the first element) ending up at any position (say, the last position) is always 1/N in a perfect shuffle.

At every step of a deterministic comparison sort algorithm, it must make some sort of binary decision based on the ordering between two selected elements, leading to a binary decision tree. The outcome of these decisions determine the final sorted output.

Now if you feed it a uniformly random comparator, the decisions made are always random. This means every decision has a 1/2 probability of going one way or another. This leads to a path p down the decision tree that eventually terminates after n[p] rounds, since we are assuming bounded time. Certain paths in this decision tree will lead to the scenario where the first element ends up at the last position. Thus, the overall probability of this scenario is the sum of the probability of every path p that leads to the desired scenario:

∑[p ∈ all_desired_paths] (1/2)^(n[p])

Adding up powers of 1/2 will always yield a fraction where the denominator is some power of 2. Since the correct probability is 1/N, unless N is also a power of two, there is simply no way for the result to be correct! Hence, there’s no way to get the exact probability 1/N in general.

Edit: Chris P. pointed out that the algorithm must have bounded time for this proof to work. Thanks!