Michael Ekstrand

Motivating Problem

The Queen of Brobdingnag has invited a number of Lilliputians to a feast, at which there will be cake. Because she is just, and unconcerned with the ongoing divisions between the Little-endians and Big-endians, she wishes to fairly decide who will get cake and who will not (the decision-making process). Our subjects are the Lilliputians, who are in one of these two factions; historically, the Little-endians have discriminated against the Big-endians, such that Big-endians generally occupy a lower position in Lilliputian society.

The Setup

For the sake of an example, I am going to use the problem of setting bail for an arrested suspect: ideally, in a just and fair world, we would set bail at the minimum amount necessary to ensure that the suspect shows up for their court appearance. This problem is one of those ProPublica considered in their analysis of COMPAS scores. Historically, the judge would set bail; increasingly, jurisdictions are using statistical tools that assess the likelihood that a suspect will commit a crime or skip bail as one factor in the bail-setting process.

The core of the paper is to break down the data-driven decision-making process into three components, called spaces. We have a subject — for example, an arrested suspect for whom we need to set bail — and they are represented in each of these components differently.

1. The underlying concept space, which we cannot directly observe, that reflects ground truth. This is each subject’s intelligence, ability, likelihood to skip bail, etc.
2. The observation space, which is the data we can observe: the subject’s prior arrest record, convictions, employment status, etc.
3. The decision space, which is the decision that our system makes about the subject (bail amount, hire/deny employment, etc.).

digraph G {
    rankdir="LR";
    O -> C [label="f"];
    C -> D [label="g"];
}

The whole of the decision-making process, therefore, can be broken down into two components: an observation process, which yields the observed data for a subject, and the decision process, which uses the observations to make a decision. Mathematically, these are represented by functions \(f: C \to O\) and \(g: O \to D\).

Within each of these spaces, we have some notion of distance between two subjects. So \(d_O(u,v)\) is how far apart two subjects \(u\) and \(v\) are in the observed space: that is, how different we would consider these two people (or objects) to be if we look at the observed information. We can also think about the distance between groups of people in a space, using a thing called the earth-mover’s distance. If we have two groups of people, the big-endians \(G_b\) and the little-endians \(G_l\), then we can talk about \(W_O(G_b, G_l)\), which is how different the two groups look when we look at their observed properties.

Note that there is nothing in these defnitions that ties the observation and decision processes to any particular mechanisms. This is what gives the paper’s work its power: we can use the tools of mathematics to make and prove statements about any implementation of these processes, including any combination of human and algorithmic mechanisms. If a judge looks at a case file and talks to a suspect and makes a decision, that is a definition of \(g\) that we can analyze. There are some mathematical tools that require us to know what is inside of a function, but there are quite a few that do not. Any of these tools that do not require knowledge of how a function works can be brought to bear, and we can analyze what is happening in a sociotechnical decision-making process with a great degree of mathematical rigor. The particular tools that the authors use are also perfectly fine with the underlying processes having uncertain outcomes: we can use probabilities to reason about the uncertainty and deal with decision processes that have a random component.

Analyzing human or semi-human processes as if they were mathematical functions is pretty cool, in my opinion. But I might be weird.

The Math of Worldviews

With this setup, we can now talk about worldviews, and describe them using math. This paper treats two worldviews; What You See is What You Get (WYSIWYG), and Structural Bias.

The WYSIWYG worldview says that our observations are basically accurate. That is, we can observe subjects so that if two suspects are pretty close ‘in reality’ (in the construct space), then they similarly close in our the observed space. The things that we observe about a suspect are good at accurately predicting their likelihood to skip bail.

The Structural Bias worldview says that our observations are not accurate and, more importantly, this inaccuracy is correlated with membership in a protected group. Mathematically, the authors define this with respect to group and individual differences: the obervation process moves groups apart, and it moves groups more than it moves individuals (they call this skew). That is, black and white suspects will look quite different in the observed space, and overall it makes the two groups more different than it makes individual black suspects or white suspects. In the extreme, it warps the data so that group differences are exaggerated and overwhelm the individual differences.

This is important. In many settings, within-group differences dominate between-group differences. That is, even if you have data that shows that little-endians score better on physics tests than big-endians, it is likely the case that if you compare the performance of two big-endians, you will probably see a bigger difference in their performance than you see between the two groups’ average performance. Structural bias skews our observations so that the between-group differences get larger, and they get larger at a bigger rate than within-group differences. The result is that group trends look bigger than they actually are. The underlying math is a little tricky, because it is a difference in differences; these things are difficult to interpret.

Structural bias can be caused by many different things. Discriminatory observations can result in two different subjects performing the same actions having different recorded results, as in student gender causing implicit bias in how an instructor grades an exam. Social factors may cause two different subjects with the same level of ability to perform differently, as in stereotype threat causing some groups of students to underperform on an exam, even though they know the material as well as other students. The mathematics in this paper do not care why the groups differ, and thus let us prove theorems about decision processes no matter what their underlying social or behavioral causes. Understanding those causes is important to developing some types of solutions, but being able to reason about bias and its effects without needing to fully understand what drives the bias is extremely powerful.

Treating People Fairly

A first definition of fairness is based on similarity: a decision process is fair if similar subjects receive similar decisions. That is, if two subjects are close together in \(C\), then they will receive similar decisions. This is the core idea of individual fairness.

In our bail example, if two suspects have the same likelihood of skipping bail or committing a crime, then they should have bail set at the same amount. The crucial thing about this fairness definition is that it should be independent of all other concerns. If a black suspect and a white suspect have the same likelihood, they shouldn’t receive different bail judgements.

The problem with this definition is that we do not know the construct space. We do not know a suspect’s true likelihood of commiting a crime or of skipping bail: all we can do is estimate it from our observations of them and of other suspects.

The authors show that under the WYSIWYG assumption, it is possible to treat people fairly under some conditions. Unfortunately, if the decision space is discrete (e.g. admit or deny, or set high / medium / low bail), they show that it is impossible to have a useful¹ decision process that guarantees the mathematical definition of fairness. Bummer.

Discrimination

Discrimination is a group property. Just as structural bias corrupts the observation process by spreading groups further apart than individuals, discrimination spreads groups apart in the decision process, and does so more than it spreads individuals. That is, in a discriminatory bail process, a black suspect may be more likely have bail set high than a white suspect with similar characteristics.

If the skew arises between the observed space and the decision space, then we have direct discrimination: the process is discriminating against members of a particular group by giving them different decisions, even though they have similar observable characteristics.

However, the skew does not need to arise there. If there is skew between the construct space and the decision space, then we can still consider the process discriminatory, even if it accurately and without bias makes decisions based on the observable information. Such indirect discrimination is a significant concern, because we humans are very good at finding ways to discriminate based on things that correlate with the characteristic we aren’t supposed to discriminate on.

It turns out that, if there is structural bias, then a ‘fair’ mechanism will be discriminatory.