In recent years the issue of fairness (or lack thereof) in machine learning models has become a far more prominent issue. A big catalyst for this shift arose from discussions surrounding the ProPublica analysis [1, 2] and subsequent counter arguments [3] of Equivant’s (formerly Northpointe) COMPAS pre-trial risk assessment tool. Analysis showed that African-American defendants were more likely to be incorrectly labelled as higher-risk of recidivism and white defendants more likely to be incorrectly labelled as lower-risk. However the model was well-calibrated across different populations (see e.g. [4, 5, 6] for various discussions).

The fact there existed these two seemingly contradictory viewpoints gave rise to important theoretical questions.

1. Which notion of fairness is correct?

2. If one notion is fair, why is the other not?

3. Can the predictive model be made to satisfy both notions?

Answering these questions has led to the so-called impossibility theorem [3, 4], which states that, in general, it is impossible for a predictive model to simultaneously satisfy multiple established definitions of fairness.

Below I attempt to use Adam Pearce’s nice interactive example [7] (which I recommend viewing prior to reading) regarding testing for medical conditions within different age groups to give a (hopefully) simple mathematical illustration of some of the key points. I will do this following a causal data generation viewpoint put forth by Chiappa and Isaac [5].

The presented model consists of three variables; a patient’s age $A$ (either child $c$ or adult $a$), the presence of condition $C$ (either 0 or 1) and a subsequent test $T$ (also 0/1 for negative/positive). In our model adults are more likely to have the condition than children (arthritis, heart disease for example) and the probability of a test result simply depends on the absence or presence of the condition.

Using those constraints the joint probability distribution of $A$, $C$ and $T$ can be written as

and represented diagrammatically by the graph in Figure 1 . Together, the graph and probability distribution form a Bayesian Network (BN).

Following the example in [7] we take the proportion of children $c$ and adults $a$ to be the same (i.e. $P(A=a)=P(A=c)=1∕2$). The probability of the condition for different age groups $P(C|A)$ we write as

with $\alpha <\beta$ as the prevalence is higher in adults.

For the probability of test result we have

Here $p$ and $q$ are the true positive/negative rate (TPR/TNR) respectively. Often there is a single tunable parameter (as in [7]) for which a result above/below is deemed as positive/negative (e.g. the concentration of virus in a covid blood sample). In that case the TNR $q$ would decrease as the TPR $p$ is improved, however we leave that assumption for now.

To answer any of our questions questions first requires a formalism. Specifically a definition of fairness that can be applied to the model. The introduction outlined two competing viewpoints that give rise to two corresponding definitions of fairness (there are also others), which are outlined below.

Interestingly, in the current context, both definitions might be considered reasonable but we will see that, unfortunately, our model can not satisfy both simultaneously. This dichotomy is the root of the impossibility theorem.

3.1 Fairness in accuracy

The first notion of fairness we might think of is that the test result $T$ only depends on whether a person has the condition $C$, and is therefore the same if the person is either an adult or child, i.e.

Formally we say the test results are conditionally independent of age given the condition, or mathematically

This form of fairness is sometimes called accuracy fairness or equal false positive rates and true positive rates and in our example we see immediately from Equation (1 ) this is satisfied. This is the form that was violated by the COMPAS tool according to the ProPublica analysis.

3.2 Fairness in calibration

A second notion might also be considered, which inverts our first notion. Suppose we receive a particular test result $T$, we then want to know how likely it is that we actually have the condition $C$, which is arguably a more useful piece of information than our previous measure. This might be considered unfair then if an adult and child receive the same test result but the chances of them actually having the condition are different. To be fair, therefore, we can argue this measure should be the same for both adults and children, i.e.

Again, we can write this in the alternative mathematical form

which conveys the conditional independence of the condition $C$ on age $A$ given the test result $T$. This alternative form is sometimes referred to as calibration fairness.

Unfortunately our example model does not satisfy this alternative definition of fairness. To see this we note that, by definition of conditional probabilities (Bayes’ Theorem),

where

Using our values from Table 2 gives for children

and adults

To satisfy calibration fairness we make them equal, as in Equation (3 ), which leads, after some algebra, to

However this only holds in general if $\alpha =\beta$, i.e. the proportion of adults and children having the condition (the base rate) is the same (or $P(C|A)=P(C)$).

Suppose we believe that calibration fairness (4 ) is more important than accuracy fairness. Obviously, we cannot alter the base rates of the condition in adults and children, however we do have some ability to alter the test results. Suppose we adjust the $p$ and $q$ in Table 2 separately for children and adults (having seperate cut-offs for the amount of virus that signifies a positive test result for example). This change leads to a new joint probability distribution

where the conditional probability $Q(T|A,C)$ would be of the form

for $A=c,a$. Mathematically, this difference means that

and we immediately see that, unfortunately, our model no longer satisfies the fairness in accuracy definition (2 ).

The new probability (5 ) is represented diagrammatically in Figure 2 . The dotted lines conveys the changes brought about by the modifications in $Q(T|A,C)$.

Our goal now is to choose $q_A,p_A$ in such a way that our calibration measure is fair. Using our conditional probability relations and that $P(A)=1∕2$ means the calibration measure can be written as

Now, for simplicity we denote the elements $\alpha$ and $\beta$ in Table 2 as $P(C=1|A)=r_A$. Then, using Table 4 ,

and similarly

To satisfy calibration fairness the conditions (6 ) and (7 ) must be the same for children and adults. Imposing that restriction results (see Appendix A for details) in the following dependencies

and

where $\psi ={\varphi }_c∕{\varphi }_a$ and

This puts constraints on the parameters. For instance, if $\psi <1$ then we can’t have $1-p_c<q_c<1-\psi p_c$, otherwise $q_a$ would be negative. However, if satisfied we can rewrite our joint probability distribution as

which is represented diagrammatically in Figure 3 .

One set of parameters that does satisfy the calibration fairness when $\alpha =0.1$ and $\beta =0.2$ is given by

which corresponds to

Thus, by maintaining high TPR and TNR for children we are left with a situation where for adults, although we have a high TPR, the TNR is much lower, meaning we are likely to see many false positives for adults.

By following the simple example from [7] we see that different base rates in the population (i.e. of condition prevalence) results in a situation where two seemingly sensible notions of fairness (accuracy and calibration) cannot be satisfied simultaneously.

Importantly, this example highlights just how important human judgement is in the interpretation of prediction models. It would be great if there was a universal definition of fairness for which each model could be benchmarked however this example shows the notion of fairness is a distinctly personal concept that can change on context and lives outside the scope of the model.

There are many more aspects concerning the difficulties in making prediction models, for instance Kleinberg et. al. [4] offers other definitions and commentary (see also the seminar on YouTube [8]). Chiappa and Isaac also offer a far more expanded view on how causality techniques techniques can tell us which fairness definitions might be valid and which not [5] (see also their DeepMind blog post [9]). For more general reading and a wider context the popular science books Weapons of Math Destruction by Cathy O’Neil [10] and The alignment problem: Machine learning and human values by Brian Christian [11] are also worth a read.

Let us derive the dependencies (8 ) and (9 ). Taking the first expression (6 ) and setting that equal for adults and children gives

Similarly, doing so for the second expression (6 ) gives

If we set

then the first equality reduces to

and the second

Writing $\psi ={\varphi }_c∕{\varphi }_a$ for simplicity, the first equality becomes

which gives us (8 ). The second equality is reduced to

Adding the two together gives

$$\begin{array}{llll}\hfill 1& =\psi \left[\frac{p_c(1-q_a)}{1-q_c}+\frac{(1-p_c)q_a}{q_c}\right]& \hfill & \\ \hfill & =\frac{\psi }{q_c(1-q_c)}[q_c{p}_c(1-q_a)+(1-q_c)(1-p_c)q_a]& \hfill & \\ \hfill & =\frac{\psi }{q_c(1-q_c)}[q_a(1-p_c-q_c)+p_c{q}_c]& \hfill & \end{array}$$

and therefore

which is the second dependency (9 ).