Identity and the limits of fair assessment
| Author | Rush T. Stewart |
| DOI | http://doi.org/10.1177/09516298221102972 |
| Published date | 01 July 2022 |
| Date | 01 July 2022 |
| Subject Matter | Articles |
Identity and the limits
of fair assessment
Rush T. Stewart
Department of Philosophy, King’s College London
Abstract
In many assessment problems—aptitude testing, hiring decisions, appraisals of the riskof recidivism,
evaluation of the credibility of testimonial sources, and so on—the fair treatment of different
groups of individuals is an important goal. But individuals can be legitimately grouped in many
different ways. Using a framework and fairness constraints explored in research on algorithmic
fairness, I show that eliminating certain forms of bias across groups for one way of classifying indi-
viduals can make it impossible to eliminatesuch bias across groups for another way of dividing peo-
ple up. And this point generalizesif we require merely that assessments be approximately bias-free.
Moreover, even if the fairness constraints are satisfied for some given partitions of the population,
the constraints can fail for the coarsest common refinement,that is, the partition generated by tak-
ing intersections of the elements of these coarser partitions. This shows that these prominent fair-
ness constraints admit the possibility of forms of intersectional bias.
Keywords
algorithmic fairness, bias, calibration, equalized odds, intersectionality
1. Introduction
Individual identity is multifaceted. Hannah, for instance, is a woman, an American, from
New York City (specifically the Upper West Side), but a resident of the South, a person
who spent several formative years in England, a Cambridge graduate, a philosophy
DPhil, an academic, from an upper middle class background, an advocate of risk literacy,
a runner, a violist, Jewish, a flexitarian, heterosexual, an effective altruism enthusiast, a
mother, a sister, a wife, and a fan of Andrei Tarkovsky and Townes van Zandt. Any one
of these properties applies to a large number of other people, defining a subgroup of a
Corresponding author:
Rush T. Stewart, Department of Philosophy, King’s College London.
Email: rush.stewart@kcl.ac.uk
Article
Journal of Theoretical Politics
2022, Vol. 34(3) 415–442
© The Author(s) 2022
Article reuse guidelines:
sagepub.com/journals-permissions
DOI: 10.1177/09516298221102972
journals.sagepub.com/home/jtp
general population of individuals. For a given individual, different social contexts may
make membership in different groups more or less salient. The relative importance
attached to membership in such groups is also a matter of individual discretion, at
least to some degree. But one and the same individual can be a member of all of these
groups without contradiction. Since similar remarks apply to any individual, there are
many legitimate ways to group individuals in a population, from marital status or nation-
ality to religion or taste in music.
Fair treatment of different groups is an objective common to many domains of assess-
ment including aptitude testing in psychometrics (Borsboom et al., 2008), hiring deci-
sions in the labor market (Fang and Moro, 2011), risk assessment in the criminal
justice system (Kleinberg et al., 2017; Pleiss et al., 2017), and evaluation of the credibility
of testimonial sources in epistemology (Stewart and Nielsen, 2020).
1
Consider the case of
risk assessment. Using the same actuarial techniques that are used to calculate insurance
premiums, statistical software is employed in the U.S. criminal justice system to assess an
individual’s risk of re-offending. Given the type of crime committed, age, sex, employ-
ment status at the time of arrest, criminal history, etc., an individual is assigned (what can
be thought of as) a probability of recidivism. Such scores are used in sentencing and
parole decisions among other things. A 2016 ProPublica analysis of the risk scores of
the COMPAS statistical tool for Broward County, Florida found a form of bias in the
data on the tool’s predictions (Angwin et al., 2016b). The rate of false positives—the per-
centage of non-recidivists given a high risk score—was roughly twice as great among
black defendants as among white. And the rate of false negatives—the percentage of reci-
divists being given a low risk—among whites was roughly twice as great as among
blacks.
2
The bias is that these types of errors were asymmetrically distributed across
black and white sub-populations, affecting the lives of black and white people in very
different ways.
Research on algorithmic fairness studies the prospects of unbiased assessment. Bias in
error rates is one form of bias, but not the only form and often considered not the most
important form. Can bias in error rates and other important forms of bias be simultan-
eously eliminated? One lesson that emerges from some of these studies is that eliminating
one form of bias can mean that it is impossible to eliminate another. Sometimes, then, we
face a conflict between eliminating different forms of bias. Here, I argue that, not only do
we face a conflict in eliminating different forms of bias, we also face a conflict in elim-
inating one form of bias across different groupings. Eliminating a certain form of bias
across groups for one way of categorizing people in a population can mean that it is
impossible to eliminate that form of bias across groups for another way of classifying
them. This conflict is significant to the extent that multiple classifications are relevant.
And they often are: consider the various classes mentioned in standard non-
discrimination clauses, for example.
3
Moreover, even if our assessments are unbiased
for certain ways of classifying people—say for both a race classification that includes
black and white categories and a gender classification that includes categories for
women and men—bias can persist for the coarsest common refinement of these classifi-
cations—in this case, the single classification that includes the groups of black women,
black men, white women, and white men. In other words, forms of intersectional bias are
possible for the prominent fairness constraints in the fair algorithms literature. Given the
416 Journal of Theoretical Politics 34(3)
To continue reading
Request your trial