The Reference Class Problem and Mathematical Models of Inference
| Published date | 01 October 2007 |
| DOI | 10.1350/ijep.2007.11.4.259 |
| Date | 01 October 2007 |
| Subject Matter | Article |
The reference class
problem and
mathematical models
of inference
By Dale A. Nance*
Professor of Law, Case Western Reserve University
he ubiquity of reference class problems is conceded. I deny, however,
that this precludes the use of a formal analysis that involves selection of
T areferenceclassbyawitnessforusebythetrieroffactorbyajudgeor
theorist as a tool in evaluating inferences. Specifically, I reject claims by Allen and
Pardo that researchers using mathematical probability to analyse the probative
value of evidence have proceeded on the assumption that for each item of
evidence there is a uniquely correct reference class that can be selected for this
purpose without an act of judgment.
Introduction
A trier of fact must think about the relative compatibility of evidence with the
contending allegations of the parties, their theories of the case. Is the evidence more
compatible with the plaintiff’s theory or with the defendant’s theory? If so, how
much more compatible with one than the other? In answering these questions, the
trier of fact appeals (explicitly or implicitly) to generalisations about the way the
world works that connect the evidence to the allegations. For example, suppose a
jury is presented with Willard’s testimony that Delbert was seen running from the
scene of a murder. Jurors assess the strength of the inference from that evidence to
the prosecution’s claim that Delbert committed the crime by appealing to various
generalisations about the motives of someone who flees a crime scene, such as
‘people who run from a crime scene are more likely to be acting out of
consciousness of guilt and a resulting fear of apprehension than out of other
*
Email dale.nance@case.edu.
THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF
(2007) 11 E&P 259–273
259
REFERENCE CLASS PROBLEM AND MATHEMATICAL MODELS OF INFERENCE
motives’. For the most part, these generalisations are based on the common experi-
ences of the trier of fact. Sometimes, however, the tribunal is assisted by the
testimony of expert witnesses who provide information about pertinent generalisa-
tions that may not otherwise be available to the trier of fact.
The measure of probative value of the evidence will depend on the reference class
selected. For example, it matters whether one takes into account the fact, if there
is reason to believe it is a fact, that Delbert belongs to a special class of people who
might have reasons to flee the police other than their consciousness of guilt for
the murder. Delbert’s belonging to the class of persons with a prior record for
serious crime or to a race of persons (say, Native Americans) having poor relations
with the local police would call for an adjustment in the applicable generalisation.
There are a great many potential reference classes that can be selected for any
evidenced event, depending upon which of the identifiable characteristics of the
event are suppressed in order to generate a usable generalisation. In the example,
one might choose any of the following reference classes, among others: (a) all
those persons who are seen running from a murder scene; (b) all those Native
Americans who are seen running from a murder scene; (c) all those persons with a
serious criminal record who are seen running from a murder scene; or (d) all those
Native Americans with a serious criminal record who are seen running from a
murder scene. The frequency (and associated proportion) of individuals fleeing
out of consciousness of guilt could be dramatically affected by which reference
class is chosen.
In their recent article, Ron Allen and Michael Pardo seize upon this well-known
problem in order to make certain claims about the value of formal models of
evidence, specifically the employment of the standard probability calculus to
model probative value in terms of likelihood ratios or otherwise.1 Their argument
can be summarised in two propositions:
There is no objective, uniquely correct reference class for assessing the
probative value of particular evidence, other than a trivial class with
only one member (namely, the class of events sharing every feature of
the evidenced event).
1
Ronald J. Allen and Michael S. Pardo, ‘The Problematic Value of Mathematical Models of Evidence’
(2007) 36 Journal of Legal Studies 107. Space constraints preclude repetition of the formal models of
probative value that Allen and Pardo address.
260
E & P
REFERENCE CLASS PROBLEM AND MATHEMATICAL MODELS OF INFERENCE
The use of the probability calculus to model the probative value of
evidence too often involves the improper selection of a reference class,
beyond the trivial one, that is claimed (or at least believed) to be objec-
tively and uniquely correct.
To be sure, Allen and Pardo acknowledge that such probabilistic modelling can be
useful so long as it does not involve the selection of specific quantitative values as
the ‘true or correct’ probative values of evidence, and they cite an example of work
modelling probative value, using subjective probabilities, that avoids their
criticism.2 In other words, probabilistic modelling is not necessarily in conflict with
proposition A. Still, even in a less extreme form, like that stated in proposition B, I
believe their claim is false.
Privileged and useful reference classes
In order to understand their argument, one must try to get clear about what they
mean by ‘objectively’ and ‘uniquely’ correct, when they deny that such a correct
reference class exists. Unfortunately, this is not easy. Allen and Pardo put the
point in various ways, including the following:
The event under consideration ... is a member of an infinite number of
reference classes, the boundary conditions of which can be gerryman-
dered in countless ways ... And—outside of the reference class
consisting only of the event itself—nothing in the natural world privi-
leges or picks out one of the classes as the right one; rather, our
interests in the various inferences they generate pick out certain
classes as more or less relevant.3
Importantly, they do not object to the selection of reference classes for use in
evaluating evidence, but only to the idea that such a selection is ‘necessarily privi-
leged’.4 If the point is simply that the appropriate reference class cannot be
logically deduced from some feature of the evidence itself or the factual
environment, without more, then the point is true but fairly obvious. Allen and
Pardo have not identified any theorist who denies this point, whether explicitly or
implicitly.
2
Ibid. at 129–30.
3
Ibid. at 112.
4
Ibid. at 115. Under any of the standard formal models, one must have at least two probabilities in
order to calculate a probative value. I take Allen and Pardo’s claims to refer to the selection of
reference classes for each of these probabilities.
THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF
261
REFERENCE CLASS PROBLEM AND MATHEMATICAL MODELS OF INFERENCE
The more pertinent question is whether and how, from among the multitude
of potential reference classes, one can be selected as optimal or at least practi-
cally useful for assessing the probative value of the evidence in a particular
context. Allen and Pardo do not elaborate on the way in which our interests in
the various inferences they generate ‘pick out certain classes as more or less
relevant’, but they thus concede that some ordering of reference classes is
possible and, indeed, that this is a part of our ordinary practices of drawing infer-
ences from evidence. They reiterate this point in slightly different language
elsewhere:
[S]ome [reference classes] will be better or worse than others because
some will provide better or worse information about what we are
trying to infer regarding the underlying event. But the question of
which is which will, like any other evidence, be the subject of
argument and, ultimately, judgment.5
Actually, the selection and use of reference classes occurs regularly, not to say
unproblematically, when people draw inferences, with or without the benefit of
‘argument’. In particular, it occurs when jurors draw inferences, but also when
experts or lay witnesses assist jurors by giving testimony...
To continue reading
Request your trial