Book Review: Advancing Validity in Outcome Evaluation: Theory and Practice (New Directions for Evaluation, No. 130)
| Author | Darrel N Caulley,Eric Weir |
| Published date | 01 September 2012 |
| Date | 01 September 2012 |
| DOI | http://doi.org/10.1177/1035719X1201200206 |
| Subject Matter | Book Reviews |
BOOK REVIEWS
46 Evaluation Journal of Australasia, Vol. 12, No. 2, 2012
This is another collection of papers
occasioned by the controversy in
the US regarding the relevance
of experimental methods in
evaluation. The specific focus is on
validity in outcome evaluation, but
that constraint is not critical.
Most of the papers began
as presentations for a panel
on the topic at the American
Evaluation Association’s 2008
annual conference. Because it has
exercised such a powerful influence
on evaluation, the Campbellian
typology of validities is taken as
the framework for discussion of the
issues.1 Unfortunately, unlike in the
Campbellian tradition, there seems
to have been little eort devoted to
achieving consensus on the issues;
unfortunate in part because from
the positions taken it appears that
at least a measure of consensus
could be achieved.
Collectively, the chapters
provide both a critique of the
typology as applied in evaluation
and suggestions for enhancing
validity in evaluation. The
criticisms are likely to be taken
seriously by all but dogmatic
adherents of the typology. Even
many who find the criticisms
telling, however, are likely to be
disappointed in the suggestions.
The problem is that they appear to
assume the very thing that they are
intended to establish—knowledge
of generalisable causal relations—
and to leave unanswered the
question of how the judgements
called for are to be evaluated.
Criticism in the context of
appreciation
In the concluding chapter of
the volume, William Shadish,
one of the co-authors of the
final volume in the Campbellian
trilogy, takes the other writers to
task for not reading the literature
of the tradition carefully and not
appreciating the extent to which the
issues raised have been addressed
by the tradition. In fact, across
the chapters there is a good deal
of appreciation for the tradition’s
contribution to the understanding
of validity.
In their introductory chapter,
for example, the editors note that
while the fundamental conception
of validity evolved over the
course of the development of
the typology, the driving concern
has been validity of inferences.
Taking validity to be a property
of methods, they say, ‘does not
fit well’ with Campbell’s broader
body of work. They acknowledge
that Campbell introduced external
validity to bring attention to
the problems of generalising
beyond research settings and
that over the years the tradition
has given increasing attention
to enhancing external validity.
And Campbell’s emphasis on the
fallibility of knowledge claims and
the attention he gave qualitative
considerations in inquiry are
acknowledged as well. While
across the chapters the typology
is not spared criticism, most of
the criticisms address the way
the typology has been applied in
evaluation, not the typology itself.
Criticisms of the typology
Inherently problematic features
and impact on evaluation
Many of the criticisms are familiar.
They are frequently advanced
in debate about the relevance
of experimental methods in
evaluation. The basic argument is
that when the typology is taken as
a guide to evaluation, evaluation
fails to meet the needs it is expected
to serve.
Some of the writers identify
features of the typology that
are considered problematic in
themselves, independent of their
impact on evaluation. Jennifer
Greene is the clearest example.
She oers an interpretivist
perspective on use of the typology
in evaluation. It assumes, she
argues, a neutral objective
evaluator, a view of facts as
autonomous and of validity as
Title: Advancing Validity in Outcome Evaluation: Theory and Practice
(New Directions for Evaluation, No. 130)
Editors: Huey T Chen, Stewart I Donaldson and Melvin M Mark
Publisher/year: Jossey-Bass, San Francisco and the American Evaluation Association, 2011
Extent/type: 123 pages, paperback
Price: US$29 from Wiley Australia which oers a 15% discount to AES members; phone 1800 777 474 (within Australia),
0800 448 200 (from NZ only), +61 7 3354 8444 (from overseas), email <cs-journals@wiley.com>,
website <http://www.wiley.com> for the latest prices
ISSN: 1097-6736
edited by Darrel N Caulley
EJA_12_2.indb 46 15/01/13 10:01 AM
To continue reading
Request your trial