Book Review: What Counts as Credible Evidence in Applied Research and Evaluation Practice?
| Author | Eric Weir |
| DOI | 10.1177/1035719X1101100110 |
| Published date | 01 March 2011 |
| Date | 01 March 2011 |
BOOK REVIEWS
Book reviews 47
This is a long slow slog through
a swamp many will feel they
have been through before: the
wars between the quantitative-
experimental and qualitative
approaches to evaluation. It is
a difficult read. Parts of it are
ponderous and pedantic in the
extreme, colouring one’s reaction
to it to the point that it becomes
difficult to continue reading. It
does not help that, with a couple
of important exceptions, it does
not contain much that is new.
Nor, that on its own terms, in
its aspiration to bring about a
meeting of minds, it must be
judged a failure.
Nevertheless, if we give
careful thought to what we
encounter, the investment does
repay itself. Both approaches
are represented by paradigmatic
presentations. Across many
chapters there is a sincere
effort to address the topic of
the book. Though most of the
contributors hold academic
positions, the realities of
practice and the challenges
they present to methodology
are well represented. Though
there is less dialogue than one
would like to see in a book on
this topic, careful readers will
be able to make connections,
pose challenges, and envision
responses where the contributors
have not.
The book consists of 12
chapters plus an epilogue, all
of which but the latter were
original presentations at a 2006
symposium on the topic at
Claremont Graduate University.
In his introductory chapter,
co-editor Stewart Donaldson
explains the context and occasion
of the symposium and the book—
increasing demand worldwide for
evaluation, especially ‘evidence-
based’ evaluation, at a time of
great controversy within the
profession about what counts
as credible evidence—and their
aspiration—by arranging a ‘full
vetting’ of positions to support
development of sufficient
consensus on the issues to
permit meaningful dialogue and
constructive collaboration across
them.
The main portion of the book
is divided into two sections, one
each devoted to experimental
and non-experimental responses
to the topic. This is unfortunate
for a number of reasons, most
importantly because it permits
the writers, if they choose,
and some do, to develop their
positions without taking into
consideration positions taken by
writers in the other approach.
In his concluding chapter,
Melvin Mark, another co-editor,
suggests that there are two issues
under discussion in the book, a
broad one about what counts as
credible evidence in general and
a more specific one regarding the
appropriateness of experimental
designs. This provides a more
helpful way of categorising the
contributions.
In addition, while Mark
seems to view the two questions
as independent, it seems not
unreasonable (given their clear
concern with causation, and with
considerations neglected by the
experimental approach) to take
the contributions that address the
broader question as attempting
to articulate a conception of
credibility that can provide a
basis for evaluating the adequacy
of the experimental responses to
the topic.
What counts as
credible evidence?
The writers who take up the
broader question, approach it
from a variety of perspectives.
All have reservations about
experimental design, especially as
setting the standard of credibility.
All, however, are sensitive to and
attempt to respond to concerns
about the validity, objectivity and
warrantability of observations,
interpretations and judgements
when evaluators no longer
have the explicit standards and
procedures of experimental
design and statistical analysis to
rely on.
The complexity of situations ■
and issues. The complexity
of the situations and issues
with which evaluation is
concerned is a theme across
the contributions focused on
the broader question. Jennifer
Greene addresses it most
directly. She sees a complexity
that is ‘deeply contextual’,
one that arises from ‘the
multiplicity of ways of
thinking, acting, and being at
work in society’ and of their
interaction with ‘structural,
economic, and political’
forces. She has no patience
for experimental methods
as the model of rigour in
evaluation. In focusing on
just a few variables at a time,
in neglecting the diversity
of stakeholder perspectives,
it ignores ‘the plurality of
legitimate epistemologies’
suggested by the philosophy
of science and privileges
certain stakeholders over
others. In its place, Greene
urges an interpretivist-
constructivist approach
focused more on ‘the
meanings people construct
in interaction with others’
Title: What Counts as Credible Evidence in Applied Research and Evaluation Practice?
Editors: Stewart I Donaldson, Christina A Christie and Melvin M Mark
Publisher/year: Sage, Los Angeles, 2009
Extent/type: 265 pages, paperback
Price: A$69/NZ$93 from Footprint Books which offers a 15% discount to AES members, phone +61 2 9997 3973,
email <info@footprint.com.au>, website <http://www.footprint.com.au> for the latest prices
ISBN: 978-1-4129-5707-6
EJA_11_1.indb 47 6/11/11 5:37:10 PM
Get this document and AI-powered insights with a free trial of vLex and Vincent AI
Get Started for FreeStart Your Free Trial of vLex and Vincent AI, Your Precision-Engineered Legal Assistant
-
Access comprehensive legal content with no limitations across vLex's unparalleled global legal database
-
Build stronger arguments with verified citations and CERT citator that tracks case history and precedential strength
-
Transform your legal research from hours to minutes with Vincent AI's intelligent search and analysis capabilities
-
Elevate your practice by focusing your expertise where it matters most while Vincent handles the heavy lifting
Start Your Free Trial of vLex and Vincent AI, Your Precision-Engineered Legal Assistant
-
Access comprehensive legal content with no limitations across vLex's unparalleled global legal database
-
Build stronger arguments with verified citations and CERT citator that tracks case history and precedential strength
-
Transform your legal research from hours to minutes with Vincent AI's intelligent search and analysis capabilities
-
Elevate your practice by focusing your expertise where it matters most while Vincent handles the heavy lifting
Start Your Free Trial of vLex and Vincent AI, Your Precision-Engineered Legal Assistant
-
Access comprehensive legal content with no limitations across vLex's unparalleled global legal database
-
Build stronger arguments with verified citations and CERT citator that tracks case history and precedential strength
-
Transform your legal research from hours to minutes with Vincent AI's intelligent search and analysis capabilities
-
Elevate your practice by focusing your expertise where it matters most while Vincent handles the heavy lifting
Start Your Free Trial of vLex and Vincent AI, Your Precision-Engineered Legal Assistant
-
Access comprehensive legal content with no limitations across vLex's unparalleled global legal database
-
Build stronger arguments with verified citations and CERT citator that tracks case history and precedential strength
-
Transform your legal research from hours to minutes with Vincent AI's intelligent search and analysis capabilities
-
Elevate your practice by focusing your expertise where it matters most while Vincent handles the heavy lifting
Start Your Free Trial of vLex and Vincent AI, Your Precision-Engineered Legal Assistant
-
Access comprehensive legal content with no limitations across vLex's unparalleled global legal database
-
Build stronger arguments with verified citations and CERT citator that tracks case history and precedential strength
-
Transform your legal research from hours to minutes with Vincent AI's intelligent search and analysis capabilities
-
Elevate your practice by focusing your expertise where it matters most while Vincent handles the heavy lifting