Developing evaluation criteria using an ethical lens
| Author | Amy M Gullickson,Mathea Roorda |
| Date | 01 December 2019 |
| DOI | 10.1177/1035719X19891991 |
| Published date | 01 December 2019 |
| Subject Matter | Academic Articles |
https://doi.org/10.1177/1035719X19891991
Evaluation Journal of Australasia
2019, Vol. 19(4) 179 –194
© The Author(s) 2019
Article reuse guidelines:
sagepub.com/journals-permissions
DOI: 10.1177/1035719X19891991
journals.sagepub.com/home/evj
Developing evaluation
criteria using an ethical lens
Mathea Roorda
Allen + Clarke, New Zealand
Amy M Gullickson
The University of Melbourne, Australia
Abstract
Selecting and justifying relevant criteria is critical to defensible evaluative reasoning, yet
there is little advice in the literature to guide practitioners in criteria development. In
this article, we introduce a framework that draws on normative ethical perspectives
to systematically identify and justify relevant dimensions of value for a public sector–
funded program. We illustrate, through an example taken from a recent research
project, how evaluators might use the framework to be more thoughtful about
identifying and selecting the values against which judgements are made. The potential
strengths and limitations of the framework are then discussed.
Keywords
criteria, descriptive valuing, evaluative reasoning, normative ethics, prescriptive
valuing, values
Introduction
Evaluation is, by definition, judgmental and values-based. This is articulated in the
general logic of evaluation and described as four steps: (a) establishing criteria, (b)
setting performance standards, (c) measuring performance and then (d) synthesising
and integrating into a judgement (Fournier, 1993, 1995; Scriven, 1982, 1993, 1995,
Corresponding author:
Mathea Roorda, Allen + Clarke, PO Box 5694, Wellington, NZ 6145.
Email: mathea.paradise@gmail.com
891991EVJ0010.1177/1035719X19891991Evaluation Journal of AustralasiaRoorda and Gullickson
research-article2019
Academic Article
180 Evaluation Journal of Australasia 19(4)
2005, 2012). The relationship between judgement and values is also recognised in
professional definitions of evaluation, such as the Australasian Evaluation Society
(AES; 2013) Guidelines for the Ethical Conduct of Evaluation, which refer to evalua-
tion as the
systematic collection and analysis of information to make judgments, usually about the
effectiveness, efficiency and/or appropriateness of an activity.
And as one of evaluation’s most eminent scholars, Greene (2011) notes:
The primary purpose of evaluation is to render judgments of merit and worth that are
grounded in defensible empirical evidence and argument and that are anchored in chosen
values. (p. 85).
Greene is not alone; many other scholars including House (1980), Nunns et al. (2015),
Schwandt (2008) and Scriven (1996) note the centrality of argument and values to quality
evaluation. For evaluative judgements to be justified, the evaluator must pay attention to
both evidence and values. As noted elsewhere in this journal (Gullickson & Hannum,
2019), values are relevant to all aspects of evaluation. In this article, the term ‘values’ refers
explicitly to the values against which judgements are made. We refer to these as criteria.
Davidson (2005) has underscored the importance of explicit criteria in recent years.
By visually representing criteria alongside standards and descriptors of performance
in rubrics, evaluators are able to make the reasoning process explicit and transparent
(Davidson 2005; Martens, 2018; Oakden, 2013). Explicit management of criteria is
consistent with U4: Explicit Values and A7: Explicit Evaluative Reasoning of the
Program Evaluation Standards (Yarbrough et al., 2011) and the Third principle:
Responsive methodologies and trustworthy results of Aotearoa New Zealand
Evaluation Association (ANZEA) standards (ANZEA and SuPERU, 2014). Explicit
criteria are a particularly important consideration when evaluating public policies and
programs, especially those considered high stakes (Chelimsky, 1995; Nunns et al.,
2015; Ryan, 2003). We assert that making criteria explicit is important but not suffi-
cient for rendering credible and defensible evaluative judgements. Those criteria must
also be carefully selected, defined and justified.
Scholars approach the development of criteria in different ways. Scriven (2013),
for instance, listed 19 possible sources of values that can contribute to criteria. His list
includes program goals but emphasises primacy of the context and needs of affected
population. House and Howe (1999) suggested that criteria be oriented explicitly
around normative ethical perspectives that focus on whether the program is ‘right’,
‘fair’ or ‘just’. They also advocated for a pluralist approach which deliberately seeks
out multiple value perspectives, as evident in participatory or democratic approaches
to evaluation. In 1991, Shadish, Cook and Leviton formalised that categorisation as
descriptive or prescriptive valuing: descriptive values come from stakeholders, pre-
scriptive values come from ethical theories. While eliciting descriptive values can
facilitate evaluation by addressing the political context in which it is practised, these
To continue reading
Request your trial