Why We Should Focus on Understanding the Individual in Evaluation
| Author | Saville Kushner |
| Published date | 01 September 2014 |
| Date | 01 September 2014 |
| DOI | http://doi.org/10.1177/1035719X1401400102 |
| Subject Matter | Article |
R E F E R E E D A R T I C L E Evaluation Journal of Australasia Vol 14 | No 1 | 2014 | p p . 4 – 1 1
Saville KuShner
Why we should focus on understanding
the individual in evaluation
This article1 is drawn from the work of Barry MacDonald
and is written as a tribute to him. It addresses an issue
he raised some while ago whose relevance intensifies
today in a world seeing the demise of independent
professional practice and public sector ethos. The
‘portrayal of persons’ relates to a paper MacDonald
wrote in 1976 (MacDonald 1976), at a time when the
discipline and the practice of evaluation was in rapid
emergence and development. He was arguing forcefully
that to understand innovation we have to understand
innovators. This was not a new message. It had been
enunciated elsewhere, including by Schwab (1978) and
Schön (1971) whose work revealed individualism, values
and exchange as key elements of innovation. But what
Saville Kushner is Professor of Public
MacDonald added was a political dimension: using the
Evaluation in the Faculty of Education at the
portrayal of persons to share control of the evaluation by
University of Auckland.
passing responsibilities for interpretation (generalisation)
Email: s.kushner@auckland.ac.nz
to the evaluand; and by empowering the organisation
by enhancing knowledge of itself (for example, against
attempts to enforce external values on professional
action through external accountability).
The rationale was the same as that for case study
evaluation: to ground generalisations in the experience
of the practitioners and to shape evaluation around
issues and dilemmas encountered in their practices. This
inevitably leads to working with small samples (cases)
but not losing the aim of deriving broad generalisation.
This was the key methodological principle that provoked
and continues to provoke reaction from those who
believe in a quantity theory of generalisation—that
4
E v a l u a t i o n J o u r n a l o f A u s t r a l a s i a Vo l 1 4 | N o 1 | 2 0 1 4
R E F E R E E D A R T I C L E
systems are made up of patterns of action
the program action but nonetheless recognises enough
and phenomena, and that we need
of it to be able to learn from it. For the practitioner to
sufficiently broad ranges of data to ensure
empathise and accept what was at least plausible, she or
that we have captured the full extension
he needed to see the ‘texture’, as it were, of the program
experience. If wise decisions were to be made on the
of a pattern. Case theorists argue that even
basis of a program’s experience then the reader needed
where this is true, it is more important
a realistic analysis of program realities and interactions.
to capture the inherent qualities of the
MacDonald (1976, p. 5) writes:
patterning and its contingent relationships
It is an axiom of sample-based generalisation that
with context. MacDonald sought to show, as
the sample must be adequately described in terms
I do in this article, that it is possible to arrive
of all its relevant characteristics. And it is a ‘finding’
at important and generalisable (provisional)
from our experience of educational evaluation … that
insights from a sample of one.
educational cases are behaviourally unique. It is a small
step from these premises to the conclusion that, if we
are to hold to the axiom, we must first seek adequate
descriptions of individual cases, their characteristics
Sampling professional action
and interactive effects. This will not enable us to
Barry MacDonald is the author of Democratic Evaluation
prescribe action to others.
(MacDonald 1974) and played ‘intellectual midwife’
I return to this theme at the end of the article. In
to evaluation case study (MacDonald & Walker 1975).
the next section, my tribute to Barry MacDonald is to
He was a pragmatist, which is to say that if an idea
carry forward his advocacy and analysis for the portrayal
or a proposition has no implications for action it is
of persons, and to ‘shift the locus of responsibility for
meaningless—that is, meaningless in the sense that it does
generalisation’.
not inform intent or motivation to act. This is why he was
drawn to educational evaluation. In fact, he distinguished
evaluation from research with two basic observations:
Personalisation
one, that evaluation in essence has consequences in action
In Personalising Evaluation (Kushner 2000) I restated
(mostly for resource distribution), whereas research
and recast the argument for case study and democratic
may have only theoretical implications; the other, that
evaluation, for methodological individualism and a focus
evaluation works from the practical dilemmas of others,
on people as agents of evaluative judgement. Perhaps
not from the priorities of the researcher. The first of
the most radical promise of these approaches is the
these—consequences—demands personal courage (and
capacity to generate ‘practical theories’, theories that
no mortgage); the second requires intellectual discipline
are collaborative attempts to understand and explain
and self-sacrifice (and no mortgage).
action—all of these approaches value the respondent not
MacDonald was also into titles with combustion.
just as ‘informant’, but as judge.
Titles for him were mini-summations of the orientation
The diagram in Figure 1 has come to be the basis
of the paper he was writing. The paper he wrote
of my advocacy in intervening years and summarises
immediately following his seminal article ‘Evaluation and
my book. The diagram is a proposal—that we invert
the control of education’ (MacDonald 1974) in which
the relationship between person and program. The
he proposed his typology of Democratic, Bureaucratic
conventional relationship sees the circle as representing
and Autocratic evaluation, was titled ‘The portrayal of
the program and the ‘X’ the individual. Most often in
persons as evaluation data’. Though the title bore none of
evaluation we take the program as the context and within
the panache of Bread and Dreams (evaluation of bilingual
it place the individual whose identity and significance
education policy), Mandarins and Lemons (evaluation
is given by the program. For example, in the study of
and the state bureaucracy) or Hard Times (education
a medical education program, the circle represents the
and accountability), it was startling and challenging for
program and the ‘X’ a nurse, patient, doctor or trainer,
the times. This was the time when he was advocating for
that is, whoever is expected to humanise the program.
what, then, people were calling ‘portrayal evaluation’—
This both denies an authentic identity to those people
what became subsumed under case study. When he
and, more importantly, distorts the significance of the
referred to portrayal MacDonald was proposing that
program. The inversion involves documenting the life of
evaluation serves to provide ‘vicarious experience’—vivid
the individual—making the circle the life context—and
accounts of life inside programs that lifted the reader
placing the program within it—the ‘X’ becomes the
into the case in its context. His purpose was to ‘shift the
program. Rather than evaluation seeking...
To continue reading
Request your trial