‘Death by evaluation’? Reflections on monitoring and evaluation in Australia and New Zealand1

‘Death by evaluation’?
Reflections on monitoring and
evaluation in Australia and
New Zealand1
mandatory evaluation
Bill Ryan
This paper reviews the state of play in outcome-oriented
monitoring and evaluation in New Zealand in the light of recent
Australian experience. Given issues arising from a ‘big bang’
approach to evaluation adopted with the Commonwealth of
Australia’s 1988 Evaluation Strategy, and the lead times involved
in building capacity, it argues that New Zealand should focus in
the short-term on creating ‘evaluative management cultures’
inside organisations rather than opting exclusively for building
formal and technical evaluation capacity and expectations –
although that too should be developed, even though it will take
some years to mature. Actually, a practice rather than
institutional approach to capacity building that tries to ‘build from
within’ would also be consistent with current thinking in
management and organisational development. More may be
gained by developing a learning approach to public
management, strategic emphasis on internal evaluation, and
focusing on immediate and intermediate outcomes especially in
relation to monitoring.
Some in Aotearoa/New Zealand believe that the Commonwealth suffered ‘death by
evaluation’ during the 1990s. Considering the truth or otherwise of this assertion has led me
to certain proposals about where this country should head in coming years regarding
Bill Ryan is Associate
performance monitoring and policy and program evaluation. This paper works through the
Professor and Director
arguments.
of Programs in the
Some evaluators may find the proposals controversial. Public managers, however, are less
School of Government,
likely to do so, if only because they bridge the gap between ‘evaluation’ and ‘management’
Victoria University of
and are driven by pragmatic and contextual concerns. In overview, I argue that:
Wellington, Wellington.
■ As is now widely accepted, monitoring and evaluation in New Zealand agencies need to be
He can be contacted at
significantly upgraded, partly to improve the quality of management practices, partly to
.2
refocus public management on the outcomes being achieved, and partly to improve the
6
E v a l u a t i o n J o u r n a l o f A u s t r a l a s i a , V o l . 3 ( n e w s e r i e s ) , N o . 1 , A u g u s t 2 0 0 3

---

flow of performance information and
a reporting document, designed to sit alongside
accountability between the executive and the
the Annual Report, which was supposed to
parliament.
provide an extensive array of outcome-oriented
performance information to parliament as part
■ However, the preconditions do not exist in New
of the Estimates process.
Zealand for prescribing a rapid and intensive
introduction of performance monitoring and
It is also worth noting that the approach to
program evaluation as attempted by the
evaluation promoted by the DoF was decidedly
Commonwealth with its 1988 Evaluation
positivist and technocratic, and hence evaluation
Strategy.
methodologies and methods were deemed to be
relatively straightforward and needed only to be
■ Instead, New Zealand should adopt a two-
adopted to produce the desired results. These
pronged strategy designed to deepen practice
proposals, of course, were problematic (see various
over several years. A start should be made on
papers in O’Faircheallaigh and Ryan 1992).
building formal requirements and expert
Some State governments also increased their
capacity for evaluation – recognising, however,
commitment to evaluation around the same period,
that this will take time to produce results. In the
and by the early 1990s, evaluation activity
short term, consistent with current thinking in
throughout Australia seemed to be on the increase.3
management and organisational theory, more
Issues, however, were becoming apparent, including:
emphasis should be placed on building
■ the supply of sufficient skilled evaluators
evaluative cultures and practices from the
ground up amongst program staff, managers,
■ the long lead time required to build evaluative
agencies and providers, developing internal,
capacity across all agencies
learning-oriented evaluation, and refocusing
■ difficulties in obtaining and ring-fencing
monitoring activities on immediate and
evaluation budgets and preserving the
intermediate program and policy outcomes.
independence of internal evaluation units
The Commonwealth: ‘death by
■ understanding the complementarities of
evaluation’?
quantitative and qualitative methodologies and
having the legitimacy of the latter accepted
As noted above, the view from this side of the
Tasman seems to be that (a) Australia now has a
■ the availability and integrity of long-run
thriving evaluation industry but (b) the
qualitative and quantitative data sets
Commonwealth Government went overboard with
■ the need for a mature and outcome-oriented
its 1988 Evaluation Strategy. It is asserted that this
approach to performance by senior management,
strategy demanded too much evaluation too quickly,
ministers and the parliamentarians, especially in
that evaluation requirements came to dominate
the estimates process.
everything else, and that a compliance culture in
evaluation emerged as a result. The lesson drawn in
In addition, the intellectual difficulty of, and the
New Zealand, where the lack of evaluation is now
amount of work involved in undertaking, high-
widely acknowledged, is that the ‘big bang’
quality performance monitoring and program
approach should be avoided. There is probably
evaluation caught newcomers by surprise. It was not
some validity in these views.
long before the
What was the 1988 Evaluation Strategy? How
Commonwealth
As noted above, the view from
did it come about? What happened as a
portfolios were
this side of the Tasman seems to
consequence?
looking to reduce the
be that (a) Australia now has a
Barrett (1992) describes how, after three years of
compliance pressure
thriving evaluation industry but
the Financial Management Improvement Program,
through a more
(b) the Commonwealth
the (then) Commonwealth Department of Finance
selective and strategic
Government went overboard with
(DoF) concluded that the level of evaluation being
approach to program
its 1988 Evaluation Strategy.
conducted within portfolios was too low and the
evaluation and a
quality too uneven. In 1988, the Commonwealth
reduction in their
Government announced the Evaluation Strategy for
reporting load. By 1993–94 some of the formal
immediate implementation. This entailed
requirements were de facto being wound back to
principally:
what were probably more realistic levels. The
principles of the Evaluation Strategy were
■ the development of five-year Portfolio
maintained, but stretched out on a longer
Evaluation Plans (including budget allocations)
implementation timeline, and at a lower level of
■ all major programs subjected to five-year rolling
prescription.
evaluations
Unfortunately, the new Coalition government
came to power in 1996 with a different public
■ a bias in favour of publication of the results.
management agenda based on economic rather than
■ the extensive provision of outcome-oriented
managerial reform, similar to New Zealand’s widely
monitoring and evaluation information in the
applauded model of the late 1980s. This was
agency’s Program Performance Report. This was
introduced with missionary zeal – as it had been in
R y a n – ‘ D e a t h b y e v a l u a t i o n ’ ?
7

---

this country – and, convinced of its correctness,
democracies in not progressing public management
Canberra seemed to lose interest in evaluating the
in this respect. Actually, no-one should have been
actual outcomes of these changes or the effects of its
surprised, since this outcome was predictable (Ryan
policies. The rise and rise of evaluation faltered,
1993). The New Zealand model of public
kept alive only in policy arenas where evaluation
management flowed out of an economic reform
was regarded as good professional and management
agenda, rather than a managerial one, that adopted
practice, including education, social policy,
outputs as the basic unit of public management, not
community services, criminology, family studies,
only for budgetary and financial management but
regional development and labour market programs.
also for program and policy management (e.g. NZ
So what was/is the situation in the
Treasury 1987). The logic of public management
Commonwealth at the beginning of the 21st
cast in this mould prioritised economy and
century? From this side of the Tasman it appears
efficiency, with accountability focused on the
that a fair amount of monitoring and evaluation is
control of agents (public managers, providers) by
being carried out, some good, some poor, some
principals (ministers, funders) through the Output
strategic and some ad hoc, but that overall, the
Purchase Agreement and the Chief Executive’s
quantity and quality are not as good as they might
Performance Agreement.4 Performance in producing
have been. Capacity seems to again be improving,
outputs, goods and services – the things agencies
particularly amongst external evaluators in the
said they would produce and the activities they said
private, community and public (tertiary) sectors, but
they would carry out – was made the basis of
the internal capacity of agencies to understand,
accountability; an approach that militated against
design and manage monitoring and evaluation has
the eventual development of outcome-oriented
not improved at the same rate. On the other...