Evaluation: What is it Really for? A Framework for Examining Issues of Evaluation Purpose and Use

R E F E R E E D A R T I C L E
E v a l u a t i o n J o u r n a l o f A u s t r a l a s i a , V o l . 5 ( n e w s e r i e s ) , N o . 1 , S e p t e m b e r 2 0 0 5 , p p . 1 1 – 1 7
Evaluation: what is it really for?
A framework for examining issues of evaluation
purpose and use
Anne Barlow
This article considers matters relating to evaluation
purpose, and the dissemination and use of fi ndings.
The author discusses some of the challenges occurring
in a particular health services evaluation in New Zealand
and proposes an acronym framework, based on the
word ACCEPTANCE (accessibility, audience, criteria,
credibility, credentials, communication, ethics, purpose,
politics, participants, position, timeliness, tone, attitude,
appearance, numbers, critique and excellence of
evaluation) to explore associated issues in the wider
evaluation context.
Introduction
Anne Barlow works in
This article considers issues relating to evaluation purpose, and the dissemination
Auckland as an independent
and use of fi ndings. For the purposes of discussion, I have provided an evaluation
researcher and evaluator, email
example from New Zealand Maternity Services and based my comments on
.
programme evaluation. Midwifery services in a small New Zealand public
maternity unit (Birthing Unit) were comprehensively evaluated using a mixed-

method approach, to ensure that the views of diverse stakeholder groups were
included. Whilst the goals and purpose of the evaluation were initially agreed on
by the key stakeholders, a number of social, political and management changes
during the life of the project provided challenges that ultimately infl uenced the
report dissemination process.
For clarifi cation, the terms ‘evaluation’ and evaluation ‘research’ are used
interchangeably here. There is also an assumption that evaluation fi ndings are
generally presented in the form of a written report, although other reporting
mechanisms could be chosen and similar issues confronted.
Background
Maternity services/community maternity units in New Zealand
In New Zealand, small community maternity units or ‘Birthing Units’ are
midwifery-led and located within the wider ordinances of District Health Board
(DHB) maternity service provision in New Zealand. Local communities value
them for their accessibility and for the ‘non-interventionist’ approaches to
pregnancy and childbirth maintained by the hospital and self-employed midwives
working there.
All New Zealand citizens or residents are entitled to free maternity and
midwifery care, and over 75 per cent of women use the services of a Lead
Maternity Carer (LMC) who is a midwife. Women deemed ‘low risk’ may choose
a Birthing Unit, and if complications arise they can transfer at any stage to a
nearby base hospital for specialist secondary services. However, many Birthing
Units in New Zealand have either closed or faced continual pressures to remain
B a r l o w — E v a l u a t i o n : w h a t i s i t r e a l l y f o r ?
11

---

R E F E R E E D A R T I C L E
viable in recent years. Midwives and management
the DHB had been written for the project, giving the
sought evidence to illumine the practice of midwives
Board opportunities for review of published report
and to support the safety of such units for women
material. However, because of management changes
and their infants.
this process took three months.
Furthermore, the researchers were requested not
The evaluation project
to provide public feedback to the local community
The collaborative study between the DHB and
because primary health care strategies were still
university researchers was initiated because a unit
being decided on and the public might have been
manager was frustrated by not being able to obtain
confused by the report. For instance, it became
money to decorate and paint the maternity unit.
apparent that the original purpose of the report,
She also felt that the skills of the midwives were
namely to determine the quality of care provided
undervalued and unappreciated by management.
by midwives in the unit, might not be useful if there
Constant threats of closure caused additional
was an agenda to close the unit in favour of service
concern, and the then midwifery and maternity
restructuring.
management believed a formal evaluation could
The researchers, meanwhile, were concerned
be a useful way to provide ‘evidence’ of practice
not to jeopardise the positive relationship that
and reveal the safe outcomes for women in this
existed between the university and the DHB, yet
midwifery-led service.
were also keen to meet their ethical obligations of
The purpose of the evaluation was, therefore,
providing feedback to participants and publishing
aimed at determining what was working well and
for university benefi ts.
what could be improved, in order to make the unit
As a result of careful communications with
sustainable. Overall, the Maternity Services Report
management, all evaluation stakeholders and
found that the unit had relevance to the community
participants received a copy of the report. Also, the
fi ndings have been presented to
The collaborative study between the DHB and university researchers local midwives and at a national
was initiated because a unit manager was frustrated by not being
midwifery conference. During
the latter presentations, the unit’s
able to obtain money to decorate and paint the maternity unit.
anonymity was maintained. The
She also felt that the skills of the midwives were undervalued and
report dissemination process is still
unappreciated by management.
ongoing at the time of writing, and
is being undertaken in consultation
with management. Thus far, it has
and that the midwifery standards and services met a
been useful in supporting midwives in another New
high level of safety and satisfaction for those women
Zealand region wanting to maintain small units in
and families who used the unit.
their area.
Social, political and management change
During the year of the evaluation, a number of
A framework for examining issues of
changes affected the project and impacted on the
evaluation purpose and use
fi nal processes for disseminating the fi ndings.
It seemed that the Maternity Services’ evaluation
There were, for example, government-legislated
experience was probably not uncommon, and the
health policies (Section 88, Health and Disabilities
evaluation issues and questions it surfaced were
Act 2000) that came into effect in mid-2002
worth wider exploration. In particular, throughout
(Ministry of Health 2002). This Act infl uenced the
a number of levels of the evaluation, there was a
provision and structure of...