Complex evaluations in the political context: Engaging stakeholders in evaluation design
| Date | 01 June 2018 |
| DOI | 10.1177/1035719X18778699 |
| Published date | 01 June 2018 |
| Subject Matter | Articles |
research-article2018
Article
Evaluation Journal of Australasia
2018, Vol. 18(2) 122 –131
Complex evaluations in the
© The Author(s) 2018
Reprints and permissions:
political context: Engaging
sagepub.co.uk/journalsPermissions.nav
https://doi.org/10.1177/1035719X18778699
DOI: 10.1177/1035719X18778699
journals.sagepub.com/home/evj
stakeholders in evaluation
design
Rhian Stack, Nerida Leal, Sarah Stamp,
Bianca Reveruzzi, Katrina Middlin and
Alexia Lennon
Queensland Family and Child Commission, Australia
Abstract
While evaluators can often access examples of complex evaluation frameworks
and reports, detailed information about the processes used to develop these
frameworks is less available. This article provides insight into the development
of an evaluation framework for a whole-of-government multi-component reform
program, being implemented by stakeholders over a 10-year period. This program
incorporates multiple tiers (activity-level and program-level) of evaluations, planned
at critical points of implementation. This article focuses on the consultation process
adopted in developing an evaluation framework to capture implementation progress
and program outcomes and impacts at three different time points. A consultative,
iterative approach, emphasizing stakeholder engagement, was adopted to mitigate
the effects of challenges of responding to the variable nature of the program and
multiple stakeholders with a vested interest in its implementation. Continual
reflection and quality improvement allowed these challenges to be addressed, and
will continue to allow lessons learned from earlier evaluations to inform the design
of future studies.
Keywords
collaborative evaluation design, engaging stakeholders
Corresponding author:
Rhian Stack, Queensland Family and Child Commission, PO Box 15217, Brisbane City East, QLD 4002,
Australia.
Email: rhian.stack@qfcc.qld.gov.au
Stack et al.
123
Introduction
While examples of complex evaluation frameworks and reports are widely accessible,
practical advice regarding the processes used to develop evaluation designs is less
available. This article shares insights for designing an evaluation framework for a
whole-of-government reform program. It will not provide comprehensive detail about
the design of this evaluation framework, nor prescribe designs for evaluators to use in
their own evaluation frameworks, but rather outline the approaches used to mitigate
the challenges stemming from evaluating a complex intervention.
This article reflects on the development of an evaluation framework for a series of
program-level evaluations of the Queensland Child Protection Reform Program (the
reform program). This is a large-scale, multi-level intervention, characterized by 121
recommendations1 implemented, monitored and evaluated by multiple government
and non-government partners. For the purpose of this article, program-level refers to
evaluations which capture the implementation and outcomes of the entire program,
rather than individual evaluations of the 121 recommendations which comprise it.
Monitoring and evaluation of these recommendations is the responsibility of imple-
menting agencies, and are referred to as activity-level.2
The evaluand—the Queensland Child Protection Reform Program
In 2012, in response to the concern that the Queensland child protection system was
failing vulnerable children and their families, the Queensland Government established
a Queensland Child Protection Commission of Inquiry (QCPCOI) to chart a roadmap
for the child protection system for the next decade. The QCPCOI undertook a “root to
branch” review of the system to find out whether it was meeting the needs of
Queensland’s children, and if not, to explore the reasons why and determine a way
forward. This was the third of such inquiries in less than 20 years, and by far the most
comprehensive of these systemic investigations.
In response to the QCPCOI Final Report, Queensland’s child protection system
is undergoing a 10-year reform program guided by 121 recommendations targeting
all aspects of the child protection system, including early intervention and preven-
tion services to support families to safely care for their children at home, public
education campaigns encouraging parental help-seeking, and developing new gov-
ernance structures to promote efficient and effective cross-sector partnerships.
Collectively, these recommendations aim to build a child and family support sys-
tem that will have a greater focus on supporting families earlier so they can pro-
vide a safe home for their children and reduce the burden on the tertiary child
protection system.
The Queensland Government has made a significant investment to reform the
child protection and family support system, having committed to implementing this
broad range of initiatives in the first 5 years of the program’s lifespan. It is assumed
that increased investment in early intervention and prevention services (the second-
ary system) will reduce the demand in the statutory (tertiary) child protection system,
124
Evaluation Journal of Australasia 18(2)
allowing redirection of resources to enable sustainable funding of the secondary sys-
tem. With full implementation, the child protection landscape in Queensland is
expected to be...
To continue reading
Request your trial