‘Improvement focused’ evaluation of place-based initiatives: An examination of three methodologies
| Author | Lucio Naccarella,Lauren Heery,Rosemary McKenzie |
| Published date | 01 September 2018 |
| Date | 01 September 2018 |
| DOI | http://doi.org/10.1177/1035719X18793704 |
| Subject Matter | Articles |
https://doi.org/10.1177/1035719X18793704
Evaluation Journal of Australasia
2018, Vol. 18(3) 165 –182
© The Author(s) 2018
Article reuse guidelines:
sagepub.com/journals-permissions
DOI: 10.1177/1035719X18793704
journals.sagepub.com/home/evj
‘Improvement focused’
evaluation of place-based
initiatives: An examination
of three methodologies
Lauren Heery
Centre for Community Child Health, Murdoch Children’s Research Institute, Australia
Lucio Naccarella and
Rosemary McKenzie
Centre for Health Policy, Melbourne School of Population and Global Health, The University of
Melbourne, Australia
Abstract
Place-based initiatives offer a comprehensive, whole-of-community approach to solving
complex problems. Impact evaluation of complex initiatives is challenging and alternative
‘improvement focused’ methodologies, such as developmental evaluation, action
research and quality improvement, are being used. Limited understanding exists about
how these methodologies work when used in place-based initiatives, which contexts
they are individually best suited to and what they can achieve. This article examines the
methodologies of developmental evaluation, action research and quality improvement
when applied to the evaluation of place-based initiatives. The approach used a realist
evaluation methodology, involving a 10-year literature review and three ‘instrumental’
case studies, which is described in detail in an accompanying paper (Heery, Naccarella
& McKenzie, 2018). Contextual factors, mechanisms and outcomes for the application
of developmental evaluation, action research and quality improvement to place-based
evaluation were identified so as to build a theory for each methodology. The three
methodologies have similar mechanisms; they are cyclical and comprise planning,
doing, studying and acting, and all result in continuous improvement of the initiatives,
increasing collaboration and increasing evaluation capacity. A key difference between
Corresponding author:
Lauren Heery, Murdoch Children’s Research Institute, The Royal Children’s Hospital, Flemington Road,
Parkville, VIC 3052, Australia.
Email: lauren.heery@mcri.edu.au
793704EVJ0010.1177/1035719X18793704Evaluation Journal of AustralasiaHeery et al.
research-article2018
Article
166 Evaluation Journal of Australasia 18(3)
the methodologies is their perceived purpose. These findings can support practitioners,
commissioners and users of evaluation working in the place-based arena in three ways:
advocating for the adoption of ‘improvement focused’ evaluation; selecting the most
appropriate ‘improvement focused’ methodology; and identifying and addressing the
facilitating factors particular to the selected methodology to increase the effectiveness
of the evaluation.
Keywords
place-based, collective impact, developmental evaluation, action research, quality
improvement
Introduction
This study was undertaken to advance the place-based evaluative thinking and practice
of the Centre for Community Child Health (henceforth referred to as the Centre), a
research group of the Murdoch Children’s Research Institute and a department of the
Royal Children’s Hospital Melbourne, as well as the lead author’s workplace. Place-
based initiatives involve residents and service providers coming together to address
their problems by identifying the community-based contributing factors (e.g., poor
access to services, unsafe environs, poor social connectedness), and then implement-
ing a multi-pronged coordinated response. Place-based initiatives are complex adap-
tive systems, characterised by uncertain and emergent processes and outcomes (Patton,
2010) and are therefore difficult to evaluate in the traditional sense. It has been sug-
gested that the role of evaluation in place-based initiatives, particularly early on, is to
support learning and subsequent improvement of the initiative (Preskill, Parkhurst,
Splansky, & Juster, 2014a; Turner et al., 2014).
This study examines three ‘improvement focused’ methodologies used by the Centre in
place-based evaluation, namely developmental evaluation, action research and quality
improvement, and explores which contexts they are best suited to, how each of them works
when applied to a place-based initiative and what the result of applying each of them is.
Further detail of the study’s background and rationale is provided in the accompa-
nying paper (Heery, Naccarella, & McKenzie, 2018).
Method
The methodology adopted for this study was realist evaluation and it used a two-stage
design, summarised in Table 1. For full details of the study’s method see the accompa-
nying paper (Heery et al., 2018).
Findings
Developmental evaluation
Findings from the literature review. Of the 15 papers, 7 were identified as using develop-
mental evaluation as the key evaluation methodology. Only one of these papers was
To continue reading
Request your trial