Using rapid evaluation methods to assess service delivery changes: Lessons learned for evaluation practice during the COVID-19 pandemic
| Author | Desiree Terrill,Milbert Gawaya,Eleanor Williams |
| Published date | 01 March 2022 |
| Date | 01 March 2022 |
| DOI | http://doi.org/10.1177/1035719X211057630 |
| Subject Matter | Practice Articles |
Practice Article
Evaluation Journal of Australasia
2022, Vol. 22(1) 30–48
© The Author(s) 2022
Article reuse guidelines:
sagepub.com/journals-permissions
DOI: 10.1177/1035719X211057630
journals.sagepub.com/home/evj
Using rapid evaluation
methods to assess service
delivery changes: Lessons
learned for evaluation
practice during the
COVID-19 pandemic
Milbert Gawaya, Desiree Terrill and Eleanor Williams
Victorian Department Health, Centre for Evaluation and Research Evidence, Melbourne, Australia
Abstract
The COVID-19 pandemic required large-scale service delivery changes for govern-
ment, and provided the opportunity for evaluators to step up and support decision
makers to understand the impact of these changes. Rapid evaluation methods (REM)
provide a pragmatic approach for generating timely information for evidence-based
policy and decision-making. Grounded in developmental and utilisation-focused
evaluation theory, REM incorporates a team-based, mixed methods design, exe-
cuted over a 6–8-week period. Customised rubrics were used to rigorously assess
effectiveness and scalability of practice changes to inform COVID-19 response
planning. REM is an alternative approach to full-scale evaluation models frequently
implemented to assess policies and programs. Adapted use of REM suggests that
meaningful insights can be gained through use of smaller scale evaluations. This article
shares lessons learned from a novel rapid evaluation method applied in the context of
the COVID-19 pandemic. The rapid evaluation approach was implemented to provide
real-time insights and evaluative conclusions for 15 program and practice adaptations
across Victorian health and human service settings. The article shares insights about the
practical applicability of balancing rigour and timeliness when implementing a rapid
evaluation, and strengths and limitations of working within a fast-paced evaluation
Corresponding author:
Milbert Gawaya, Victorian Department of Health, Centre for Evaluation and Research Evidence, 50
Lonsdale Street, Melbourne, Victoria 3000, Australia.
Email: Milbert.Gawaya@health.vic.gov.au
framework. Findings can inform evaluative practice in resource and time-limited
settings.
Keywords
rapid evaluation, evaluation methods, real-time evaluation, team-based analysis,
COVID-19 pandemic
Introduction
The COVID-19 pandemic brought about inevitable changes to the way in which
healthcare and human services care are delivered worldwide. The pandemic disrupted
or halted critical health services, and in many places led to increased demand for health
care (World Health Organization, 2020). Within Victoria, Australia, service changes
were shaped by policy direction from State Government which aimed to limit physical
interaction in the community and ensure infection control within high-risk settings
while maintaining service continuity. The unfolding crisis led to increased demand by
government policy makers for real-time evidence to optimise the development, im-
plementation, and adaptation of the health system, and related areas of human services,
in response to the COVID-19 outbreak.
In evolving public health contexts, such as the COVID-19 pandemic, traditional
comprehensive evaluation approaches often lag behind fast-paced innovation (Laura
et al., 2020). Faster techniques are required to understand effective service delivery
changes within a public health emergency context and evaluation must balance the
competing priorities of rigour and utility of evaluation findings (Raine et al., 2016).
This requires novel approaches to feed into rapid evaluation processes and provide
timely insights to inform effective response planning. The utilisation of ethnographic
methods makes rapid evaluation methods (REM) valuable for providing contextually
relevant evaluation findings in a timely and cost-effective manner.
The authors are staff at the Centre for Evaluation and Research Evi dence (CERE), an
internal evaluation unit within the Victorian Department of Health. The purpose of this
article is to share insights drawn from CERE’s adapted rapid evaluative practice during
the pandemic between March 2020 and June 2021. This article also responds to
growing interest in the Australian evaluation community around the question of how
should rapid evaluation methods be deployed and whether they can be adapted or
improved? (Australian Evaluation Society, 2020). Weshare experiences on how CERE
deployed the REM, incorporating a standard methodological evaluation framework and
customised rubric to increase rigour. This article highlights the appropriateness of rapid
evaluations in supporting innovation, adaptation and systems change in a dynamic and
complex health crisis response. Findings are significant to inform practice for eval-
uators who intend to apply rapid evaluation techniques in resource and time-limited
settings.
Gawaya et al. 31
To continue reading
Request your trial