Time To Reset The Clock On The Design Of Impact Evaluations In Criminology: The Case For Multi-Methodology Designs

AuthorStephen Morris, Andrew Smith, Chris Fox
Pages28-45
British Journal of Community Justice
©2020 Manchester Metropolitan University
ISSN 1475-0279
Vol. 16 (2) 28-45
TIME TO RESET THE CLOCK ON THE DESIGN OF
IMPACT EVALUATIONS IN CRIMINOLOGY: THE
CASE FOR MULTI-METHODOLOGY DESIGNS
Stephen Morris, Andrew Smith and Chris Fox, Manchester Metropolitan University
Abstract
This paper highlights how qualitative research can enhance causal explanation in impact
evaluations and provide additional causal leverage to findings from randomised
experiments. We assess the extent to which randomised studies in criminology adopt mixed
or multi-methodological approaches as seen in other fields such as health care, education
and international development. We reviewed current practice in the design of experimental
evaluations within criminology. Structured searched terms previously used to identify
qualitative research components within randomised studies in health research, were used
to search for evidence of mixed method design in 46 primary studies involving
randomisation, published in four leading journals in criminology since 2013. Although such
mixed-method randomised studies are increasingly seen in other fields such as health,
education and international development, among the studies we identified in criminology
and criminal justice our review reveals almost an entire absence of designs in which
qualitative research is formally and explicitly integrated into study designs. We argue that
randomised studies are significantly enhanced through incorporating explicit and planned
mixed-method elements, and particularly qualitative research. We suggest reasons for this
absence and what might be done to address it.
Keywords
Randomised designs, criminology, mixed methods, qualitative research, evaluation
Introduction
In recent years there has been a welcome growth in intervention studies in criminology that
adopt randomised designs. Evidence from these studies is playing an important role in
advancing knowledge of ‘what works’ (Sullivan & Welsh, 2017). The US National Institute of
Justice for example, through its Crime Solutions portal, provides details of nearly 100
programmes that have been deemed ‘effective’ as a result of evidence from randomised
studies1. In Europe, the growth in the use of randomised experiments in criminology has
1 https://www.crimesolutions.gov/Programs.aspx (accessed 16th April,2018)
28
Time to reset the clock on the design of impact evaluations in criminology
been slower but arguments in their favour no less compelling (Andersen & Hyatt, 2018).
Between 1982 and 2004, Farrington & Welsh (2005) found that of 83 randomised trials in
criminology worldwide, two were undertaken in Europe with not a single study found in the
UK. By 2018, the UK centre for ‘What Works in Crime Reduction’ recorded some 24
randomised evaluations since 2009 of which 11 were ongoing2. Andersen & Hyatt (2018) in
their survey of experiments published in the Journal of Experimental Criminology since 2010
(n=64) find that 19 per cent (n=12) were conducted in Europe. They also note that 8 per
cent of the panel presentations at the 2014 Conference of the European Society of
Criminology involved a study with a randomised design (Andersen & Hyatt, 2018, 5).
We argue that the growth in randomised intervention studies in criminology is a very
welcome development (Farrington, 1983, 2003; Weisburd, Lum, & Petrosino, 2001) and
that the many examples of successful randomised studies in criminology attest to the
previously perceived practical and ethical barriers being readily surmountable. However,
the central contention of this paper, i s that usefulness and insights from randomised
intervention studies are significantly enhanced through incorporating explicit and planned
mixed method elements (Campbell, 1987), and particularly qualitative research. We
provide evidence, however, that such mixed method intervention study designs are rare in
criminology and highlight how qualitative research can promote better causal explanation.
In part this paper is motivated by experience in other fields. In international development,
for example, what are variously termed mixed-method randomised controlled trials or RCT+
designs have been discussed and implemented (Bamberger, Tarsilla, & Hesse-Biber, 2016;
White, 2013). In health research there has been a long tradition of promoting mixed method
intervention studies (Boeije, Drabble, & O’ Cathain, 2015; Hansen & Jones, 2017; R. B.
Johnson & Schoonenboom, 2016; Moore et al., 2015; Oakley, Strange, Bonell, Allen, &
Stephenson, 2006). In education, the growing use of randomised designs has been
accompanied by an increased emphasis on studies that combine randomisation with mixed
method implementation process evaluation (Humphrey et al., 2016; Lendrum & Humphrey,
2012). Whilst there are calls consistent with such designs in criminology (see discussions
around the EMMIE framework, Johnson, Tilley, & Bowers, (2015)), we will show that there
is a dearth of such mixed method designs in the academic literature, and this in our view
reveals a less well developed and possibly more polarised debate around mixed methods in
the sector.
First, this paper outlines the role qualitative research can play in causal inference. This
discussion is addressed to those from a quantitative background with a more sceptical
position toward what qualitative might offer. In opposition to the notion of a fundamental
incompatibility between qualitative and quantitative research, we outline the different
ways researchers have thought about how qualitative research m ight be integrated with
randomised designs and acknowledge the importance of philosophical pragmatism as
providing an epistemological unpinning for this task. We further advance our discussion by
providing three examples (among the many we could have called upon) from education,
health and international development of the successful integration of qualitative research
2 Site access on 1st February, 2018 see
http://whatworks.college.police.uk/Research/Research-Map/Pages/Research-Map.aspx
29

To continue reading

Request your trial

VLEX uses login cookies to provide you with a better browsing experience. If you click on 'Accept' or continue browsing this site we consider that you accept our cookie policy. ACCEPT