The mystery shopper: a tool to measure public service delivery?
| Author | Steve Jacob,Benjamin Biard,Nathalie Schiffino |
| Published date | 01 March 2018 |
| DOI | 10.1177/0020852315618018 |
| Date | 01 March 2018 |
| Subject Matter | Articles |
Review of
Administrative
Article
Sciences
International Review of
Administrative Sciences
2018, Vol. 84(1) 164–184
The mystery shopper: a tool to
! The Author(s) 2016
Reprints and permissions:
measure public service delivery?
sagepub.co.uk/journalsPermissions.nav
DOI: 10.1177/0020852315618018
journals.sagepub.com/home/ras
Steve Jacob
Laval University, Canada
Nathalie Schiffino
Universite´ catholique de Louvain, Belgium
Benjamin Biard
Universite´ catholique de Louvain, Belgium
Abstract
Originally the exclusive preserve of the private sector, the mystery shopper technique is
increasingly being used in the public sector. In the wake of the reforms to modernise the
state, accountability and performance-monitoring exercises are on the rise. They focus,
in particular, on service quality and user-customer satisfaction. The article makes a
twofold contribution to this topic: methodological and substantive. First of all, the
article undertakes a scoping review of the literature on the mystery shopper. This
review makes it possible to present the mystery shopper technique and its use in the
public sector. For this bibliometric study, a sample of 34 papers was analysed. Second,
the article offers a summary of the research into the mystery shopper technique, its
potential and its limitations.
Points for practitioners
This article describes the use of the mystery shopper technique in the public sector. The
areas for which mystery shopper surveys are commissioned are relatively limited, most
being undertaken in the health sector. However, the scoping review emphasises the
potential importance of mystery shopping for the purposes of the evaluation. As such,
investigating other areas can be very interesting and promising for the public authorities.
We also observe from this literature review that the challenges identified during mys-
tery shopper studies can be overcome.
Keywords
accountability, ethics, evaluation, mystery shopper, mystery study, quality
Corresponding author:
Steve Jacob, Universite´ Laval, Faculte´ des sciences sociales, De´partement de science politique (local 4443),
1030, avenue des Sciences Humaines, Que´bec G1V 0A6, Canada.
Email: steve.jacob@pol.ulaval.ca
Jacob et al.
165
Introduction
In the wake of the reforms to modernise the state, inspired by New Public
Management (NPM), accountability and performance-monitoring exercises are
on the rise. They focus, in particular, on service quality and user-customer satis-
faction (Ferlie et al., 1996). We are therefore witnessing a transformation of the
relationship between the beneficiaries and producers of public services that is
inspired by consumerism (Clarke, 2007). Use of the terms ‘user’, ‘customer’, ‘con-
sumer’ or combinations thereof (e.g. ‘user-customer’) is becoming more wide-
spread. These lexical changes are not only anecdotal, but bear witness to a ‘well
advanced process of revolution of the foundations of public action, whose implicit
model is now the customer’ (Chauvie`re, 2006: 93, own translation). This results in
the implementation of more personalised tangible and intangible public service
production methods, geared towards the needs and satisfaction of beneficiaries
whose competences are sometimes mobilised to ‘co-produce’ services according
to an accountability and capacity-building rationale. This customer-centric orien-
tation poses several challenges for public governance. To touch on only the most
common, these include: the difficulty or impossibility of reducing a public service to
a commercial service and of putting a price on it or of giving users a choice; the lack
of alternatives or lack of accessible and relevant information; and the customer’s
legitimacy as an interlocutor in public services.
The use of evaluation has increased concomitantly with the adoption of reforms
inspired by the school of thought of NPM (Jacob, 2009; Jacob and Varone, 2004;
Jacob et al., 2015). The literature on the evaluation of public policies and pro-
grammes grants special attention to observation, which is presented as a means of
collecting information, alongside testimonials, questionnaires, interviews or check-
lists (Widmer and Binder, 1998).
Evaluators who wish to make use of observation take recourse to the
approaches conventionally used in the social sciences or resort to new approaches
derived from practices used in the private sector, in particular, that of the mystery
shopper (MS).1 Initially developed in the private sector, the MS method fulfils two
roles in the public sector: evaluation of service quality; and evaluation of public
interventions. This twofold purpose of the evaluation reinforces the interest in this
method for public administration researchers. However, although many research-
ers are familiar with the method, defend it, use it and tend to promote it, as we will
see in the pages that follow, it is clear that very little of the theoretical literature on
public policy and/or their evaluation covers this technique. One exception is the
manual published by Wholey et al. (2004), where an entire chapter is dedicated to
this method (Turner and Zimmermann, 2004: 310–339). However, the term ‘mys-
tery shopper’ is not used, being replaced by that of ‘role play’.
Drawing on the preceding, this article sets out to take stock of the current state
of the MS technique and its practice in the public sector by means of a scoping
review. This form of literature review provides an overview of the available scien-
tific knowledge in a particular domain and helps outline the object under scrutiny
(Jacob and Schiffino, 2015; Levac et al., 2010). To glean information on the
166
International Review of Administrative Sciences 84(1)
innovative practice of the MS in the public sector, we have selected theoretical and
practical contributions relating to it.
Our basic body of research is composed of scientific papers identified in seven
databases containing the major journals in the field of public administration
(Worldwide Political Science Abstract; International Bibliography of the Social
Sciences; Erudit; Spire; Web of Science; Francis; Pais International), using key-
words
(‘Mystery
Shopper’,
‘Mystery
Visit’,
‘Public’,
‘Private’,
‘Ethics’,
‘Evaluation’, ‘Government’ and ‘State’). We have added papers cited in the biblio-
graphies of the papers found in the databases.
The sample includes 62 papers divided into two categories: theoretical (present-
ing the concepts and methods generically) and empirical (describing at least one
case of evaluation in the field). The first category represents 45% of our sample and
the second represents 55%. These 55% (i.e. the 34 articles presented in Table 1) can
be broken down into evaluations conducted to measure the quality of a service
(65%), evaluations conducted to measure/monitor compliance with legal provi-
sions (32%) or evaluations combining these two goals (3%).
The MS technique: implementation
The MS technique involves putting a person in contact (e.g. visit, telephone call)
with an organisation to evaluate the quality of service provided by the latter. By
focusing on ‘the ability of local units to meet the requirements of an ‘‘extraordin-
ary’’ customer, it makes it possible to evaluate the quality of service offered and the
capacity of local units to make the offering and production more flexible’
(Jougleux, 2006: 14, own translation).
Given the popularity of the use of this technique in private and public organ-
isations, specialised agencies offer their expertise in this area (Calvert, 2005).
Mystery shoppers are central and essential actors of this method of evaluation.
They are recruited and motivated in different ways. The literature distinguishes
between intrinsic and extrinsic motivations. The first motivations result from the
activity as such, and the satisfaction linked to participation is about pleasure or
excitement, for example. As regards extrinsic motivations, monetary compensation
and personal experience are the dominating factors. These motivations are not
mutually exclusive and can overlap (Allison and Severt, 2012; Allison et al.,
2010). The cost of recruiting and training mystery customers is relatively high.
This induces a limitation in the number of contacts with the evaluated organisa-
tions, which does not always make it possible to observe any evolutions (Ford
et al., 2011).
To conduct the evaluation, several conditions must be met. Mystery shopping
must preserve the ethical duty of confidentiality of the customer. Next, a specific
scenario must be written and scrupulously followed by the customer during their
mission. This scenario is carefully prepared to ensure its quality and credibility.
Finally, the evaluation – using the same criterion-referenced grid for all organisa-
tions studied in the same survey – must take place immediately after the mission.
s
o
(continued)
Prior
information
Ye
No
NA
No
N
NA
r
e
1
1
002
2
y
y
–
ne
u
ch
2005
September–
20
Novemb
2011
2010
Januar
2002
Mar
uration
1
5
A
D
April–Ma
2
April–J
2
N
NA
of
Number
interactions
per
establishment
2
1
NA
1
1
NA
f
o
tal
To
number
interactions
292
1098
NA
315
100
NA
valuation.
e
elephone
Methods
used
Visit
Visit
Visit
T
Visit
Visit
es
action
ice/
of
chiv
y
ar
serv
public
stal
theor
harmacology
harmacology
o
Domain
study
P
Pharmacology
Public
P
Pharmacology
P
of
f
logic
o
y
a
ealand
ritain
B
en
Z
t
in
object
w
Location
the
German
Swed
Ne
USA
Grea
France
-
-
-
-
-
w/
valu-
vice
ser
ser
ser
ser
ser
method
of
la
e
+
ser
f
f
f
f
f
f
o
o
o
o
o
the
ol
o
quality
quality
quality
quality
quality
MS...
To continue reading
Request your trial