Question order bias revisited: A split‐ballot experiment on satisfaction with public services among experienced and professional users
| Published date | 01 March 2021 |
| Author | Mads Thau,Maria Falk Mikkelsen,Morten Hjortskov,Mogens Jin Pedersen |
| Date | 01 March 2021 |
| DOI | http://doi.org/10.1111/padm.12688 |
ORIGINAL ARTICLE
Question order bias revisited: A split-ballot
experiment on satisfaction with public services
among experienced and professional users
Mads Thau
1
| Maria Falk Mikkelsen
1
| Morten Hjortskov
2
|
Mogens Jin Pedersen
3
1
VIVE—The Danish Center for Social Science
Research, Aarhus and Copenhagen, Denmark
2
Department of Political Science, Aarhus
University, Aarhus, Denmark
3
Department of Political Science, University
of Copenhagen, Copenhagen, Denmark
Correspondence
Mads Thau, VIVE—The Danish Center for
Social Science Research, Herluf Trolles Gade
11, Copenhagen 1052, Denmark.
Email: math@vive.dk
Funding information
The data collection was supported by the
National Board of Social Services in Denmark.
Abstract
Public decision-makers increasingly rely on satisfaction sur-
veys to inform budget and policy decisions. Yet, our knowl-
edge of whether, and under what conditions, this input
from public service users provides valid performance infor-
mation remains incomplete. Using a preregistered split-
ballot experiment among government grant recipients in
Denmark, this article shows that the ordering of survey
questions can bias satisfaction measures even for highly
experienced and professional respondents. We find that
asking about overall satisfaction before any specific service
ratings lowers overall user satisfaction, compared to the
reverse order, while the correlations between specific rat-
ings and overall satisfaction are relatively stable. Also, the
question order effect outweighs that of a large-scale
embezzlement scandal, which unexpectedly hit the investi-
gated government agency during the data collection. Our
results support rising concerns that subjective performance
indicators are susceptible to bias. We discuss how practi-
tioners should approach satisfaction surveys to account for
the risk of question order bias.
1|INTRODUCTION
How satisfied are you with the overall state of public administration research? Imagine that you were asked this
question either before or after another set of questions asking you to rate the latest public administration study that
you had read. Would the order in which the questions were asked affect your response?
Received: 5 November 2019 Revised: 20 April 2020 Accepted: 22 June 2020
DOI: 10.1111/padm.12688
Public Admin. 2021;99:189–204. wileyonlinelibrary.com/journal/padm © 2020 John Wiley & Sons Ltd 189
If you are like the average individual, the answer is most likely yes. For years, survey methodologists have
emphasized the issue of question order bias in survey settings (e.g., McFarland 1981). A long line of research shows
that respondents answer questions differently based on where they appear in a given questionnaire (Tourangeau
et al. 2000; Auh et al. 2003; Kaplan et al. 2013; Stark et al. 2018).
This insight has important implications for both public administration research and public decision-makers. The
performance of public services is increasingly evaluated using surveys of user satisfaction, and the survey results are
widely used to motivate decisions ranging from budget allocations to policy reforms and organizational changes
(James 2009; Miller et al. 2009; Andersen and Hjortskov 2016; Song and Meier 2018). As Van de Walle and Van
Ryzin (2011, p. 1436) have noted, even ‘minor changes in levels of citizen satisfaction with public services are often
taken at face value by policy-makers and can become the bases for important decisions about service budgets and
service delivery’. This raises a concern that public decision-makers might sometimes act on changes in user satisfac-
tion that are not substantial, but rather artificial methods effects, leading to suboptimal or illegitimate decisions
(Hjortskov 2017, p. 913).
This article provides new evidence on the size and direction of question order bias in surveys on user satisfac-
tion with public services. Despite the understanding that question order matters, the issue has received little atten-
tion in public administration research on satisfaction. An important exception is Van de Walle and Van Ryzin (2011).
Using a nonprobability sample of US citizens, this study found significant question order effects in satisfaction with
the local public services. Yet, contrary to expectations and prior studies on question order bias in other contexts, the
results suggested that asking about specific service ratings before overall satisfaction (as opposed to after) lowered
overall satisfaction with local government. This led Van de Walle and Van Ryzin (2011, p. 1447) to conclude that
‘more experimental work should be done, with particular attention to question order, before accepting the assump-
tion that citizens have a more positive outlook on specific services than on the performance of government in
general’.
Heeding this call, our aim is to examine if the findings from Van de Walle and Van Ryzin (2011) hold up in
another context. Specifically, we test for question order bias in a user satisfaction survey carried out on behalf of a
large national government agency in Denmark called the National Board of Social Services (NBSS). Our sample com-
prises 460 individuals, who represent non-profit or public organizations that received government funding for social
work projects in 2017 and 2018. These individuals all had recent, personal, and often extensive experience with the
NBSS, and they answered the survey as professionals rather than as private participants. The respondents should
therefore be particularly likely to hold stable and well-informed attitudes about the NBSS (Zaller 1992), providing a
least-likely case in which to observe biased responses. Compared to the general citizen sample in Van de Walle and
Van Ryzin (2011), a targeted user sample allows for a harder test of the question order bias hypothesis.
Our survey measured grant recipients' overall satisfaction with the NBSS services as well as their ratings of a
series of specific agency services relating to the application process and various administrative tasks associated with
receiving and spending the grant. We embedded a preregistered split-ballot experiment that randomly assigned
respondents to different survey versions: (1) one that asked about overall satisfaction before asking about specific
services, (2) and another that asked about overall satisfaction after having probed specific service ratings.
Our results support that question order bias occurs in satisfaction surveys, but also qualify recent findings. Con-
trary to Van de Walle and Van Ryzin (2011), we find that overall satisfaction is higher if specific service ratings come
first, and that the strongest predictors of overall satisfaction do not significantly differ across treatments. One likely
explanation for the divergent results is that our survey asked about various aspects of the same public service,
whereas the Van de Walle and Van Ryzin study asked about a list of distinct public services. As discussed later, this
difference between studies might have triggered a so-called assimilation effect in our case and a contrast effect in
the Van de Walle and Van Ryzin case (Tourangeau et al. 2000).
Still, our main finding is that question order matters. Even in a high-expertise sample, we find an effect that is
similar in size to previous estimates. This suggests that question order bias in satisfaction measures is more universal
than is sometimes assumed (see also Stark et al. 2018). Our experimental results gained additional and unanticipated
190 THAU ET AL.
Get this document and AI-powered insights with a free trial of vLex and Vincent AI
Get Started for FreeStart Your Free Trial of vLex and Vincent AI, Your Precision-Engineered Legal Assistant
-
Access comprehensive legal content with no limitations across vLex's unparalleled global legal database
-
Build stronger arguments with verified citations and CERT citator that tracks case history and precedential strength
-
Transform your legal research from hours to minutes with Vincent AI's intelligent search and analysis capabilities
-
Elevate your practice by focusing your expertise where it matters most while Vincent handles the heavy lifting
Start Your Free Trial of vLex and Vincent AI, Your Precision-Engineered Legal Assistant
-
Access comprehensive legal content with no limitations across vLex's unparalleled global legal database
-
Build stronger arguments with verified citations and CERT citator that tracks case history and precedential strength
-
Transform your legal research from hours to minutes with Vincent AI's intelligent search and analysis capabilities
-
Elevate your practice by focusing your expertise where it matters most while Vincent handles the heavy lifting
Start Your Free Trial of vLex and Vincent AI, Your Precision-Engineered Legal Assistant
-
Access comprehensive legal content with no limitations across vLex's unparalleled global legal database
-
Build stronger arguments with verified citations and CERT citator that tracks case history and precedential strength
-
Transform your legal research from hours to minutes with Vincent AI's intelligent search and analysis capabilities
-
Elevate your practice by focusing your expertise where it matters most while Vincent handles the heavy lifting
Start Your Free Trial of vLex and Vincent AI, Your Precision-Engineered Legal Assistant
-
Access comprehensive legal content with no limitations across vLex's unparalleled global legal database
-
Build stronger arguments with verified citations and CERT citator that tracks case history and precedential strength
-
Transform your legal research from hours to minutes with Vincent AI's intelligent search and analysis capabilities
-
Elevate your practice by focusing your expertise where it matters most while Vincent handles the heavy lifting
Start Your Free Trial of vLex and Vincent AI, Your Precision-Engineered Legal Assistant
-
Access comprehensive legal content with no limitations across vLex's unparalleled global legal database
-
Build stronger arguments with verified citations and CERT citator that tracks case history and precedential strength
-
Transform your legal research from hours to minutes with Vincent AI's intelligent search and analysis capabilities
-
Elevate your practice by focusing your expertise where it matters most while Vincent handles the heavy lifting
