Interviewer variance and its effects on estimates

Published date03 April 2018
DOIhttps://doi.org/10.1108/QAE-06-2017-0030
Date03 April 2018
Pages227-242
AuthorGeert Loosveldt,Celine Wuyts,Koen Beullens
Subject MatterEducation,Curriculum, instruction & assessment,Educational evaluation/assessment
Interviewer variance and its
eects on estimates
Geert Loosveldt,Celine Wuyts and Koen Beullens
Department of Sociology (Survey Methodology Research Unit), KU Leuven,
Leuven, Belgium
Abstract
Purpose In survey methodology, it is well-knownthat interviewers can have an impact onthe registered
answers. This paperaims to focus on one type of interviewer effectthat arises from the differences between
interviewersin the systematic effects of eachinterviewer on the answers. In the rstcase, the authors evaluate
interviewer effects on the measurement of alcohol consumption in European countries. The second case is
about the interviewer effects on the respondentstendency to (non)differentiate their responses and the
consequencesof this response stylefor the correlation between variables.
Design/methodology/approach The interviewer effects are evaluated by means of interviewer
variance analysis. Because respondents are nested within interviewers,we can specify a two- or three-level
random intercept model to calculatethe proportion of variance explained by the interviewers.Data from the
seventhround of the European Social Survey are used.
Findings The results in therst case show that the substantive conclusions about theeffect of gender and
education on the alcohol measures continue to hold when interviewer effects are taken into account. The
results of the second case make clear thatinterviewer effects on attitudinal questions are considerable. There
is also a signicant effect of the interviewers on the degree that respondents differentiate their responses.
The results alsoillustrate that correlations between attitudinalvariables are inuenced. This also implies that
the results of statisticalprocedures using a correlation or covariancematrix can be strongly inuenced by the
tendencyto (non)differentiateand the interviewersimpact on this tendency.
Originality/value The results clearly demonstrate that there are considerable differences between countries
concerning the impact of the interviewers on substantive variables. Cross-national differences are striking and the
importance and necessity to evaluate interviewer effects in a cross-national survey becomes clear.
Keywords Cross-national survey, Interviewer effects, Alcohol measures, Nondifferentiation,
Response style
Paper type Research paper
Introduction
In survey methodology, it is well-known that interviewers can have an impact on the registered
answers of respondents in different ways. They may give insufcient or incorrect instructions
on how to answer the questions, clarify questions in a wrong way, ask questions or probe in a
suggestive way, selectively interpret answers and so on. This kind of inappropriate interviewer
behavior can cause systematic or variable measurement errors. Bias occurs when all the
interviewers have the same impact in the same direction on the answers. Variable errors differ
within and between interviewers and create additional noise in the data (Loosveldt, 2008;
Biemer and Lyberg, 2003). Both types of interviewer error can seriously inuence the data
quality. Therefore, the evaluation of interviewer effects in telephone or face-to-face surveys is
an essential and important aspect of survey data quality assessment.
Two approaches can be applied to evaluate interviewer effects. The rst focuses on the
interaction between interviewerand respondent. The unit of analysis in interaction analysis
is a question-and-answer sequence for one single question. For each sequence, all
interviewersand respondentsutterances are coded (Ongena, 2005). In this approach, the
Interviewer
variance
227
Received30 June 2017
Revised18 October 2017
Accepted18 December 2017
QualityAssurance in Education
Vol.26 No. 2, 2018
pp. 227-242
© Emerald Publishing Limited
0968-4883
DOI 10.1108/QAE-06-2017-0030
The current issue and full text archive of this journal is available on Emerald Insight at:
www.emeraldinsight.com/0968-4883.htm
focus is on deviances from the paradigmatic(Schaefer and Maynard, 1996) sequence in
which the interviewer poses the question, as in the questionnaire,and the respondent reacts
to this prompt with an adequateanswer.
The second approach makes use of an analysis of how the total variance can be decomposed
into interviewer effects and other sources. In the basic analysis model of this type, the
interviewer is used as a random independent variable to create groups of respondents at the
second level and a substantive variable is used as the dependent variable. The analysis results in
an estimate of the proportion of variance in the substantive variable explained by the
interviewers. This is the so-called interviewer variance or intra-class correlation coefcient
(Loosveldt, 2008;Beullens and Loosveldt, 2016). Interviewer differences are not intended to
explain variance in substantive variables, and a high interviewer variance in such an analysis is
considered to be an indication of inferior data quality. Notice that interviewer variance analysis
can only estimate one type of interviewer effect. This type arises from the differences between
interviewers in the systematic effects of each interviewer on the answers (Loosveldt, 2008).
In this paper, we elaborateand illustrate the evaluation of interviewer effects by meansof
interviewer variance analysis. First, we present the basic multilevel model to estimate
interviewer variance and discuss the consequences for substantial analysis. Next, two
examples are presented. In the rst, we analyze the interviewer effects on the measurement
of alcohol consumption in European countries and the impact of the interviewer effects on
the assessment of the relationship between alcohol consumption and gender as well as
education levels. In the second case, interviewer effects are estimated for attitudinal
questions. There, we examine the impact of interviewers on the respondentstendency to
(non)differentiate their responses and whether this kind of interviewereffect has an impact
on the correlations betweenother attitudinal variables.
The basic model
In this section, we present the basic multilevel models we will use in the elaboration of the
two cases in the next sections. Multilevel models are particularly suitable to analyze data
with a hierarchical structure. Respondentsnested within interviewers are a typical example
of a two-level hierarchical data structure. The model treats the respondents as the rst
(lower) level and the interviewersas the second (higher).
The simplest model to evaluate interviewereffects is a two-level random intercept model
with no independentvariables. The model can be formally written as:
Yij ¼
b
0jþ
«
ij
b
0j¼
g
00 þ
m
0j
We can integrate both expressionsinto one equation:
Yij ¼
g
00 þ
m
0jþ
«
ij
In this model, Y
ij
is the value of variable Yfor the i-th respondent interviewed by the j-th
interviewer,
b
0j
is the intercept for interviewer jand
«
ij
is the residual error term for each
respondent interviewed by interviewer j. This intercept for interviewer jcanbe divided into
axed (overall) intercept
g
00
and an interviewer-specic residual errorterm
m
0j
. A normal
distribution for the error terms is assumed. In these distributions, the mean is zero, and the
variance at the respondent level and the interviewer-related error terms are respectively
equal to
s
2
eand
s
2
u. There are signicant differences betweenthe respondent groups nested
within interviewers when
s
2
udiffers signicantly from zero. The interviewereffect for each
variable in the analysisis estimated by means of the intra-interviewer correlation:
QAE
26,2
228

To continue reading

Request your trial

VLEX uses login cookies to provide you with a better browsing experience. If you click on 'Accept' or continue browsing this site we consider that you accept our cookie policy. ACCEPT