Applying the total survey error framework to PIAAC

Date03 April 2018
Published date03 April 2018
Pages153-168
DOIhttps://doi.org/10.1108/QAE-07-2017-0035
AuthorLars Lyberg,Kristen Cibelli Hibben,Beth-Ellen Pennell
Subject MatterEducation,Curriculum, instruction & assessment,Educational evaluation/assessment
Applying the total survey error
framework to PIAAC
Lars Lyberg
Inizio, Tyreso, Sweden
Kristen Cibelli Hibben
Institute for Social Research,
University of Michigan, Ann Arbor, Michigan, USA, and
Beth-Ellen Pennell
Institute for Social Research, Chelsea, Michigan, USA
Abstract
Purpose Surveys in multinational, multiregional and multicultural contexts (or 3MCsurveys) are
becoming increasingly importantto global and regional decision-making and theory building. To serve this
purpose, the surveys needto be well managed, with an awareness of key sources of survey error and howto
minimize them, mechanismsin place to control the implementation process and an abilityto intervene in that
process when necessary in a spirit of continuous improvement (Pennell et al., 2017). One key approach for
managing andassessing the quality of 3MC surveys is the total survey error (TSE) framework andassociated
survey processquality. This paper aims to examine the application of the TSE frameworkand survey process
qualityto the Programmefor the International Assessment of Adult Competencies(PIAAC).
Design/methodology/approach The authors begin with a background on TSE and discuss recent
adaptations of TSE and survey process quality for 3MC surveys. They then present a TSE framework
tailored with examples of potential contributions to error for PIAAC and ways to address those through
effectivequality assurance (QA) and quality control(QC) approaches.
Findings Overall, the authors nd that the designand implementation of the rst cycle of PIAAC largely
reect the current best practice for 3MC surveys. However, the authors identify several potential
contributions to error that may threatencomparability in PIAAC and ways these could be addressed in the
upcomingcycle.
Originality/value With a view toward continuous improvement,the nal section draws on the survey
process quality approach adapted from Hansen et al.s study (2016) to summarize the recommendations in
terms of additionalQA elements (inputs and activities) and associated QC elements(measures and reports) for
PIAACs considerationin the next cycle.
Keywords Quality control, Quality assurance, Cross-national surveys, Data quality,
Total survey error
Paper type Research paper
Introduction
Surveys in multinational, multiregional and multicultural contexts (or 3MCsurveys)
compare two or more populationsusing different types of designs as outlined by Kish (1994)
and Jowell (1998). To serve this purpose, these surveys need to be well managed, with an
awareness of key sources of survey error and how to minimize them, with mechanisms in
place to control the implementationprocess and an ability to intervene in that process when
necessary in a spirit of continuousimprovement (Pennell et al.,2017). One key approach for
managing and assessing the quality of 3MC surveys is the total survey error (TSE)
framework and the associatedsurvey process quality.
Survey error
framework
153
Received7 July 2017
Revised2 November 2017
Accepted13 December 2017
QualityAssurance in Education
Vol.26 No. 2, 2018
pp. 153-168
© Emerald Publishing Limited
0968-4883
DOI 10.1108/QAE-07-2017-0035
The current issue and full text archive of this journal is available on Emerald Insight at:
www.emeraldinsight.com/0968-4883.htm
TSE is widely accepted as the organizing framework in the design and evaluation of
single-country surveys and it is increasingly being applied to 3MC surveys. Briey, TSE
provides a theoretical frameworkfor optimizing a survey design so that the highest possible
data quality is achieved (in terms of mean squared error) within budgetary and other
constraints (Biemer, 2010). TSE is a composite of sampling (or in some models,
representation) error and measurement error. Representation errors indicate how well
survey estimates generalize to the target population, whereas measurement error indicates
how well survey questions measure the constructsof interest (Groves et al.,2009). Thus, the
TSE framework serves as a planning criterion whose purpose is to take all major error
sources into account at the design stage rather than actually estimate the TSE once the
survey results are presented,which is infeasible in most cases.
Survey process quality is an approach for managing and assessing survey quality that
acknowledges the critical effect of the survey design and implementation process on the
accuracy and quality of the end result. Fitnessfor intended use is another widely recognized
approach for managing and assessing survey quality that accounts for the degree to which
the survey data meet the needs of its users (Biemer and Lyberg, 2003). The following seven
dimensions are often used to assess the qualityof national ofcial statistics in terms of both
survey error and tness for intendeduse: comparability, relevance, accuracy, timelinessand
punctuality, accessibility,interpretability and coherence (Eurostat, 2011).In this framework,
TSE may be viewed as encompassingthe accuracy dimension.
Thus, TSE is one of the dimensions that constitute tness for intended use (tness for
purpose) as a means of measuring survey quality. The number and labels of the survey
quality dimensions tend to vary dependingon survey organization. Some of the dimensions
may obviously conict. For instance, nonresponse follow-up takes time, which will hurt
timeliness and punctuality. Therefore, tness for purpose, in other words the relative
importance among quality dimensions, varies across users, which is a problem for the
survey and the data producers. In practice,it is up to the survey organization designing and
collecting the data to nd a middle ground thatmost users can live with.
This paper examines the potentialapplication of the TSE framework and survey process
quality to the Programmefor the International Assessment of Adult Competencies (PIAAC).
PIAAC is a multinational survey that combines questionnaire-based face-to-face interviews
with direct assessments of skills. We begin with a review of TSE and then discuss recent
adaptations of TSE and survey process quality in 3MC surveys. We then present a TSE
framework tailoredwith examples of potential contributions to error for PIAAC and ways to
address these through effective quality assurance/quality control (QA/QC) approaches.
Looking ahead to the next cycle of PIAAC, we conclude with a set of recommendations for
areas of focus with a view to continuousquality improvement.
Background: total survey error and survey process quality
Total survey error
The concept of total survey error has evolved over many decades(see Groves and Lybergs
study [2010] for a discussion). TSE encompasses both representation error and
measurement error. Representation error includes coverage error, sampling error,
nonresponse error and adjustment error, which are indicators of how well survey estimates
generalize to the target population. Measurement error includes aspect of validity,
measurement error and processing error, which are indicators of how well the survey
questions measure the constructsof interest. Each of the TSE error components also include
both a bias component (systematic deviations from a target value) and a variance
component (a reection of estimate instability over conceptual replications) (Groves, 2004;
QAE
26,2
154

To continue reading

Request your trial

VLEX uses login cookies to provide you with a better browsing experience. If you click on 'Accept' or continue browsing this site we consider that you accept our cookie policy. ACCEPT