Paradata and dashboards in PIAAC
| DOI | https://doi.org/10.1108/QAE-06-2017-0031 |
| Published date | 03 April 2018 |
| Date | 03 April 2018 |
| Pages | 263-277 |
| Author | Leyla Mohadjer,Brad Edwards |
| Subject Matter | Education,Curriculum, instruction & assessment,Educational evaluation/assessment |
Paradata and dashboards
in PIAAC
Leyla Mohadjer
Statistics, Westat Inc, Rockville, Maryland, USA, and
Brad Edwards
Westat Inc, Rockville, Maryland, USA
Abstract
Purpose –This paper aims to provides a brief review of the dashboard literature, an account of the
development of performancedashboards for field data collection at Westat, and more specifically for the first
cycle of the Programme for the InternationalAssessment of Adult Competencies (PIAAC). A brief concluding
sectionoffers suggestions for improvements in surveydashboards for PIAAC’s next cycle.
Design/methodology/approach –To manage field work at Westat, the authors create views into
various paradata databases and present them in a dashboard, showing key performance indicators at a
glance. Users can drilldown from the dashboard into underlying databases to investigatepotential problems.
The US PIAAC dashboardis a monitoring system that supports daily review of manyactivities. For example,
it provides overnight alerts to the field supervisor when global positioning system (GPS) data from an
interviewer’ssmartphone shows the interview occurred far from the respondent’shome.
Findings –Performance dashboards may represent best practice for monitoring field activities. Paradata
sources and systemsvary greatly across the PIAAC countries, but a multitude of process data exists in every
country and can be used to create quality indicators and a monitoring system. PIAAC can establish
standards/guidelinesto improve visualization of qualitymetrics and management data, regardless of the local
survey infrastructure.
Originality/value –The core of the paper is a case study of the experiences on the US PIAAC
implementation of dashboardsto monitor survey quality, productionand costs, with special attention to the
issue of fabrication.
Keywords Paradata, Data quality, Interviewer effects, Data falsification, Interviewer error,
Performance dashboards
Paper type Research paper
Overview
This paper provides a general introductionto survey dashboards, followed by an account of
the development of performance dashboards for field data collection at Westat. The core
of the paper is a case study of our experiences with the US implementationof the first cycle
of the Programme for the International Assessment of Adult Competencies (PIAAC), with a
focus on Round 3 and its use of dashboards to monitor survey quality, productionand costs.
We give special attention to the issue of fabrication. A brief concluding section offers
thoughts about directions for futureresearch on survey dashboards and offers suggestions
for PIAAC’s next cycle.
Application of dashboards in survey operations
Why use a dashboard?
An automobile dashboard enables a driver to keep movingwhile checking critical systems.
Without these dashboards, we would increase the risk of running out of gas, going too fast
Paradata and
dashboards in
PIAAC
263
Received30 June 2017
Revised26 October 2017
Accepted14 December 2017
QualityAssurance in Education
Vol.26 No. 2, 2018
pp. 263-277
© Emerald Publishing Limited
0968-4883
DOI 10.1108/QAE-06-2017-0031
The current issue and full text archive of this journal is available on Emerald Insight at:
www.emeraldinsight.com/0968-4883.htm
To continue reading
Request your trial