Guest editorial
| Published date | 03 April 2018 |
| DOI | https://doi.org/10.1108/QAE-01-2018-0010 |
| Pages | 150-152 |
| Date | 03 April 2018 |
| Author | Irwin S. Kirsch,William Thorn,Matthias von Davier |
| Subject Matter | Education,Curriculum, instruction & assessment,Educational evaluation/assessment |
Guest editorial
Foreword
Thepapersinthisspecialissuearebasedonpresentations at a two-day international seminar
on managing the quality of data collection in large-scale assessments. The seminar was held on
May 11-12, 2017, at the Organisation for Economic Co-operation and Development (OECD)
headquarters in Paris. The purpose of this event was to bring together psychometricians and
survey methodologists to discuss issues around the identification, treatment and prevention of
errors associated with data collections in large-scale assessments, as well as prospects for the
evolution of data collection methods. This was the firstofaplannedseriesofseminarson
methodological issues relevant to the Programme for the International Assessment of Adult
Competencies (PIAAC) and other internationallarge-scaleassessmentssuchastheOECD’s
Programme for International Student Assessment (PISA).
The topic of managing quality in data collection was chosen for two main reasons: first,
data collection and field operations represent major sources of potential error in any large-
scale survey, particularly those such as PIAAC that administer questionnaires and tests of
cognitive skills (literacy, numeracy and problem-solving) using interviewer-based methods.
The behavior and skills of interviewers,the setting and conditions in which the interview or
assessment takes place, the dispositions and motivation of respondents and the capability
and capacity of survey organizations collecting the data all have effects on data quality.
These effects range from data falsificationat the different levels of survey operations at one
extreme to satisficing(Krosnick, 1991) by both interviewers and respondentsat the other.
Second, developments in information and communications technologies offer an
opportunity to achieve considerable improvements in data quality through the
identification, treatment and prevention of errors. New technologies are also opening up
potential new avenues for data collection. The use of computer-aided personal interviewing
and computer-based testing has already led to demonstrable improvements in data quality
in large-scale survey assessments and other testing programs. Automatic scoring and
automatic range checks for responses reduce the chance for human error, for example. The
availability of process data that represent the interactions between interviewers and
respondents and the interview/testing application, including timestamps, provides a rich
source of informationfor detecting problems and potentially adjustingfor them.
This seminar on managing the quality of data collection in international large-scale
assessments was organized into five sessions, each including two presentations. The sessions
were preceded by an introduction and overview presentation focusing on the variability and
potential sources of errors in comparative surveys. This overview discussed the concept of total
survey error (TSE; Groves and Lyberg, 2010), first introduced by Hansen et al. (1951,1953), as a
framework for identifying and addressing various sources of non-sampling variance.
The guest editors would like to thank the authors of the papers in this special issue as well as the
reviewers who participated in the blind review at the penultimate stage of the editorial process. The
authors acknowledge the assistance of Sabrina Leonarduzzi who organized the seminar, the work of
Larry Hanover who managed the editorial process and the effort of Mary Louise Lennon who
undertook copy editing of the articles.
The guest editors would also like to acknowledge the role of Madhabi Chatterji, Professor of
Measurement, Evaluation and Education at Teachers College, Columbia University, and Co-Editor of
Quality Assurance in Education, without whose interest and support this special issue would not have
been possible.
QAE
26,2
150
QualityAssurance in Education
Vol.26 No. 2, 2018
pp. 150-152
© Emerald Publishing Limited
0968-4883
DOI 10.1108/QAE-01-2018-0010
To continue reading
Request your trial