Does Repeated Measurement Improve Income Data Quality?
| DOI | http://doi.org/10.1111/obes.12296 |
| Published date | 01 October 2019 |
| Date | 01 October 2019 |
| Author | Paul Fisher |
989
©2019 TheAuthors. OxfordBulletin of Economics and Statistics published by Oxford University and John Wiley & Sons Ltd.
Thisis an open access article under the ter ms of the CreativeCommons Attribution License, which permits use, distribution and reproduction in any medium, provided
the original work is properlycited.
OXFORD BULLETIN OF ECONOMICSAND STATISTICS, 81, 5 (2019) 0305–9049
doi: 10.1111/obes.12296
Does Repeated Measurement Improve Income Data
Quality?*
Paul Fisher†
†Institute for Social and Economic Research, University of Essex, Wivenhoe Park,
Colchester, Essex, CO4 3SQ, UK (e-mail: pfishe@essex.ac.uk)
Abstract
This paper exploits a natural experiment created by a survey design to showthat the quality
of income data systematically changes across waves of a panel.We estimate that the effect
of being interviewed for a second time, relative to the first, is to increase mean monthly
income by 8%. Dependent interviewing – a recall device commonly used in panel surveys
– explains one third of the observed increase. The remaining share is attributed to changes
in respondent behaviour (panel conditioning). We review the evidence for and against a
reporting improvement vs. a behavioural response by survey participants.
I. Introduction
An extensive body of work in economics is underpinned by household survey data on
incomes. Recent examples include: Chetty et al. (2017) [Current Population Survey (CPS)],
Hoynes, Schanzenbach and Almond (2016) (Panel Survey of Income Dynamics), Yang
(2017) (Survey of Income and Program Participation), and Blundell et al. (2016) (British
Household Panel Survey). Distinctly, smaller scale household panel surveys are regularly
used to analyse randomized trials and relevant income variables. Recent examples include:
Haushofer and Shapiro (2016), Banerjee et al. (2015) and Karlan and Zinman (2011).
Government statistics on the income distribution are commonly based on large household
surveys, for example, the official US poverty rate is based upon the CPS. Official income
statistics may also relyon panel data, where they monitor changes in income over a lifetime.
Examples include official statistics of the European Union (European Union Income and
Living Conditions survey); and the UK Government (UK Household Longitudinal Study).
JEL Classification numbers: C83, D31, I32
*Thanks to Tom Crossley for reading drafts of this article and providing insightful feedback. I thank Jonathan
Burton; Mike Brewer; Nick Buck; Brian Bucks; Emanuele Ciani; Annette Jackle; PeterLynn and Steve Pudney for
their useful comments. I also thank seminar participants at the ‘Improving the measurement of household finances’
workshop, University of Essex; and the International Association for Research in Income and Wealth conference
2016, Dresden, Germany.Financial support received from the ESRC is also gratefully acknowledged, awardnumbers
ES/K005146/1, ES/N00812X/1 and ES/N006534/1. Understanding Society is an initiative funded by the Economic
and Social Research Council and various Government Departments, with scientific leadership by the Institute for
Social and Economic Research, University of Essex, and survey delivery by NatCen Social Research and Kantar
Public. The research data are distributed by the UK Data Service.
990 Bulletin
This paper exploits a natural experiment created by a survey design to study how
measurement error in income evolveswith repeated inter viewingin a panel sur vey. Previous
research has shown that state transfers and self-employment income are under-reported in
household surveys (Meyer and Sullivan, 2003, 2011; Lynn et al., 2012; Hurst, Li and
Pugsley, 2014; Meyer and Mittag, 2015; Brewer, Etheridge and O’Dea, 2017), but there
is little evidence on whether measurement error is on average stable across waves of a
panel. We find that the quality of measured income systematically changes across the early
waves of a leading panel survey. If the changes represent reporting improvements, then this
suggests a major benefit of repeated interviewing. Irrespectively, estimates of distributional
change based on the early waves of the panel will confound true changes with data quality
changes and be biased.
Panel experience mayaffect a given respondent’sincome report, for a given year, for two
reasons. First, panel conditioning (PC) effects mayoperate where panel participants change
their behaviour (reporting or economic) as a result of being part of the panel. PC improves
data quality if it reflects a respondents improved understanding of the questionnaire content
or a growing trust in the interviewer or data holders. PC reduces data quality if respondents
learn to strategically answer questions with the aim of reducing the interview length.
Related, the data will become unrepresentative if survey participation leads to changes in
real behaviour (Das and Leino, 2011; Zwane et al., 2011; Crossley et al., 2017; Bach and
Eckman, 2018). Second, dependent interviewing (DI) – a tool that survey respondents of
their reports at the previous interview – will lead to differences in data quality between the
baseline and subsequent interviews where it takes effect.
Few studies have examined the stability of measurement error in income across waves
of a panel. David and Bollinger (2005) find that false negative reporting of US food
stamps is highly correlated across wave one and two of the Survey of Income and Pro-
gram Participation. Das, Toepoeland van Soest (2011) propose a methodology to quantify
PC effects. They compare responses of first-time responders in refreshment samples to
responses from experienced panel members and make assumptions on attrition.1In this
spirit, Halpern-Manners, Warren and Torche (2014) note that experienced panel members
in the US General Social Survey are less likely to refuse questions about their income.
Similarly,Frick et al. (2006) find that experienced panel members report higher income in
the German Socio-Economic Panel. Nevertheless, despite the large interest of economists
in living standards, we know of no study that has performed a systematic analysis of how
measurement error in income and its components evolves across waves of a panel.
In this study, we provide evidence on the comparability of reported income across the
waves of a large general purpose panel survey: the UK Household Longitudinal Study
(UKHLS). Our approach is novel in that it exploits two features of the survey design to
estimate the causal effect of panel experience on reported income (and does not require data
linkage or refreshment samples). First, the fieldwork period for adjacent waves overlaps
by one year. This generates a natural experiment in which randomly selected samples of
individuals are interviewed at different waves of the panel but in the same calendar year.
Second, UKHLS uses ‘reactive’DI. This means that individuals are prompted with previous
1Taking this approach, Van Landeghem (2014) finds a drop in a stated utility measure across the first-rounds of
interviews in two panel surveys.
©2019 The Authors. Oxford Bulletin of Economics and Statistics published by Oxford University and JohnWiley & Sons Ltd.
To continue reading
Request your trial