Multiple Visits and Data Quality in Household Surveys

Date01 April 2018
AuthorMatthias Schündeln
Published date01 April 2018
DOIhttp://doi.org/10.1111/obes.12196
380
©2017 The Department of Economics, University of Oxford and JohnWiley & Sons Ltd.
OXFORD BULLETIN OF ECONOMICSAND STATISTICS, 80, 2 (2018) 0305–9049
doi: 10.1111/obes.12196
Multiple Visits and Data Quality in Household
Surveys*
Matthias Sch ¨
undeln
Goethe University Frankfurt, 60629 Frankfurt am Main, Germany,
(e-mail: schuendeln@wiwi.uni-frankfurt.de)
Abstract
In order to increase data quality some household surveys visit the respondent households
several times to estimate one measure of consumption. For example, in Ghanaian Living
Standards Measurement surveys, households are visited up to 10 times over a period of
1 month. I find strong evidence for conditioning effects as a result of this approach: In
the Ghanaian data the estimated level of consumption is a function of the number of prior
visits, with consumption being highest in the earlier survey visits. Telescoping (perceiving
events as being more recent than they are) or seasonality (first-of-the-month effects)cannot
explain the observed pattern. To study whether earlier or later survey visits are of higher
quality, I employ a strategy based on Benford’s law. Results suggest that the consumption
data from earlier survey visits are of higher quality than data from later visits. The findings
have implications for the value of additional visits in household surveys, and also shed
light on possible measurement problems in high-frequency panels. They add to a recent
literature on measurement errors in consumption surveys (Beegle et al., 2012, Gibson et al.,
2015), and complement findings by Zwane et al. (2011) regarding the effect of surveys on
subsequent behaviour.
I. Introduction
Consumption data are of central importance to the study of a large range of empirical
questions that are of interest to both academics and policy makers. They are frequently
used, for example, to study levelsand distributions of welfare, poverty, or vulnerability, and
their determinants. Collecting accurate data on consumption and expenditure, however, is
not straightforward, and approaches differ, sometimes substantially, across countries and
over time. Specifically, Beegle et al. (2012, p. 4) list four ‘primary dimensions [in which]
the main methods of consumption data collection’ vary: ‘diary vs. recall, the level of
aggregation or detail in the commodity list, the reference period and the levelof respondent’.
Therefore, a number of papers have recently used experimental methods to assess the role
JEL Classification numbers: C81, O12, I32, D12.
*I thank Ghana Statistical Service for sharing the data. I gratefully acknowledge very helpful discussions with
Luc Christiaensen and valuable inputs on an earlier version of this paper fromAhmed Ragab.
Multiple visits and data quality 381
of different aspects of survey design for measuring consumption and possible biases that
result from non-classical measurement error in consumption measures (Beegle et al., 2012,
Caeyers, Chalmers and De Weerdt, 2012, Gibson et al., 2015, Friedman et al., 2016). The
present paper adds to this literature in two dimensions. First, it considers in depth one
aspect of survey design that seems to be largely unexplored as of yet, namely the number
of interviews that is used to calculate a single measure of consumption. Second, the paper
suggests a novel approach to deal with the ‘fundamental problem in assessing survey bias’
(Meyer, Mok and Sullivan, 2015, p. 200), namely the lack of a benchmark measure of the
true outcome. To deal with this, the paper proposes Benford’s law as an analytical tool to
investigate the quality of consumption data over time and applies it to the data at hand.
More specifically, the paper studies nationally representative surveys conducted in
Ghana (the Ghana Living Standards Surveys 3, 4 and 5), in which households are in-
terviewed repeatedly – responding to the same consumption module up to 10 times over
a period of 1 month – with the purpose of obtaining one precise measure of consumption
by combining data from a number of interviews. I find that the consumption measures
obtained from any individual interview drop significantly over the interview period. Thus,
the frequency of visits affects measured annualized consumption, and as a consequence,
will also affect estimates of poverty and inequality, among others. The implications are
economically significant: for example, as shown in section III, a measure related to food
poverty in rural Ghana increases by about 13 percentage points if data from all visits are
used instead of only using data from the first visit.1
Thus, the question arises: which consumption data should be considered the one that
best reflects the true consumption of interviewed households? Theoretically, a number of
arguments, which are reviewed below, would predict that data quality is higher in earlier
interview visits. However, there are also arguments that suggest that later visits provide
higher quality data. Finally, a combination of arguments could imply that the data quality
is (inversely) U-shaped. To answer this question, ideally, a measure of the true outcome
would be available. One possible approach is to compare survey results with administrative
data (e.g. Meyer et al., 2015). In the absence of administrative data, as is common in many
developing country contexts, Beegle et al. (2012) and Gibson et al. (2015) use a ‘personal
diary’ with dailyvisits by a local assistant and visits every two days by a survey enumerator
to establish a benchmark.2This paper introduces a new approach to study the question of
how data quality changes over time, based on Benford’s law.
Benford’s law describes a statistical regularityfor the frequency distribution for first dig-
its of numerical data.3The following analysisis based on the hypothesis that first significant
digits of true (precisely measured) data will be more likely to conform to the distribution
predicted by Benford. On the other hand, false – or, more generally, low quality – data will
1Variation in the number of interviews used across surveys will therefore also affect comparability of poverty
measures over time and place (Lanjouw and Lanjouw, 2001).
2The paper by Beegle et al. (2012) also features two treatments that allow for a comparison of frequent (daily)
and infrequent (a total of three) visits over a period of 14 days.But that paper’s interest is in the total difference, and
does not ask the question whether earlier or later visits provide more accurate data. Moreover,in both cases diaries
are used, as opposed to interviews with recall.
3Benford’slaw also has implications for the distribution of second- and higher-order digits. This paper will only
make use of first significant digits.
©2017 The Department of Economics, University of Oxford and JohnWiley & Sons Ltd

To continue reading

Request your trial

VLEX uses login cookies to provide you with a better browsing experience. If you click on 'Accept' or continue browsing this site we consider that you accept our cookie policy. ACCEPT