How and how much does expert error matter? Implications for quantitative peace research
| Author | Kyle L Marquardt |
| Published date | 01 November 2020 |
| Date | 01 November 2020 |
| DOI | http://doi.org/10.1177/0022343320959121 |
| Subject Matter | Regular Article |
How and how much does expert error
matter? Implications for quantitative
peace research
Kyle L Marquardt
School of Politics and Governance & International Center for the Study of Institutions and Development,
National Research University Higher School of Economics
Abstract
Expert-coded datasets provide scholars with otherwise unavailable data on important concepts. However, expert
coders vary in their reliability and scale perception, potentially resulting in substantial measurement error. These
concerns are acute in expert coding of key concepts for peace research. Here I examine (1) the implications of these
concerns for applied statistical analyses, and (2) the degree to which different modeling strategies ameliorate them.
Specifically, I simulate expert-coded country-year data with different forms of error and then regress civil conflict
onset on these data, using five different modeling strategies. Three of these strategies involve regressing conflict onset
on point estimate aggregations of the simulated data: the mean and median over expert codings, and the posterior
median from a latent variable model. The remaining two strategies incorporate measurement error from the latent
variable model into the regression process by using multiple imputation and a structural equation model. Analyses
indicate that expert-coded data are relatively robust: across simulations, almost all modeling strategies yield regression
results roughly in line with the assumed true relationship between the expert-coded concept and outcome. However,
the introduction of measurement error to expert-coded data generally results in attenuation of the estimated
relationship between the concept and conflict onset. The level of attenuation varies across modeling strategies: a
structural equation model is the most consistently robust estimation technique, while the median over expert codings
and multiple imputation are the least robust.
Keywords
Bayesian methods, civil conflict, conflict onset, ethnic politics, expert-coded data, latent variable models
Expert-coded datasets such as the Chapel HillExpert Sur-
vey, Electoral Integrity Project, Human Rights Measure-
ment Initiative, and Varieti es of Democracy (V-Dem)
allow scholars to conduct cross-national longitudinal
research on vital concepts (Bakker et al., 2012; Norris,
Frank & Martı
´nez i Coma, 2014; Clay et al., 2020; Cop-
pedge etal., 2018). However, expert-coded datacome with
potentialdisadvantages. Expertsare susceptible to different
sources of error (Clinton & Lewis, 2008; Bakker et al.,
2014;Marquardt & Pemstein,2018b); such error maybias
results in statistical analyses (Lindsta
¨dt, Proksch & Slapin,
2018). Theseconcerns are particularlyacute in the context
of quantitativepeace research. Sinceoutcomes such as con-
flict onset are rare events, quantitative analyses will be
sensitive to measurement error on the right-hand side.
Moreover, expert perceptions of key correlates of conflict
may be endogenous to this outcome.
Given these concerns, awareness of the degreeto which
different forms of expert error substantively matter is of
great importance, as is understanding of the extent to
which different modeling strategies can correct for these
errors. I provide insight into these two issues by conduct-
ing a series of ecologically valid simulation analyses in
which I vary expert error in the measurement of a latent
Corresponding author:
kmarquardt@hse.ru
Journal of Peace Research
2020, Vol. 57(6) 692–700
ªThe Author(s) 2020
Article reuse guidelines:
sagepub.com/journals-permissions
DOI: 10.1177/0022343320959121
journals.sagepub.com/home/jpr
To continue reading
Request your trial