The measurement of populist attitudes: Testing cross-national scales using item response theory
| Author | Steven M Van Hauwaert,Christian H Schimpf,Flavio Azevedo |
| Published date | 01 February 2020 |
| Date | 01 February 2020 |
| DOI | http://doi.org/10.1177/0263395719859306 |
| Subject Matter | Articles |
research-article2019
Article
Politics
2020, Vol. 40(1) 3 –21
The measurement of
© The Author(s) 2019
Article reuse guidelines:
sagepub.com/journals-permissions
populist attitudes: Testing
https://doi.org/10.1177/0263395719859306
DOI: 10.1177/0263395719859306
journals.sagepub.com/home/pol
cross-national scales using
item response theory
Steven M Van Hauwaert
Johannes Gutenberg University of Mainz, Germany, and University of Surrey, UK
Christian H Schimpf
GESIS – Leibniz-Institute for the Social Sciences, Germany, and GESS - University of Mannheim, Germany
Flavio Azevedo
University of Cologne, Germany
Abstract
Recent research in the populism literature has devoted considerable efforts to the
conceptualisation and examination of populism on the individual level, that is, populist attitudes.
Despite rapid progress in the field, questions of adequate measurement and empirical evaluation
of measures of populist attitudes remain scarce. Seeking to remedy these shortcomings, we apply
a cross-national measurement model, using item response theory, to six established and two new
populist indicators. Drawing on a cross-national survey (nine European countries, n = 18,368), we
engage in a four-folded analysis. First, we examine the commonly used 6-item populism scale.
Second, we expand the measurement with two novel items. Third, we use the improved 8-item
populism scale to further refine equally comprehensive but more concise and parsimonious
populist measurements. Finally, we externally validate these sub-scales and find that some of
the proposed sub-scales outperform the initial 6- and 8-item scales. We conclude that existing
measures of populism capture moderate populist attitudes, but face difficulties measuring more
extreme levels, while the individual information of some of the populist items remains limited.
Altogether, this provides several interesting routes for future research, both within and between
countries.
Keywords
Europe, item response theory, measurement, populism, scale development, scale refinement
Received: 19th October 2018; Revised version received: 8th May 2019; Accepted: 24th May 2019
Corresponding author:
Steven Van Hauwaert, Johannes Gutenberg University of Mainz, Mainz, Germany.
University of Surrey, UK.
Email: vanhauwaert@politik.uni-mainz.de
4
Politics 40(1)
Introduction
Over the past decades, populism has been a frequent object of academic investigation.
Most studies focus on its conceptualisation, its theorisation, and its influence throughout
party politics. Recently, however, a number of studies conclude that populism may also
manifest itself at the individual level. That is, individuals hold so-called populist attitudes
that are thought to underlie populist proclivities and a potential populist vote. Recent
scholarship argues that populist attitudes are prominent and persistent (Hawkins et al.,
2018b). Various empirical studies find that populist attitudes can play an important role in
shaping people’s thinking and behaviour in various countries and world regions, for
example, in the Americas (Hawkins et al., 2012; Hawkins and Riding, 2010), Chile
(Meléndez and Rovira Kaltwasser, 2017; Slovakia (Stanley, 2011), Poland (Stanley,
2019), the Netherlands (Akkerman et al., 2014, 2017; Jacobs et al., 2018), and in a broader
European context (Van Hauwaert and van Kessel, 2018).
Most of this scholarship draws from standalone measurement studies that were con-
ducted in the Americas and Europe using a similar battery of survey items assumed to
measure populist attitudes (Akkerman et al., 2014; Hawkins et al., 2012; Hawkins and
Riding, 2010). Scholarly studies – and ensuing conclusions – increasingly rely on these
(or similar) measures to examine relationships between populism and broader political
behaviour in various contexts. And while such studies are unquestionably insightful, the
integrity of their results depends on having an appropriate measurement of populist atti-
tudes. This forms the crux of a recent set of measurement studies, examining populist
scales across countries (Castanho Silva et al., 2018, 2019).
It further raises the question of to what extent existing measures allow researchers to
systematically model populist attitudes. The purpose of this study is to move beyond
assessment only, and evaluate whether a populist scale – and its substantive and methodo-
logical operationalisations – indeed (1) extracts people’s populist attitudes from self-
reported answers, and (2) provides a cross-national measurement for the European
context. That is, we explore how informative Akkerman et al.’s (2014) original 6-item
measurement is and assess its validity across countries. To accomplish these goals, we
leverage cross-sectional survey data from 2015 (n = 18,368) that covers nine European
countries (France, Germany, Greece, Italy, Poland, Spain, Sweden Switzerland, and the
United Kingdom) and includes the original six items. Methodologically, we rely on item
response theory (IRT) to provide a careful analysis of the 6-item populist measurement
and assess its validity.
This study unfolds as follows. After a brief overview of the literature on populist atti-
tudes and their measurement, we discuss a number of caveats in the existing research and
provide a general rationale for why we propose IRT as an adequate analytical tool to
develop a populism scale. We proceed to cross-nationally test the 6-item scale that is typi-
cally only applied nationally but is quickly becoming a point of reference for other studies
throughout the literature. We continue by analysing the performance of two novel items,
and their contribution to the accurate measurement of populist attitudes. From this initial
evaluation, we make a series of recommendations towards an equally comprehensive but
more parsimonious populist measurement that can serve as a cornerstone for future
research. Our analysis reveals that existing items generally have difficulties capturing the
extreme ends of the populism scale, while they conceptually capture some attributes of
populism, especially anti-elitism, better than others.
Van Hauwaert et al.
5
Conceptualising populist attitudes
The populism scholarship brings forward an extensive conceptual debate that spans from
foundational writings (Canovan, 1981, 1984; Hofstadter, 1960; Ionescu and Gellner, 1969;
Weber, 1919) to contemporary studies (Albertazzi and McDonnell, 2008; Aslanidis, 2016;
Canovan, 2005; Moffitt and Tormey, 2014; Mudde, 2004; Weyland, 2001). But even if
there are a number of conceptual accounts of populism, academics have gradually con-
verged around a shared understanding of populism as a set of ideas or a so-called ‘idea-
tional’ definition (Hawkins, 2009; Hawkins et al., 2018a; Mudde and Rovira Kaltwasser,
2017). By integrating several distinct theoretical schemes, the ideational approach allows
for the analysis of different expressions of populism and the harmonisation of various key
concepts.
Through this study, we limit our analysis to one such expression, namely populism as:
(…) an ideology that considers society to be ultimately separated into two homogenous and
antagonistic groups, “the pure people” and “the corrupt elite”, and which argues that politics
should be an expression of the volonté générale [general will] of the people. (Mudde, 2004: 543)
Considering populism as an idiosyncratic construct (cf. Stanley, 2008), we recognise that
it conveys a minimalist, yet dynamic collection of coherent ideas (Hawkins, 2009) related
to the way in which democracy is organised (cf. Caramani, 2017; Riker, 1982). Thus, it
allows for a unique interpretation of the political reality through the interaction of its core
concepts, namely anti-elitism, the sovereignty of the people, and a Manichean division of
the world into good and bad.
While the majority of research examines populism and its components (or concepts in
Freeden’s language) as a party-level phenomenon (e.g. van Kessel, 2015), we should not
restrict ourselves to only one level – just like we are not in the studies of liberalism,
nationalism, or conservatism. If we follow Freeden (1996) and understand ideologies as
larger-scale interpretative frameworks that combine concepts, then it also becomes tena-
ble for individuals to hold populist views and be considered populist. In line with the vast
majority of scholarship, we argue in favour of an attitudinal interpretation (cf. Akkerman
et al., 2014).
An attitude in its most general form can be described as a ‘psychological tendency,
expressed by evaluating a particular entity with some degree or favor or disfavor’ (Eagly
and Chaiken, 1993: 1). Attitudes, in other words, are ‘evaluate reactions associated with
a target object’ (Meffert et al., 2004: 63). More specifically, an attitude encompasses three
elements: tendency, attitude object, and evaluation (Eagly and Chaiken, 2007). First, an
appraisal component refers to individual sentiment regarding a statement or object.
Second, an informational component refers to individuals’ beliefs about that statement or
object. Third, tendency implies that, based on past experiences, respondents reply to an
attitude object with ‘some degree of positivity or negativity’ (Eagly and Chaiken, 2007:
585). Only this latter component can be observed; the other two can only be inferred.1
Applied to our case, populism is the target object, and we assume individuals...
To continue reading
Request your trial