THE VALIDITY OF SUBJECTIVE PERFORMANCE MEASURES: SCHOOL PRINCIPALS IN TEXAS AND DENMARK
| Author | SIMON CALMAR ANDERSEN,SØREN C. WINTER,KENNETH J. MEIER,LAURENCE J. O'TOOLE,NATHAN FAVERO |
| Date | 01 December 2015 |
| Published date | 01 December 2015 |
| DOI | http://doi.org/10.1111/padm.12180 |
doi : 10. 1111/p adm .12180
THE VALIDITY OF SUBJECTIVE PERFORMANCE
MEASURES: SCHOOL PRINCIPALS IN TEXAS
AND DENMARK
KENNETH J. MEIER, SØREN C. WINTER, LAURENCE J. O’TOOLE Jr, NATHAN
FAVERO AND SIMON CALMAR ANDERSEN
Public management studies are increasingly using survey data on managers’ perceptions of perfor-
mance to measure organizational performance. These perceptual measures are tempting to apply
because archival performance data or surveys of target group outcomes and satisfaction are often
lacking, costly to provide, and are highly policy specic rendering generalization difcult. But are
perceptual performance measures valid, and do they generate unbiased ndings? Weexamine these
questions in a comparative study of middle managers in schools in Texasand Denmark. The ndings
are remarkably similar.Managers systematically overestimate the performance of their organization,
perceptual performance is only weakly associated with archival performance, and managers do not
provide sophisticated assessment of performance by giving their organization credit for the con-
straints it meets or discounting the resources it has. Even worse, the use of perceptual performance
measures seems to providebiased estimates when examining how management affects performance.
This is due to both random measurement error and common source bias.
INTRODUCTION
A dilemma of public management scholarship, particularly concerning the question of
whether management affects organizational performance, is the choice between general-
ization versus valid measures of performance. There are few, if any,valid generic measures
of public organizational performance that can be used if organizations are doing such dif-
ferent things as providing job training, educating children, and delivering healthcare for
the aged. The public agencies in each area generally have established performance indi-
cators that are specic to agency objectives. This concern with agency-specic indicators
and issues of measurement validity and the desire to convince practitioners and scholars
from other disciplines that our studies of management are useful in turn limit the general-
izability of our ndings. Weare sure, for example, that creating external ties to manage the
environment pays benets in local law enforcement (Nicholson-Crotty and O’Toole 2004)
or education in Texas (Meier and O’Toole 2003), but it is not clear that we can generalize
this nding to water quality projects in the Netherlands or road construction in Zambia.
Some scholars have opted to use self-evaluations by agency employees or agency
managers who respond to questions such as ‘How would you rate the performance of
your workgroup compared to others in …?’ Such measures are part of several data sets
(the Federal Employee Viewpoint Survey, the National Administrative Studies Project,
the Merit Principles Survey, and the American State Administrators’ Project) that have
spawned a substantial literature on public management and performance. Often studies
using these data directly refer to such measures as evaluating performance rather than
evaluating perceived performance (Pandey and Moynihan 2006). Although bureaucrats
who manage or implement a programme know a great deal about their programmes
Kenneth J. Meier is at Texas A&M University, College Station, Texas, USA; SFI (the Danish National Centre for Social
Research), Copenhagen, Denmark; and Cardiff University, UK. Søren C. Winter is at SFI. Laurence J. O’Toole Jr is at
SFI; and the University of Georgia, Athens, Georgia, USA. Nathan Favero is at TexasA&M University. Simon Calmar
Andersen is at Aarhus University,Denmark.
Public Administration Vol.93, No. 4, 2015 (1084–1101)
© 2015 John Wiley & Sons Ltd.
VALIDITYOF SUBJECTIVE PERFORMANCE MEASURES 1085
and this question appears general enough to use across a wide variety of organizations,
an issue can be raised concerning how strongly this measure is correlated with actual
performance, essentially an issue of measurement validity.
This matter of generalizability versus validity is especially important in public manage-
ment because a recent set of studies of top-level managers presented systematic evidence
that: (1) managers overestimated the performance of their organization; (2) the differ-
ence between managers’ evaluations and existing archival performance measures did not
reect systematic adjustments to account for more difcult tasks or variation in resources;
and (3) the measures contained substantial common source bias with other questions in
the same survey (Meier and O’Toole2013a, 2013b). This last point was especially alarming;
one article found evidence that as many as half of all relationships examined were spuri-
ous (Meier and O’Toole 2013b). The clear implication was that scholars using perceptual
measures have produced a number of ndings that do not meet the high standards of a
design science and should not be used as guides for practice.
These studies are not the nal word on the use of perceptualmeasures, however, because
they covered only a single management setting, Texas school districts, and were only for
top-level managers. These managers may have greater incentives to misperceive results,
given the high-stakes performance appraisal system (performance clauses are frequently
in the contracts of these managers). Although the authors argued that their study was
the most ideal one for nding congruence between ofcial measures of performance and
perceptions (one question asked about performance on specic standardized tests that
could be compared directly to such test results), prudence suggests additional assessments
in other contexts.
This study examines the linkage between managerial assessments of performance and
actual performance by replicating this prior study,but with two notable differences. First,
rather than examining chief executives, it examines middle managers. Middle managers,
who are closer to programme delivery, might have more accurate views of how well an
organization is performing. Second, we site the study in two different countries to see
whether the high-stakes performance environment in Texas inuenced the earlier results.
The study examines Danish and Texas school principals using parallel surveys and
archival data. Although both studies deal with middle managers in education, there are
sufcient differences in the job of principals in Texas and Denmark (see below) that could
lead to more or less accurate self-perceptions. We carefully consider issues of overestima-
tion, overlap between perceptual and archival measures, adjustment of perceptions based
on constraints and resources, same source bias, and sensitivity to empirical context. The
main contribution of the article is methodological in examining the validity of perceptual
measures of performance among managers closer to programme delivery and in contexts
with different levels of high-stakes performance measurement as well as other contextual
variables that might affect the validity.We also examine if selected aspects of management
have different impacts on organizational performance when applying perceptual and
archival performance measures.
CONCEPTUAL FOUNDATIONS
The cognitive processes used by respondents to answer survey questions can be broadly
understood in terms of Groves et al.’s (2009, pp. 218–23) model of survey response. Their
model identies four sets of processes which follow a loose ordering, although respon-
dents may skip, backtrack, or simultaneously engage in multiple processes. The rst set
Public Administration Vol.93, No. 4, 2015 (1084–1101)
© 2015 John Wiley& Sons Ltd.
To continue reading
Request your trial