Meta‐Regression Methods for Detecting and Estimating Empirical Effects in the Presence of Publication Selection*
| DOI | http://doi.org/10.1111/j.1468-0084.2007.00487.x |
| Published date | 01 February 2008 |
| Date | 01 February 2008 |
| Author | T. D. Stanley |
103
©Blackwell Publishing Ltd and the Department of Economics, University of Oxford, 2007. Published by Blackwell Publishing Ltd,
9600 Garsington Road, Oxford OX4 2DQ, UK and 350 Main Street, Malden, MA 02148, USA.
OXFORD BULLETIN OF ECONOMICS AND STATISTICS, 70, 1 (2008) 0305-9049
doi: 10.1111/j.1468-0084.2007.00487.x
Meta-Regression Methods for Detecting and
Estimating Empirical Effects in the Presence of
Publication Selection*
T. D. Stanley
Department of Economics, Hendrix College, Conway, AR 72032 USA
(e-mail: Stanley@Hendrix.edu)
Abstract
This study investigates the small-sample performance of meta-regression methods
for detecting and estimating genuine empirical effects in research literatures tainted
by publication selection. Publication selection exists when editors, reviewers or
researchers have a preference for statistically significant results. Meta-regression
methods are found to be robust against publication selection. Even if a literature is
dominated by large and unknown misspecification biases, precision-effect testing and
joint precision-effect and meta-significance testing can provide viable strategies for
detecting genuine empirical effects. Publication biases are greatly reduced by com-
bining two biased estimates, the estimated meta-regression coefficient on precision
(1/Se) and the unadjusted-average effect.
[P]ublication bias is leading to a new formulation of Gresham’s law – like bad money,
bad research drives out good. – Bland (1988, p. 450)
I. Empirical economics and its publication selection bias
This paper offers a statistical approach to estimating and testing empirical effects
in the presence of publication selection and simulates its properties under realistic
*I wish to thank Chris Doucouliagos, Stephen Jarrell, Randall Rosenberger, Alex Sutton, and an anony-
mous referee for their helpful comments. I also gratefully acknowledge the support of a US Environmental
Protection Agency STAR (Science To Achieve Results) grant #RD-832-421-01. Although the research has
been funded in part by the US-EPA, it has not been subjected to theAgency’s peer and policy review and
therefore does not necessarily reflect the views of the Agency.Any remaining error or omission is solely my
responsibility.
JEL Classification numbers: C12, C13, B40.
104 Bulletin
research conditions. Publication bias has long been recognized as another serious
threat to empirical economics (De Long and Lang, 1992). More recently, Card and
Krueger (1995),Ashenfelter, Harmon and Oosterbeek (1999), G ¨org and Strobl (2001),
Doucouliagos, Laroche and Stanley (2005), Abreu, de Groot and Florax (2005),
Doucouliagos (2005), Nijkamp and Poot (2005), Rose and Stanley (2005), and
Stanley (2005a) have all used meta-regression analysis (MRA) to uncover evidence
of publication bias in specific areas of economic research. Publication bias, or the
‘file drawer problem’, is the consequence of choosing research papers for the statis-
tical significance of their findings. ‘Statistically significant’ results are often treated
more favourably by researchers, reviewers and/or editors; hence, larger, more
significant, effects are over-represented. Studies with small, ‘insignificant’ effects
will tend to remain in the ‘file drawer’ (Rosenthal, 1979). Publication selection
biases a literature’s average reported empirical effect away from zero.1This bias
is a problem for any summary of empirical research, including narrative literature
reviews (Laird and Mosteller, 1988; Phillips and Goss, 1995; Sutton et al., 2000a;
Stanley, 2001).2
Econometric estimates can easily be overwhelmed by publication selection
because there are so many plausible econometric models to choose from. Conventional
literature reviews and econometric techniques are powerless to address publication
bias. If, for example, only half the studies select the results that they report, the aver-
age estimates across a literature and the proportion of studies that find a significant
effect will be dominated by publication bias, irrespective of the underlying empiri-
cal ‘truth’. Such an empirical literature is indistinguishable, by conventional econo-
metric methods, from a literature that contains an authentic effect and yet refrains
from publication selection. Therefore, current econometric methodology cannot reli-
ably assess the empirical merit of any economic hypothesis. Issues of publication
selection, its identification and circumvention are crucial to a genuinely empirical
economics.
Without some correction for publication bias, a literature that appears to contain
a large empirical effect offers little, if any, reason for accepting this effect. At best,
conventional narrative reviews serve as vote-counts of the number of studies that
find a significant effect vs. those that do not (Stanley, 2001, pp. 144–146). When
there is no authentic effect, but only publication selection, the expected proportion
of research studies that will report a significant effect is: +(1 −); where =
the incidence of publication selection (i.e. the proportion of studies that choose to
report only significant effects), and is a conventional significance level (0.05). Thus,
even if a minority of the reported effects is the result of selection, the majority of the
literature can be expected to report a significant effect. In particular, for a research
1Stanley (2005a) offers a more detailed discussion of publication selection bias and its effects on empirical
economics.
2As an anonymous referee points out, publication bias also makes the interpretation from a single study
problematic, no matter how well this study is conducted. The advantage of a summary, of course, is that
random misspecification and sampling errors are averaged and thereby lessened.
©Blackwell Publishing Ltd and the Department of Economics, University of Oxford 2007
To continue reading
Request your trial