Variable Selection in Cross‐Section Regressions: Comparisons and Extensions
| Author | Christoph Hanck,Thomas Deckers |
| Date | 01 December 2014 |
| Published date | 01 December 2014 |
| DOI | http://doi.org/10.1111/obes.12048 |
841
©2013 The Department of Economics, University of Oxford and JohnWiley & Sons Ltd.
OXFORD BULLETIN OF ECONOMICSAND STATISTICS, 76, 6 (2014) 0305–9049
doi: 10.1111/obes.12048
Variable Selection in Cross-Section Regressions:
Comparisons and Extensions
Thomas Deckers† and Christoph Hanck‡
†Bonn Graduate School of Economics (BGSE), Universit¨at Bonn, Bonn, Germany
(e-mail: thomas.deckers@uni-bonn.de)
‡Faculty of Economics, Universit¨at Duisburg-Essen, Universit¨atstrasse 12, 45117, Essen
Germany (e-mail: christoph.hanck@vwl.uni-due.de)
Abstract
Cross-section regressions often examine many candidate regressors. We use multiple
testing procedures (MTPs) controlling the false discovery rate (FDR) — the expected
ratio of false to all rejections — so as not to erroneously select variables because many
tests were performed, yielding a simple model selection procedure. Simulations compar-
ing the MTPs with other common model selection criteria demonstrate that, for conven-
tional tuning parameters of the selection procedures, only MTPs consistently control the
FDR, but have slightly lower power. In an empirical application to growth, MTPs and
PcGets/Autometrics identify similar growth determinants, which differ somewhat from
those obtained by Bayesian ModelAveraging.
I. Introduction
Model uncertainty is one of the most frequent problems in applied econometric work. It
describes the common situation in which the investigator is faced with a large number of
candidate explanatory variables for some dependent variable of interest. Given the typical
size of economic data sets, the investigator then needs a procedure to select the relevant
determinants from the pool of candidate variables. That is, he or she needs to perform
‘model selection’, or equivalently, ‘variable selection.’ Unfortunately,there is no generally
accepted, let alone efficient or most powerful, way to do so. A leading example of this
situation is that of selecting growth determinants in a cross-section growth regression
(e.g., Levine and Renelt, 1992).
*We thank the editor (JonathanTemple), two anonymous referees, J¨org Breitung, Matei Demetrescu, Jan Jacobs,
Franz Palm, Stephan Smeekes, Jean-Pierre Urbain and Stefan Zeugner for useful comments that helped improve the
paper.Steve Perez and Stefan Zeugner kindly providedsome code. Thomas Deckers gratefully acknowledges financial
support from the Deutsche Forschungsgemeinschaftthrough the Bonn Graduate School of Economics (BGSE). Part of
the article was written while the authors wereat Maastricht University, whose hospitality is gratefully acknowledged.
All errors are ours. An extended version of the article containing online appendices with additional results and the
data and programs used in this article are available at http://www.oek.wiwi.uni-due.de/team/christoph-hanck.
JEL Classification numbers: C12, O47.
842 Bulletin
A seemingly simple (and often adopted) solution to model selection is to test each
variable jindividually at some level using appropriate P-values, rejecting hypothesis
Hjif ˆpj. But, given the large set of regressors, the data are given many chances
of falsely rejecting (‘multiplicity’). Hence, one is bound to erroneously declare some
irrelevant variables to be significant using this approach.
The statistical literature has a long history of dealing with this issue of multiplicity (e.g.,
Holm, 1979). In this article, we set out some of these so-called multiple testing procedures
(MTPs), which we believedeser ve more attention from applied econometricians.We com-
pare these to other well-established model selection procedures (for details see below).
More specifically, we follow Romano, Shaikh and Wolf (2008b) who suggest to employ
MTPs to perform model selection when there is a large set of candidate variables, as is the
case in for instance growth econometrics. These methods are fast and easy to implement.
Concretely,we employ the widely-used procedures of Benjamini and Hochberg (1995) and
Romano, Shaikh and Wolf (2008a), which are reviewed in section II. For instance, all it
takes to implement the Benjamini and Hochberg (1995) method is to compare the P-value
of each variable in a regression including all candidate regressors to a specific cutoff j
rather than to compare each of them to .
Commensurate with the practical relevance of model selection, a large number of pro-
posals have been made to offer empirical researchers less arbitrary and more rigorous
ways of selecting an empirical model. Prominent examples, reviewed in section II, include
the ‘two million regressions approach’ of Sala-i-Martin (1997), Bayesian Model Averag-
ing (e.g., Fernandez, Ley and Steel, 2001, BMA), General-to-Specific/Autometrics (e.g.,
Hoover and Perez, 1999; Krolzig and Hendry, 2001) and the Least Absolute Shrinkage
and Selection Operator (Lasso) (Tibshirani, 1996; Zou, 2006). In practice, all methods
require the user to specify some criterion, controlling for instance nominal type I errors
in the General-to-Specific/Autometrics search paths, the tolerated ‘multiple’ type I error
(see below) for the MTPs or some threshold for variable importance in BMA (e.g. the
popular choice of a posterior inclusion probability of more than 50%). Employing common
choices for these specifications, this article provides a thorough comparison of the above-
mentioned MTPs with these widely used model selection methods. This is done by means
of a Monte Carlo study as well as by empirically investigating the prominent example of
variable selection in cross-section growth regressions.
The Monte Carlo study in section III investigates data generating processes (DGPs)
that intend to mimic data sets often found in growth empirics. Unsurprisingly, the results
demonstrate that there exists no uniformly best model selection procedure. Model selection
– in fact, any inferential procedure – always implies a tradeoff between size and power.
Specifically, higher tolerated size generallyleads to higher power, where we define powerto
be the number of correctly selected variables. (Our concrete measure of size in the present
multiple testing situation is explained in the following paragraph.) Hence, the notion of
‘the best procedure’ strongly depends on the researcher’s preferences regarding the ‘size–
power tradeoff’. Our approach to investigating the effectiveness of the procedures is to
compare their power in situations in which they have the same, or very similar, size. That
is, we workwith a measure of ‘size-adjusted’ power.Similarly, we look for constellations in
which two procedures havevery similar power, but differ in the required size to achievethat
power. Using these benchmarks, one central message of our article is that some methods
©2013 The Department of Economics, University of Oxford and JohnWiley & Sons Ltd
Variable selection in cross-section regressions 843
are practically dominated by others: among the MTPs, the bootstrap method of Romano
et al. (2008a) identifies most relevant variables for a given size requirement. Moreover, we
find that General-to-Specific/Autometrics and some variants of BMA seem to be a bit more
powerful given a certain size than the MTPs. The approaches of Sala-i-Martin (1997) and
the Lasso are dominated, for example, by the General-to-Specific/Autometrics approach
which identifies roughly as many relevant variables while having a much smaller size.
In a multiple testing situation such as model selection, one needs to suitably generalize
the notion of a type I error. The notion we focus on in this article is the false discovery
rate (FDR, Benjamini and Hochberg, 1995). The FDR is defined as the expected value of
the number of falsely rejected hypotheses divided by the overall number of rejections. We
argue that FDR-control is a useful notion in the present problem: researchers maybe willing
to tolerate a small expected fraction of erroneously selected variables among all selected
variables. That is, they wish to avoid overly many ‘false positives’. For the example of
selecting growth determinants, this implies that researchers are willing to expect that a small
number of all growth determinants found significant actually do not driveeconomic growth.
Hence, the FDR can also be used as a possible measure for ‘size’ in these multiple testing
situations: the observed FDR in simulations gives an indication of the fraction of all rejec-
tions one needs to expect to be false positives.As such, it is related to the definition of size in
a single hypothesis test, where size is defined as the probability of obtaining a false positive.
Concerning control of the FDR, the Monte Carlo study reveals that the MTPs are the
only model selection procedures to consistently control the FDR. To the extent that the
FDR is agreed to be a useful multiple type I error rate, we consider this to be an important
finding: the well-established standard approach for single hypothesis tests is to focus on
tests controlling size at some prespecified level over a wide range of DGPs. One then
proceeds to look for tests with high power within this class. It therefore does not seem
implausible to adopt an analogous strategy in multiple testing situations.
In the empirical application of section IV, the variables jointly selected by all model
selection procedures mostly havea plausible economic or cultural and religious motivation.
The MTPs find few growth determinants beyond initial GDP when controlling the FDR at
very small levels, providing some evidence for conditional convergence. We further find
that the MTPs, PcGets/Autometrics and Lasso identify similar variables, which may differ
substantially from those identified by BMA. Initial GDP is included by all procedures.
Section II illustrates the problem of multiplicity using the example of cross-section
growth regressions to then sketch the different model selection procedures including the
MTPs. Section III presents the setup and findings of the Monte Carlo study. Section IV
applies the model selection procedures to the empirical example of cross-section growth
regression. Section V concludes.
II. Problem and methods
Cross-section growth regressions
To further motivate the testing problem and to prepare the ground for the empirical appli-
cation of section IV, let us discuss the leading example of selecting growth determinants
in some more detail.
©2013 The Department of Economics, University of Oxford and JohnWiley & Sons Ltd
To continue reading
Request your trial