Forecast Evaluation Under Asymmetric Loss: A Monte Carlo Analysis of the EKT Method
| Published date | 01 April 2019 |
| Date | 01 April 2019 |
| DOI | http://doi.org/10.1111/obes.12268 |
| Author | Julian LeCrone,Jens J. Krüger |
437
©2018 The Department of Economics, University of Oxford and JohnWiley & Sons Ltd.
OXFORD BULLETIN OF ECONOMICSAND STATISTICS, 81, 2 (2019) 0305–9049
doi: 10.1111/obes.12268
Forecast Evaluation Under Asymmetric Loss:A
Monte Carlo Analysis of the EKT Method
Jens J. Kr ¨
uger† and Julian LeCrone‡
†Department of Law and Economics, Darmstadt University of Technology, Hochschulstr. 1,
64289 Darmstadt, Germany (e-mail: krueger@vwl.tu-darmstadt.de)
‡Deutsche Bundesbank, Directorate General Statistics, Wilhelm-Epstein-Straße 14, 60431
Frankfurt/Main, Germany (e-mail: julian.lecrone@bundesbank.de)
Abstract
This paper contributes to the literature on forecast evaluation by conducting an extensive
Monte Carlo experiment using the evaluationprocedure proposed by Elliott, Komunjer and
Timmermann.We consider recent developmentsin weighting matrices for GMM estimation
and testing. We pay special attention to the size and power properties of variants of the
J-test of forecast rationality.Proceeding from a baseline scenario to a more realistic setting,
our results show that the approach leads to precise estimates of the degree of asymmetry of
the loss function. For correctly specified models, we find the size of the J-tests to be close
to the nominal size, while the tests have high power against misspecified models. These
findings are quite robust to inducing fat tails, serial correlation and outliers.
I Introduction
Forecast evaluation exercises using sophisticated statistical methods are increasingly be-
coming standard. One popular forecast evaluation method is that proposed by Elliott,
Komunjer and Timmermann (2005, 2008), EKT henceforth. The EKT procedure consists
of the estimation of the asymmetry parameter (and sometimes the curvature parameter)
of a loss function jointly with the statistical test of forecast efficiency. The main focus
of applications of the EKT approach has been to business cycle forecasts (output growth
and inflation).1In more recent years there is an increasing number of studies focusing on
other forecasts, chiefly energy prices, financial and fiscal variables.2The knowledgeof the
finite sample properties of those forecast evaluation procedures is very important for the
interpretation of their results.
JEL Classification numbers: C32, C53, E37 .
1See e.g. Capistr´an (2008), Capistr´an and Timmermann (2009), Christodoulakis and Mamatzakis (2008), D¨opke,
Fritsche and Siliverstovs (2010), EKT (2005, 2008), Kr¨uger and Hoss (2012), Pierdzioch, R¨ulke and Stadtmann
(2012, 2015) and Wangand Lee (2014).
2See e.g. Aretz, Bartram and Pope(2011), Auffhammer (2007), Christodoulakis and Mamatzakis (2009), Fritsche
et al. (2014), Krol (2013), Mamatzakis (2014) and Pierdzioch, R¨ulke and Stadtmann (2013).
438 Bulletin
There exists quite a number of methodological and applied papers on forecast evaluation
reporting the results of less extensive Monte Carlo exercises for evaluating finite sample
properties. EKT themselves, Capistr´an (2005), Christodoulakis and Mamatzakis (2009)
and Naghi (2015) are to be mentioned in this respect. Most of these papers focus only on
test size and neglect test power,with Capistr ´an (2005) being the notable exception. Missing
to date, however, is a more comprehensive investigation of the finite sample properties of
the EKT procedure, looking at the estimation of the asymmetry parameter as well as size
and power of the test for efficiency. The main aim of this study is to provide such an
extensive Monte Carlo investigationof the finite-sample proper ties of the EKT procedure.
In the course of this analysis, we also employ recent advances of weighting matrices and
tests for GMM estimation in the EKT framework(see Sun (2013) and Sun and Kim (2012)).
We investigate a wide range of different scenarios to shed light on the ability of the
EKT procedure to detect the asymmetry of the loss function and to explore the size and
powerproperties of the associated tests for forecast efficiency.The scenarios are specifically
designed to investigate the effects of the statistical properties of the forecast errors (such
as variance, serial correlation, fat tails) simultaneously with variations of the information
set available to the forecaster (i.e. omitting variables or inducing irrelevant variables). In
addition, we induce a single large outlier to mimic a major crisis analogous to the Great
Recession. Moreover, while most Monte Carlo exercises are based on parameter values
which are quite arbitrarily chosen or inspired by previous studies in the literature, we also
use the forecast errors from a predictive regression equation estimated with real data to
obtain more realistic parameter values and thereby get closer to evaluating the properties
of the EKT procedure in applied situations.
The investigation starts in the following section II with the formal statement of the
EKT loss function and the outline of the estimation and test procedure. This section also
contains a review of previouswork and describes the design of the Monte Carlo experiment.
In the following section III the results are presented using a specific graphical device which
we call the fishbone plot. The presentation starts with the discussion of the results from
a baseline scenario and then proceeds to the more realistic scenario based on real data.
Several variations of the baseline scenario can be found in the online appendix. Section IV
concludes the study with a summary of the main lessons for applied work and suggestions
for future research.
II Methodology
Before we turn to the discussion of the results in the next section we present the family of
loss functions introduced by EKT, give an overview of previous simulation studies, explain
the recent advances of GMM estimation applied later and finally outline the design of our
Monte Carlo experiment.
Loss function
The following analysis is based on the approach proposed by Elliott et al. (2005, 2008),
which is focused on gaining insights into forecaster preferences concerning the asymmetry
of the underlying loss function L(.) from a given series of forecast errors et+1. The forecast
©2018 The Department of Economics, University of Oxford and JohnWiley & Sons Ltd
To continue reading
Request your trial