Ranking Economics Journals Using Data From a National Research Evaluation Exercise
| Published date | 01 October 2017 |
| DOI | http://doi.org/10.1111/obes.12185 |
| Date | 01 October 2017 |
621
©2017 TheAuthors. OxfordBulletin of Economics and Statistics published by Oxford University and John Wiley & Sons Ltd.
Thisis an open access article under the ter ms of the CreativeCommons Attribution License, which permits use, distribution and reproduction in any medium, provided
the original work is properlycited.
OXFORD BULLETIN OF ECONOMICSAND STATISTICS, 79, 5 (2017) 0305–9049
doi: 10.1111/obes.12185
Ranking Economics Journals Using Data From a
National Research Evaluation Exercise*
Arne Risa Hole
Department of Economics, University of Sheffield, 9 Mappin Street, Sheffield, S1 4DT, UK
(e-mail: a.r.hole@sheffield.ac.uk)
Abstract
This paper describes an algorithm for creating a ranking of economics journals, using data
from the 2014 UK Research Excellence Framework(REF) exercise. The ranking generated
by the algorithm can be viewed as a measure of the average quality of the papers published
in the journal, as judged by the REF Economics and Econometrics sub-panel, based on the
outputs submitted to the REF.
I. Introduction
Research evaluations are increasingly used to inform the allocation of public research
funding. A recent large-scale example is the 2014 Research Excellence Framework (REF),
which evaluated a sample of research conducted by researchers in UK higher education
institutions. As well as research outputs, which count for 65% of the overall score,
departments were also evaluated according to their research environment (15%) and their
non-academic impact (20%). The latter is the main difference between the REF and the
2008 Research Assessment Exercise (RAE), as the RAE did not evaluate departments on
the basis of non-academic impact.
The quality of each submitted output (i.e. a journal article, working paper, book chapter
or authored book) was assessed by the members of one of 36 sub-panels and given an
individual score. Out of the 2600 outputs submitted to the Economics and Econometrics
sub-panel 28% of outputs were classified as ‘world-leading’(4*), 49% as ‘inter nationally
excellent’ (3*), 20% as ‘recognisedinter nationally’ (2*) and 3% as ‘recognised nationally’
(1*).1The scores were made publicly available at the level of the department only, along
with a list of the outputs submitted by each department.2
This paper uses the publicly available data on research outputs from the REF 2014
to construct a ranking of economics journals. This is done by using a simple algorithm
JEL Classification numbers: A1.
*I thankAndy Dickerson, Maria Jos ´e Gil Molt´o, Alberto Montagnoli, Jenny Roberts, Karl Taylor,Nic Vande Sijpe,
Peter Wright and twoanonymous reviewers for very helpful comments and suggestions. I take full responsibility for
any errors.
1In addition, a small number of outputs were ‘unclassified’.
2The individual scores assigned to each output weredestroyed once the results for the departments were complete.
622 Bulletin
which allocates a rank to each submitted output based on the journal it was published in
and compares the predicted share of outputs in the different categories at the departmental
level to the actual shares. The algorithm systematically changes the rank of the journals
to find the combination that best reproduces the actual department-level shares. To my
knowledge, this is the first attempt to construct a ranking of economics journals using data
from the REF, or any other research evaluation, complementing analysis of such data in
other fields (see e.g. Varin, Cattelan and Firth, 2016).
The ranking generated by the algorithm can be viewed as a measure of the average
quality of the papers published in the journal, as judged by the REF sub-panel, based on
the outputs submitted to the REF. Since the outputs were assessed individually it should
not be viewed as an attempt to construct an ‘official’UK ranking of economics journals. It
is likely that the actual scores given to the outputs bythe sub-panel members varied among
papers published in the same journal, which is why the ranking is best viewed as an attempt
to infer the average quality of the papers published in the journal.
The paper is organized as follows. Section II describes the algorithm used to construct
the journal ranking and section III presents the ranking along with some robustness checks.
Section IV offers some concluding remarks.
II. Methodology
Each of the 2,600 outputs submitted to the Economics and Econometrics REF sub-panel
is given an initial rank. Using the initial ranks assigned to each paper, we can predict
the proportion of 4*, 3*, 2* and 1* submissions for each department. We then calculate
the squared difference between the predicted and actual proportions for each category and
sum the squared differences overdepartments (i=1, 2,…, 28) and categories (j=1, 2,…, 4).
The sum of squared differences (SSD) weighted by the number of submissions from each
department (Ni)isgivenby:
SSD =
28
i=1
4
j=1
Ni(pij −ˆpij)2(1)
where pij is the actual proportion of j-star submissions in department i, and ˆpij is the
predicted proportion:
ˆpij =
1
Ni
Ni
n=1
I(rni =j), (2)
where rni is the rank assigned to output nsubmitted by department i.I(·) is the indicator
function which is equal to one if the expression in the parenthesis is true and zero otherwise.
The journals are then sorted in random order and the following algorithm is run:
1. Star ting with the first journal re-calculate the SSD after temporarily assigning the
journal each of the four possible ranks
2. Assign the journal the rank which leads to the lowest SSD in step 1
3. Repeat steps 1 and 2 for all the journals in the ranking
4. Repeat steps 1 to 3 until the algorithm converges. Convergence is declared when an
iteration (i.e. a run through steps 1 to 3) decreases the SSD by <0.0001.
©2017 The Authors. Oxford Bulletin of Economics and Statistics published by Oxford University and JohnWiley & Sons Ltd.
To continue reading
Request your trial