‘Environment’ Submissions in the UK's Research Excellence Framework 2014
| Date | 01 July 2018 |
| Published date | 01 July 2018 |
| DOI | http://doi.org/10.1111/1467-8551.12248 |
British Journal of Management, Vol. 29, 571–587 (2018)
DOI: 10.1111/1467-8551.12248
‘Environment’ Submissions in the UK’s
Research Excellence Framework 2014
Andy Thorpe, Russell Craig, Dennis Tourish,1Glenn Hadikin2
and Sasa Batistic3
Portsmouth Business School, University of Portsmouth,UK, 1University of Sussex, UK, 2School of Languages
and Area Studies, University of Portsmouth, UK, and 3Department of Human Resource Studies, Tilburg
University, The Netherlands
Corresponding author email: andy.thorpe@port.ac.uk
There has been much debate about university research assessment exercises. In the UK,
a major element of the 2014 Research Excellence Framework (REF2014) has been the
research environment. Here we analyse 98 REF2014 ‘environment’ submissions in Busi-
ness and Management Studies. We explorewhether there are distinctive language-related
dierences betweensubmissions of high and low ranked universities and conclude that sub-
mission writers have a strong incentiveto exaggerate strengths and conceal problems. In
addition, innate biases such as the ‘halo’ and ‘velcro’ eects may distractthe attention of
assessors from a submission’s strengthsand weaknesses, since they are likely to influence
their pre-existing impressions. We propose several changes to improve how environment
is evaluated. We also argue that the researchenvironment would be more likely to be en-
hanced if the number of outputs submitted in future was an averageof two and a maximum
of four per academic, rather than the maximum of five currently being considered.
Introduction
Linking research funding to evaluations of the
perceived quality of research reflects embrace of
a ‘new public management’ (NPM) mentality by
higher education policymakers (Craig, Amernic
and Tourish, 2014). NPM assumes that the pro-
motion of markets, managers and measurement
improves performance in the public sector (Ferlie
et al., 1996). In the higher education sector, the
NPM approach assumes that the quality of re-
search can be quantified and measured accurately.
We subject this assumption to critical inquiry by
focusing on submissions regarding the (research)
‘environment’ that were made in the UK’s Re-
search Evaluation Framework in 2014 (REF2014).
The need for such analysis arises from the grow-
ing complexity and costs imposed by successive
research assessment exercises. REF2014 was esti-
mated to cost £250 million (Times Higher Edu-
cation, 2015). Thirty-six expert panels completed
peer reviews in their respective Unit of Assessment
(UoA) of submissions regarding ‘outputs’ (65%),
‘impact’ (20%) and ‘environment’ (15%).
We focus on the language used in the ‘en-
vironment’ element of REF2014 in submissions
to UoA19, Business and Management Stud-
ies (B&M). Although the impact component of
REF2014 has been subjected to detailed scholarly
scrutiny (Derrick and Samuel, 2016; Kellard and
Sliwa, 2016; Manville et al., 2015), the environ-
ment element has not. Here we explore how re-
search environment submissions extolled virtues,
minimized diculties, inflated performance out-
comes, and crafted what their authors hoped was
a compelling story. Our analysis supports emerg-
ing critiques of research assessment exercises for
inaccurately measuring quality, costs and eects.
Those critiques havealso explored how assessment
exercises might be improved (Geuna and Piolatto,
2016; Linkova, 2014; Mingers and White, 2015;
Saunders, Wong and Saunders, 2011).
We were motivated to conduct the present
study by three factors. First was the disturbing
© 2017 British Academy of Management. Published by John Wiley & Sons Ltd, 9600 Garsington Road, Oxford OX4
2DQ, UK and 350 Main Street, Malden, MA, 02148, USA.
572 A. Thorpe et al.
conclusion of assessor bias by Taylor (2011,
p. 211) with respect to the Research Assessment
Exercise (RAE) 2008 for ‘Research environment
and esteem’ in UoAs for Accounting and Fi-
nance, Business and Management, and Economics
and Econometrics.1We were curious to explore
whether the same alleged bias persisted in the
REF2014 assessment of environment. Our curios-
ity was piqued by knowledge that 12 of the 17
academic assessors for RAE2008 were among the
24 academic assessors in B&M for REF2014.
Second, a Higher Education Funding Council
for England (HEFCE) report expressed concerns
that‘ ...the narrative elements[of REF2014 sub-
missions] were hard to assess, with diculties in
separating quality in research environment from
quality in writing about it’ (Wilsdon et al., 2015,
p. 129). This concern was reiterated forcefully in
the consultation document on the next UK REF
exercise (HEFCE, 2016, note 112).
A third motivator was the extent to which
quantified metrics appeared to influence REF2014
B&M environment scores. We performed a multi-
variate regression (ordinary least squares) analy-
sis, similar to that of Taylor (2011), with the en-
vironment grade point average (GPA) score as the
dependent variable. Like Taylor, our explanatory
variables included size (full time equivalent (FTE)
sta submitted), research income per FTE and
number of postgraduate research degree (PGR)
completions per FTE. We also inserted dummy
variables to control for whether the higher educa-
tion institution (HEI) was a member of the Russell
Group of Universities (designated as ‘Russell’) or
had an assessor (‘Assessor’) on the panel.
Consistent with Taylor (2011), we found that
PGR completions had no significant eect on GPA
scores. The model of best fit revealed no evidence
of multicollinearity. This model included FTEs
submitted, research income per FTE, Russell and
Assessor. Our findings mirror Taylor’s inasmuch
as both income per FTE and FTE submitted were
highly significant (1% level) but diverged in terms
of the membership dummies. In RAE2008, Tay-
lor (2011) found the eects of RussellGroup mem-
bership was also highly significant (1% level) and
generated a GPA premium of 0.43 points. This led
him to speculate that such universities may have
1Taylor also found that membership of the ‘1994 Group’
of ‘research-intensive’ universities had a strong impact
(0.1% significance) on GPA scores in RAE2008.
benefited from ‘a “halo eect” independent of their
recent research activity’ (p. 214). Our research in-
dicates that Russell Group membership was less
important and the GPA premium considerably
less (0.21 points) in REF2014. In contrast, while
Taylor(2011, p. 211) reported ‘zero panel member-
ship bias across all three research profiles [UoAs]’
in RAE2008, we found having a panel member was
more strongly significant, contributing to an in-
crease of 0.375 in recorded GPA.2
Importantly, the explanatory power of our re-
gression (62.9%) suggests that the accompanying
narrative played an important role in determin-
ing REF2014 environment scores. This finding
prompted us to study the role of narrative in influ-
encing research environment scores. The research
question we therefore explore is: are REF2014
environment submissions in B&M distinctively
dierent in language-related characteristics be-
tween high ranked and low ranked university
submissions?
This question has important implications. If as-
sessment ratings are associatedpositively with nar-
rative, this should invite re-thinking of whether to
include environment submissions (in their current
form) in future RAEs.
We begin by noting the strong parallels be-
tween university impression management (exer-
cised through ‘environment’ submissions) and
methods of corporate reputation building through
language use.We then explain our research method
before discussing results. We conclude that strong
consideration should be given to re-thinking how
environment is assessed in the future and oer
some policy recommendations to help promote
debate.
Impression management and language
choice
The relationship between language choice and
corporate reputation has been explored exten-
sively in marketing and businessmanagement (e.g.
Amernic and Craig, 2007; Geppert and Lawrence,
2008). Impression management theory explains
that text can be manipulated deliberately by
2Followingthis reasoning, Sussex, Liverpool and Stirling
(all with GPA=3.125) would have replaced Birmingham,
Loughborough and Warwickin quartile 1 (see Table 1) if
they had an assessor on the UoA19 panel.
© 2017 British Academy of Management.
To continue reading
Request your trial