ASSESSING THE ASSESSMENT OR, THE RAE AND THE OPTIMAL ORGANIZATION OF UNIVERSITY RESEARCH
| Date | 01 November 2008 |
| DOI | http://doi.org/10.1111/j.1467-9485.2008.00469.x |
| Author | Manfredi M. A. La Manna |
| Published date | 01 November 2008 |
ASSESSING THE ASSESSMENT OR,
THE RAE AND THE OPTIMAL
ORGANIZATION OF UNIVERSITY
RESEARCH
Manfredi M. A. La Manna
n
Abstract
The UK Research Assessment Exercise (RAE) is assessed as an incentive scheme
affecting the allocation of research talent of varying ‘quality’ across departments.
The ‘centres of excellence’ policy implicitly pursued through the RAE is an optimal
allocation strategy only if all departments in all disciplines are of the generalist
variety, i.e. each pursues a research path through all its stages. Conversely, the
RAE-induced research allocation minimizes efficiency if applied to specialist
departments, when resources are concentrated on one specific research obstacle.
It is argued that the RAE should not take the organization of University research
as exogenous, but rather should encourage specialization. All results are obtained
by applying to University research concepts and solutions borrowed from
the mathematical theory of systems reliability.
I Intro ductio n
What is going to have a direct cost of d10m with an additional indirect cost of
$35m, and to be used to allocate nearly d8000m of research funding in the
United Kingdom in 2009? The answer, of course, is the 2008 Research
Assessment Exercise (RAE), the latest round of the appraisal of University
research regarded by some as ‘the most sophisticated exercise of its type in the
world’. In 2001,
1
69 panels of experts assessed the ‘research performance’ of
2560 units of assessment (departments) from 190 British higher education
institutions, involving, among other things, the evaluation of 203,743 publica-
tions written by 50,672 active researchers.
The estimate of man- (and woman-) hours expended for the preparation and
assessment of thousands of departmental submissions – involving highly
distinguished and senior researchers – is difficult to calculate. The Higher
Education Funding Council for England estimates that RAE 2008 will have a
n
University of St Andrews
1
The figures for the 1996 RAE were: 60 panels, 2896 units of assessment, 192 HE institutions,
212,553 publications, and 55,893 researchers.
Scottish Journal of Political Economy, Vol. 55, No. 5, November 2008
r2008 The Author
Journal compilation r2008 Scottish Economic Society. Published by Blackwell Publishing Ltd,
9600 Garsington Road, Oxford, OX4 2DQ, UK and 350 Main St, Malden, MA, 02148, USA
637
direct cost of d10m, with the indirect cost sustained by universities bringing the
likely cost to d45m (source: HEFCE, 2004). These costs pale into insignificance
when compared with the d8000m of research funding that are going to be
distributed across the British HE system on the basis of the outcome of RAE
2008. Entire departments have been closed as a direct result of poor rankings in
the 1996 and 2001 RAEs. Moreover, the RAE is a critical factor in the hiring
decisions (and their timing) by Universities, especially in the period preceding
the RAE deadline for submissions (see Hayri, 1997). The quality of the debate
on the merits and demerits of the RAE has been quite disappointing, with rivers
of ink and endless Common-Room conversations devoted to the minutiae of the
assessment procedure. Academic post-mortems on the RAE have tended to
concentrate on the number and type of relevant ‘research outputs’, the revealed-
preference rankings of various academic journals, etc.,
2
rather than the
fundamental purpose of the exercise.
This paper adopts an altogether different view in so far as it side-steps the
question of how to measure ‘research performance’ and focuses directly on one
of the main problems that the RAE is (implicitly) assumed to be addressing,
namely the allocation of research resources of varying quality to a given number
of research units. Obviously, no single paper can analyse satisfactorily the
complicated interplay between the process whereby new ideas are produced
(‘research’) and the monitoring and funding of research specifically undertaken
within University departments. In order to make some progress in this under-
researched area, the present paper makes some drastic simplifications concern-
ing both the research process and the management of University research.
Specifically, out of the many characteristic features of research, the paper
focuses on just two (albeit fundamental) aspects of research, namely:
(i) the successful completion of almost any research project involves the
overcoming of more than one intellectual obstacle (the multi-task
nature of research) and
(ii) typically there is more than one way to overcome any given research
obstacle (the multi-path nature of research).
As far as the organization and management of University research is concerned,
the paper simplifies dramatically by assuming that
(iii) each discipline pursues only one (multi-task) programme;
3
(iv) attention can be restricted to research carried out by individuals
within the same department/unit;
4
2
After each RAE, most disciplines undertake some post-mortem analysis of the outcome.
For some examples from economics and management studies, see Ball and Butler (2004),
Bessant et al. (2003), Ball (1997), Taylor and Izadi (1996), Johnes (1990), Johnes and Johnes
(1993), and Doyle et al. (1996).
3
The main results of the paper still hold if multiple programmes are allowed, and indeed
would be strengthened in the presence of any inter-programme inter-task correlation.
4
In this paper I restrict the analysis to the case where the units of assessments are individual
departments and not research groupings across departments (both domestic and foreign). The
reason for this restriction is twofold: firstly, I wish to analyse the RAE scheme as is currently
MANFREDI M. A. LA MANNA638
r2008 The Author
Journal compilation r2008 Scottish Economic Society
To continue reading
Request your trial