Innovating peer review, reconfiguring scholarly communication: an analytical overview of ongoing peer review innovation activities
| DOI | https://doi.org/10.1108/JD-01-2022-0022 |
| Published date | 03 August 2022 |
| Date | 03 August 2022 |
| Pages | 429-449 |
| Subject Matter | Library & information science,Records management & preservation,Document management,Classification & cataloguing,Information behaviour & retrieval,Collection building & management,Scholarly communications/publishing,Information & knowledge management,Information management & governance,Information management,Information & communications technology,Internet |
| Author | Wolfgang Kaltenbrunner,Stephen Pinfield,Ludo Waltman,Helen Buckley Woods,Johanna Brumberg |
Innovating peer review,
reconfiguring scholarly
communication: an analytical
overview of ongoing peer review
innovation activities
Wolfgang Kaltenbrunner
Centre for Science and Technology Studies, Leiden University,
Leiden, The Netherlands
Stephen Pinfield
The University of Sheffield Information School, Sheffield, UK
Ludo Waltman
Centre for Science and Technology Studies, Leiden University,
Leiden, The Netherlands
Helen Buckley Woods
The University of Sheffield Information School, Sheffield, UK, and
Johanna Brumberg
VolkswagenStiftung, Hannover, Germany
Abstract
Purpose –The study aims to provide an analytical overview of current innovations in peer review and their
potential impacts on scholarly communication.
Design/methodology/approach –The authors created a survey that was disseminated among publishers,
academic journal editors and other organizations in the scholarly communication ecosystem, resulting in a data
set of 95 self-defined innovations. The authors ordered the material using a taxonomy that compares
innovation projects according to five dimensions. For example, what is the object of review? How are reviewers
recruited, and does the innovation entail specific review foci?
Findings –Peer review innovations partly pull in mutually opposed directions. Several initiatives aim to
make peer review more efficient and less costly, while other initiatives aim to promote its rigor, which is
likely to increase costs; innovations based on a singular notion of “good scientific practice”are at odds
with more pluralistic understandings of scientific quality; and the idea of transparency in peer review is
the antithesis to the notion that objectivity requires anonymization. These fault lines suggest a need for
better coordination.
Originality/value –This paper presents original data that were analyzed using a novel, inductively
developed, taxonomy. Contrary to earlier research, the authors do not attempt to gauge the extent to
which peer review innovations increase the “reliability”or “quality”of reviews (as defined according to
often implicit normative criteria), nor are they trying to measure the uptake of innovations in the
Overview of
peer review
innovation
activities
429
© Wolfgang Kaltenbrunner, Stephen Pinfield, Ludo Waltman, Helen Buckley Woods and Johanna
Brumberg. Published by Emerald Publishing Limited. This article is published under the Creative
Commons Attribution (CC BY 4.0) licence. Anyone may reproduce, distribute, translate and create
derivative works of this article (for both commercial and non-commercial purposes), subject to full
attribution to the original publication and authors. The full terms of this licence may be seen at http://
creativecommons.org/licences/by/4.0/legalcode
Data availability: The survey responses are available on Figshare (Kaltenbrunner et al., 2022).
The current issue and full text archive of this journal is available on Emerald Insight at:
https://www.emerald.com/insight/0022-0418.htm
Received 27 January 2022
Revised 6 June 2022
Accepted 28 June 2022
Journal of Documentation
Vol. 78 No. 7, 2022
pp. 429-449
Emerald Publishing Limited
0022-0418
DOI 10.1108/JD-01-2022-0022
routines of academic journals. Instead, they focus on peer review innovation activities as a distinct
object of analysis.
Keywords Peer review, Innovation, Survey, Scholarly communication, Infrastructure
Paper type Research paper
1. Introduction
In the last two decades, we have witnessed significant efforts to innovate peer review practices in
the publishing domain. While the terminology to describe innovations is often inconsistent
(Ross-Hellauer, 2017;Tennantet al., 2017), we can readily identify initiatives experimenting with
different forms of transparency in the review process (“open peer review”), efforts to make
accessible research outputs prior to peer review in the shape of so-called preprints, new forms of
recruiting reviewers (“crowd review”), as well as a host of digital tools to support editors and
reviewers in detecting plagiarism and other forms of misconduct. In this paper, we present the
results of a survey designed to create an overview of ongoing projects to innovate peer review
practices. The survey was an initiative of the Research on Research Institute (RoRI), in which our
research team is developing a collaborative research agenda with publishers and other
organizations in the scholarly communication system. The survey was disseminated among
diverse stakeholders, including journal editors, smaller and larger comm ercial publishers, not-
for-profit or non-traditional commercial actors, as well asvarious public organizations.
Contrary to earlier research, we are not primarily interested in studying the extent to which
peer review innovations increase the “reliability”or “quality”of reviews (Bruce et al., 2016;
Tennant et al., 2017), nor are we trying to measurethe uptake of innovations in the routines of
academic journals (Wolfram et al.,2020;Horbach and Halffman, 2020). Instead, our data set
focuses our analytical attention on peer review innovation activities as a distinct object of
analysis. Our analysis will unpack in significant detail how the innovation projects reported to us
as part of the survey purport to configure current review practices, both in terms of the stated
intentions of their developers and in terms of their broader implications for disseminating
scientific research. For this purpose, we rely on an inductively developed taxonomy that
describes and compares innovation projects in terms of five main dimensions. For example, what
is the objectof review in a given project? How arereviewers recruited,and does the innovation
entail specific review foci?
Our work is restricted to peer review in the context of scholarly publishing and scholarly
communication more generally. Peer review in other contexts, such as peer review of grant
proposals and peer review in research assessment settings, falls outside the scope of our work.
The paper is structured as follows. We firstly situate our analysis in a literature review of
research on peer review innovations, showing exactly how our approach complements
previous attempts to create overviews of innovation activity. We then introduce the details of
the survey we used to collect empirical material, as well as the inductive taxonomical
framework we relied on to order it. In the main empirical section of the paper, we provide a
narrative summary of our material, in which we compare how the various innovation projects
configure review processes. In a concluding section, we attempt to observe cross-cutting
trends across diverse types of innovations, which will allow us to identify the main thrusts of
innovation and some of the tensions between them, and which will provide the background
for recommendations on how to better coordinate ongoing initiatives.
2. Peer review innovations as an object of study
While often criticized for its shortcomings, peer review continues to be widely seen as one
among a range of distinct features that help set scientific knowledge apart from other forms of
knowledge, thus warranting its special status in modern societies (Hackett and Chubin, 1990;
JD
78,7
430
To continue reading
Request your trial