Terms in journal articles associating with high quality: can qualitative research be world-leading?
| Date | 07 February 2023 |
| Pages | 1110-1123 |
| DOI | https://doi.org/10.1108/JD-12-2022-0261 |
| Published date | 07 February 2023 |
| Subject Matter | Library & information science,Records management & preservation,Document management,Classification & cataloguing,Information behaviour & retrieval,Collection building & management,Scholarly communications/publishing,Information & knowledge management,Information management & governance,Information management,Information & communications technology,Internet |
| Author | Mike Thelwall,Kayvan Kousha,Mahshid Abdoli,Emma Stuart,Meiko Makita,Paul Wilson,Jonathan M. Levitt |
Terms in journal articles
associating with high quality:
can qualitative research
be world-leading?
Mike Thelwall, Kayvan Kousha, Mahshid Abdoli, Emma Stuart,
Meiko Makita, Paul Wilson and Jonathan M. Levitt
Statistical Cybermetrics Research Group, School of Technology,
University of Wolverhampton, Wolverhampton, UK
Abstract
Purpose –Scholars often aim to conduct high quality research and their success is judged primarily by peer
reviewers. Research quality is difficult for either group to identify, howeverand misunderstandings can reduce
the efficiency of the scientific enterprise. In response, we use a novel term association strategy to seek
quantitative evidence of aspects of research that are associated with high or low quality.
Design/methodology/approach –We extracted the words and 2–5-word phrases most strongly associated
with different quality scores in each of 34 Units of Assessment (UoAs) in the Research Excellence Framework
(REF) 2021. We extracted the terms from 122,331 journal articles 2014–2020 with individual REF2021 quality
scores.
Findings –The terms associating with high- or low-quality scores vary between fields but relate to
writing styles, methods and topics. We show that the first-person writing style strongly associates with
higher quality research in many areas because it is the norm for a set of large prestigious journals. We
found methods and topics that associate with both high- and low-quality scores. Worryingly, terms
associated with educational and qualitative research attract lower quality scores in multiple areas. REF
experts may rarely give high scores to qualitative or educationalresearch because the authors tend to be
lesscompetent, because it is harder to do world leading research with these themes, or because they do not
value them.
Originality/value –This is the first investigation of journal article terms associating with research quality.
Keywords Research assessment, Research quality, REF 2021, Research excellence framework, Term
frequency analysis, Bibliometrics
Paper type Research paper
Introduction
Academic research in increasingly many countries is evaluated by post-publication peer
review for quality assurance, formative feed or to direct research funding (Engels and
Guns, 2018;Woelert and McKenzie, 2018;Sivertsen, 2018;Jeon and Kim, 2018;Nielsen,
2017). The results can influence the reputations, funding, actions and careers of the
researchers involved. It is therefore important to investigate any systematic causes of
high- or low-quality scores to find areas of good or bad practice and to check for bias. This
is inherently difficult for academic peer review because even experts can strongly disagree
on what is good research. Thus, whilst quality differences or biases can be tested for in
researcher characteristics (e.g. gender, career status) or institutional status (e.g. prestige,
JD
79,5
1110
This study was funded by Research England, Scottish Funding Council, Higher Education Funding
Council for Wales and Department for the Economy, Northern Ireland as part of the Future Research
Assessment Programme (https://www.jisc.ac.uk/future-research-assessment-programme). The content
is solely the responsibility of the authors and does not necessarily represent the official views of the
funders.
The current issue and full text archive of this journal is available on Emerald Insight at:
https://www.emerald.com/insight/0022-0418.htm
Received 11 December 2022
Revised 6 January 2023
Accepted 13 January 2023
Journal of Documentation
Vol. 79 No. 5, 2023
pp. 1110-1123
© Emerald Publishing Limited
0022-0418
DOI 10.1108/JD-12-2022-0261
Get this document and AI-powered insights with a free trial of vLex and Vincent AI
Get Started for FreeStart Your Free Trial of vLex and Vincent AI, Your Precision-Engineered Legal Assistant
-
Access comprehensive legal content with no limitations across vLex's unparalleled global legal database
-
Build stronger arguments with verified citations and CERT citator that tracks case history and precedential strength
-
Transform your legal research from hours to minutes with Vincent AI's intelligent search and analysis capabilities
-
Elevate your practice by focusing your expertise where it matters most while Vincent handles the heavy lifting
Start Your Free Trial of vLex and Vincent AI, Your Precision-Engineered Legal Assistant
-
Access comprehensive legal content with no limitations across vLex's unparalleled global legal database
-
Build stronger arguments with verified citations and CERT citator that tracks case history and precedential strength
-
Transform your legal research from hours to minutes with Vincent AI's intelligent search and analysis capabilities
-
Elevate your practice by focusing your expertise where it matters most while Vincent handles the heavy lifting
Start Your Free Trial of vLex and Vincent AI, Your Precision-Engineered Legal Assistant
-
Access comprehensive legal content with no limitations across vLex's unparalleled global legal database
-
Build stronger arguments with verified citations and CERT citator that tracks case history and precedential strength
-
Transform your legal research from hours to minutes with Vincent AI's intelligent search and analysis capabilities
-
Elevate your practice by focusing your expertise where it matters most while Vincent handles the heavy lifting
Start Your Free Trial of vLex and Vincent AI, Your Precision-Engineered Legal Assistant
-
Access comprehensive legal content with no limitations across vLex's unparalleled global legal database
-
Build stronger arguments with verified citations and CERT citator that tracks case history and precedential strength
-
Transform your legal research from hours to minutes with Vincent AI's intelligent search and analysis capabilities
-
Elevate your practice by focusing your expertise where it matters most while Vincent handles the heavy lifting
Start Your Free Trial of vLex and Vincent AI, Your Precision-Engineered Legal Assistant
-
Access comprehensive legal content with no limitations across vLex's unparalleled global legal database
-
Build stronger arguments with verified citations and CERT citator that tracks case history and precedential strength
-
Transform your legal research from hours to minutes with Vincent AI's intelligent search and analysis capabilities
-
Elevate your practice by focusing your expertise where it matters most while Vincent handles the heavy lifting