Which structure of academic articles do referees pay more attention to?: perspective of peer review and full-text of academic articles
| Date | 01 September 2022 |
| Pages | 884-916 |
| DOI | https://doi.org/10.1108/AJIM-05-2022-0244 |
| Published date | 01 September 2022 |
| Author | Chenglei Qin,Chengzhi Zhang |
Which structure of academic
articles do referees pay more
attention to?: perspective of peer
review and full-text of
academic articles
Chenglei Qin
School of Economics and Management,
Nanjing University of Science and Technology, Nanjing, China, and
Chengzhi Zhang
School of Economics and Management,
Nanjing University of Science and Technology, Nanjing, China and
Key Laboratory of Rich-media Knowledge Organization and Service of Digital
Publishing Content, Institute of Scientific and Technical Information of China,
Beijing, China
Abstract
Purpose –The purpose of this paper is to explore which structures of academic articles referees would pay
more attention to, what specific content referees focus on, and whether the distribution of PRC is related to the
citations.
Design/methodology/approach –Firstly, utilizing the feature words of section title and hierarchical
attention network model (HAN) to identifythe academic article structures. Secondly, analyzing the distribution
of PRC in different structures according to the position information extracted by rules in PRC. Thirdly,
analyzing the distribution of feature words of PRC extracted by the Chi-square test and TF-IDF in different
structures. Finally, four correlation analysis methods are used to analyze whether the distribution of PRC in
different structures is correlated to the citations.
Findings –The count of PRC distributed in Materials and Methods and Results section is significantly more
than that in the structure of Introduction and Discussion, indicating that referees pay more attention to the
Material and Methods and Results. The distribution of feature words of PRC in different structures is obviously
different, which can reflect the content of referees’concern. There is no correlation between the distribution of
PRC in different structures and the citations.
Research limitations/implications –Due to the differences in the way referees write peer review reports,
the rules used to extract position information cannot cover all PRC.
Originality/value –The paper finds a pattern in the distribution of PRC in different academic article
structures proving the long-term empirical understanding. It also provides insight into academic article
writing: researchers should ensure the scientificity of methods and the reliability of results when writing
academic article to obtain a high degree of recognition from referees.
Keywords Peer review, Distribution of peer review comments, IMRaD, Citations
Paper type Research paper
Introduction
The peerreview mechanism playsa critical role in scientificcommunicationand is the closest to
the actual stateof the evaluated object (Narin, 1978), whichis a practicalguarantee of scientific
quality (Mark Ware Consulting, 2016;P€
oschl, 2004). As a carrier forrecording the peer review
AJIM
75,5
884
This study is supported by the National Natural Science Foundation of China (Grant No. 72074113) and
the opening fund of the Key Laboratory of Rich-media Knowledge Organization and Service of Digital
Publishing Content (Grant No. zd2022-10/02).
The current issue and full text archive of this journal is available on Emerald Insight at:
https://www.emerald.com/insight/2050-3806.htm
Received 9 May 2022
Revised 5 August 2022
Accepted 12 August 2022
Aslib Journal of Information
Management
Vol. 75 No. 5, 2023
pp. 884-916
© Emerald Publishing Limited
2050-3806
DOI 10.1108/AJIM-05-2022-0244
process, the review report records the evaluation of the manuscript, which contains rich
information such as views, sentiment and expert knowledge. For a long time, due to the
limitationsof the traditional peer reviewmechanism, there has been no large-scale open access
to peer review comments (PRC). And limited by the development of text mining technology,
researcherscannot unveil the mystery of peer reviewfrom the perspective of text content.
Due to the lengthyreview cycle and prejudiceof the traditional peer reviewmechanism, the
open peer review came into being further to improve the fairness and transparency of the
review process. Some influential journals, well-known publishing groups and scientific
research institutions began to adopt the open peer review as early as 2001. ACP journal
(Atmospheric Chemistry and Physics), belongs to the European Geosciences Union (EGU),
opened the interaction among referees, authors and public. Then, the journal published the
interactioncontent together with papers (P€
oschl, 2004,2012).As of December 2020, 19 journals
of the EGU have adoptedthe peer review mechanism. In January2016, Nature Communication
began implementing a transparent review mechanism, publishing referee comments, author
rebuttal letters and editordecisions [1]. The journal was an early signatory to the open letter
Open letter on the publication of peer review reports [2] published by ASApbio (Accelerating
Science and Publication in Biology) in 2018. To date,377 journals have signed and executed the
openletter. In May 2019, all PloS journals offered authorsthe option to publishtheir peer review
history alongside thei r accepted manuscripts (Madison, 2019). In December 2019, Nature
Research added eight new journals, including Natureand Nature Biomedical Engineering, to
adopt the transparent peer review mechanism [3]. In addition, the OpenReview.net platform,
foundedin 2013, provides a configurablesolution to peer reviewfor conferences and journalsto
promote opennessin scientific communication.As seen above, open peer review is a trend, and
the open access of PRC provides a data basis for analyzing and mining thepeer review.
With the rapid development of deep learning, text mining technology has significantly
improved its performance in text representation, text classification and other text processing
tasks. It is now time to conduct research on peer review from the perspective of text content.
Three research questions will be explored in this paper:
RQ1. Which article structures do referee pay more attention to?
RQ2. What specific content do referees focus on in different structures?
RQ3. Is the distribution of PRC in different structures related to the citations?
We take the original manuscripts and PRC published in ACP from 2001 to 2016 as the
research objects, and mainly utilize the hierarchical attention network (HAN) model (Yang
et al., 2017) to identify the article structures. Then, we study the distribution of PRC in
different structures of academic articles. Moreover, we combined the Chi-square test (Said
et al., 2020) with TF-IDF (Singhal, 2001) to study the distribution of feature words of PRC in
different structures. In addition, we employ the Spearman correlation coefficient, cumulative
distribution function, K-S test and negative binomial regression to analyze whether the
distribution of PRC in different structures correlates with the citations.
Related work
The research in this paper mainly involves the recognition of academic article structures, the
mining of PRC and the relationship between the distribution of PRC and the citations. This
section briefly introduces the related work of these three aspects.
Academic article structures recognition
Academic articles of social sciences, natural sciences, engineering and computer science
usually can be divided into four parts: Introduction, Materials and Methods, Results and
Perspective of
peer review
mining
885
Discussion, denoted as IMRaD or IMRD(Williams, 2018), and the taking shape can be found in
the book Etudessur la Biere published in 1876 (Day, 1989;Pasteur, 1876). The IMRaDstructure
intuitively reflects the process of scientific discovery (George Mason University Writing
Center, 2014). The standardization structure of articles ensures the effective communication
and disseminationof scientific discoveries in the academic community,which is convenient for
readers to read papers from different perspectives and find relevant information from a
specific location (Parlindungan, 2012). Wu (2011) has a similar view that scientific progress
depends on a rigorous publishing process. The IMRaD structure can help authors organize
content and help editorsand referees evaluate manuscripts. For readers, theIMRaD structure
can help locate specific information efficientlywithout browsing the entire paper. In addition,
Teodosiu (2019)thinks that the IMRaD structure is the best form of expression for articles, and
a clear and standardizedstructure helps to publish papers. As early as the1970s, the American
NationalStandard Institute (ANSI) and the InternationalCommittee of Medical Journal Editors
(ICMJE) used the IMRaD structure asa standard, and it became the written format for most
journals (International Committee ofMedical Journal Editors, 1991).
The recognition of the structural function of sentences in academic articles was concerned
firstly by researchers. As early as 2003, McKnight and Srinivasan (2003) divided the
sentences of 7,253 abstracts into four functions: Instruction, Method, Result and Conclusion.
They used the bag-of-words model and trained classifiers to predict the function of sentences
in unstructured abstracts. The experiments showed that the method achieved good results.
Agarwal and Yu (2009) explored whether sentences can be divided into IMRaD structures in
the biomedical field with a consistency of 82.14% annotation. They then tested the effect of
rules-based, SVM and other methods to identify the sentence functions, and the paper found
the performance of Naı€ve Bayes models was better than others. Nam et al. (2016) tested the
effect of the bag-of-words, language features, grammatical features and structural features in
the function recognition of summary sentences. They found that language features can
contribute to classify sentence functions.
The current typical approach to identifying the academic article structures is to identify
which structure the article section belongs to. Researchers can divide the article structures in
some fields according to the section title. Hu et al. (2013) explored the distribution of citations
in the body of scientific articles based on 350 full text articles from the Journal of Informetrics.
They divided the articles into four parts based on the section title, namely, Introduction,
Method, Results and Conclusions (IMRC). Zhang et al. (2021a,b) explored the distribution of
adverbs and adjectives used by reviewers in different structures of academic papers based on
3,329 review reports from the British Medical Journal. They divided the structures into seven
parts by manual: Overall, Abstract, Introduction, Methods, Results, Discussion and Other.
Usually, if the structure of the paper is clear, it is not necessary to automatically identify
the structure of academic articles, such as those from PLoS. However, there is a situation that
the paper structure cannot be directly judged according to the section title. Ribeiro et al. (2018)
thought that reading efficiency would be improved if researchers could identify the article
structures and extract it to the readers who need part of the structures. They tested the
performance of five classifiers in recognizing the 129 articles structures from the PubMed
Central database. They found that the Voting Feature Intervals algorithm performed best,
and the best result of the accuracy value is 71.38%. Ahmed and Afzal (2020) thought that
term-based literature retrieval could not meet special needs. For example, the current retrieval
system cannot return the literature containing a specific term (e.g. “PageRank”) in the
structure of the results. Traditional paper structure recognition methods based on keywords,
paper templates and references ignore some valuable features. They utilized the features of
in-text citation count, figure count and table count and section subheadings to map the
structural function of section to IMRaD based on 5,000 articles from CiteSeer. The F
1
value is
97.50% which is higher than several traditional methods. Ma et al. (2020) hold a similar view
AJIM
75,5
886
Get this document and AI-powered insights with a free trial of vLex and Vincent AI
Get Started for FreeStart Your Free Trial of vLex and Vincent AI, Your Precision-Engineered Legal Assistant
-
Access comprehensive legal content with no limitations across vLex's unparalleled global legal database
-
Build stronger arguments with verified citations and CERT citator that tracks case history and precedential strength
-
Transform your legal research from hours to minutes with Vincent AI's intelligent search and analysis capabilities
-
Elevate your practice by focusing your expertise where it matters most while Vincent handles the heavy lifting
Start Your Free Trial of vLex and Vincent AI, Your Precision-Engineered Legal Assistant
-
Access comprehensive legal content with no limitations across vLex's unparalleled global legal database
-
Build stronger arguments with verified citations and CERT citator that tracks case history and precedential strength
-
Transform your legal research from hours to minutes with Vincent AI's intelligent search and analysis capabilities
-
Elevate your practice by focusing your expertise where it matters most while Vincent handles the heavy lifting
Start Your Free Trial of vLex and Vincent AI, Your Precision-Engineered Legal Assistant
-
Access comprehensive legal content with no limitations across vLex's unparalleled global legal database
-
Build stronger arguments with verified citations and CERT citator that tracks case history and precedential strength
-
Transform your legal research from hours to minutes with Vincent AI's intelligent search and analysis capabilities
-
Elevate your practice by focusing your expertise where it matters most while Vincent handles the heavy lifting
Start Your Free Trial of vLex and Vincent AI, Your Precision-Engineered Legal Assistant
-
Access comprehensive legal content with no limitations across vLex's unparalleled global legal database
-
Build stronger arguments with verified citations and CERT citator that tracks case history and precedential strength
-
Transform your legal research from hours to minutes with Vincent AI's intelligent search and analysis capabilities
-
Elevate your practice by focusing your expertise where it matters most while Vincent handles the heavy lifting
Start Your Free Trial of vLex and Vincent AI, Your Precision-Engineered Legal Assistant
-
Access comprehensive legal content with no limitations across vLex's unparalleled global legal database
-
Build stronger arguments with verified citations and CERT citator that tracks case history and precedential strength
-
Transform your legal research from hours to minutes with Vincent AI's intelligent search and analysis capabilities
-
Elevate your practice by focusing your expertise where it matters most while Vincent handles the heavy lifting