User evaluation of a task for shortlisting papers from researcher’s reading list for citing in manuscripts
| DOI | https://doi.org/10.1108/AJIM-01-2017-0020 |
| Pages | 740-760 |
| Date | 20 November 2017 |
| Published date | 20 November 2017 |
| Author | Aravind Sesagiri Raamkumar,Schubert Foo,Natalie Pang |
| Subject Matter | Library & information science,Information behaviour & retrieval,Information & knowledge management,Information management & governance,Information management |
User evaluation of a task
for shortlisting papers from
researcher’s reading list for
citing in manuscripts
Aravind Sesagiri Raamkumar and Schubert Foo
Wee Kim Wee School of Communication and Information,
Nanyang Technological University, Singapore, Singapore, and
Natalie Pang
Nanyang Technological University, Singapore, Singapore
Abstract
Purpose –Although many interventional approaches have been proposed to address the apparent gap
between novices and experts for literature review (LR) search tasks, there have been very few approaches
proposed for manuscript preparation (MP) related tasks. The purpose of this paper is to describe a task and
an incumbent technique for shortlisting important and unique papers from the reading list (RL) of
researchers, meant for citation in a manuscript.
Design/methodology/approach –A user evaluation study was conducted on the prototype system which
was built for supporting the shortlisting papers (SP) task along with two other LR search tasks. A total of 119
researchers who had experience in authoring research papers participated in this study. An online
questionnaire was provided to the participants for evaluating the task. Both quantitative and qualitative
analyses were performed on the collected evaluation data.
Findings –Graduateresearch students prefer this taskmore than research and academic staff.The evaluation
measuresrelevance, usefulness and certaintywere identified as predictorsfor the output quality measure“go od
list”. The shortlisting feature and information cues were the preferred aspects while limited data set and rote
steps in the study wereascertained as critical aspects fromthe qualitative feedback of the participants.
Originality/value –Findings point out that researchers are clearly interested in this novel task of SP from
the final RL prepared during LR. This has implications for digital library, academic databases and reference
management software where this task can be included to benefit researchers at the manuscript preparatory
stage of the research lifecycle.
Keywords Digital libraries, Citation networks, Citation recommendation, Manuscript preparation,
Scientific paper recommender systems, Shortlisting citations
Paper type Research paper
1. Introduction
The scientific research lifecycle encompasses the activities performed by researchers across
different disciplines. There have been multiple versions of this lifecycle put forth in previous
studies. The term “Scientific Publication Lifecycle (SPLC)”was used for referring to the
lifecycle of a publication (journal article) in Björk and Hedlund (2003). The term “Scholarly
Communication Lifecycle”has been loosely used in certain studies (Harley et al., 2010;
Wright et al., 2007) to highlight the different avenues of publication such as traditional
journals, open-access journals and pre-print services. The research lifecycle schematic put
forth in Nicholas and Rowlands (2011) was for collecting data on social media use in
the research workflow. The eight steps in this schematic are an adequate representation of
the lifecycle. The eight steps are identify research opportunities, find collaborators, secure
Aslib Journal of Information
Management
Vol. 69 No. 6, 2017
pp. 740-760
© Emerald PublishingLimited
2050-3806
DOI 10.1108/AJIM-01-2017-0020
Received 12 January 2017
Revised 28 April 2017
6June2017
Accepted 1 September 2017
The current issue and full text archive of this journal is available on Emerald Insight at:
www.emeraldinsight.com/2050-3806.htm
This research was supported by the National Research Foundation, Prime Minister’s Office, Singapore
under its International Research Centres in Singapore Funding Initiative and administered by the
Interactive Digital Media Programme Office.
740
AJIM
69,6
support, review the literature, collect research data, analyse research data, disseminate
findings and manage the research process. Among these activities, the three major activities
are literature review (LR), actual research work and dissemination of results through
publication venues. These three activities comprise of multiple sub-activities that require
specific expertise and experience (Levy and Ellis, 2006). Information behaviour research
has shown that researchers with low experience face difficulties in completing research
related activities (Du and Evans, 2011; Karlsson et al., 2012). These researchers rely on
assistance from supervisors, experts and librarians for learning the required skills to pursue
such activities.
Apart from above mentioned process-based human interventions, technology-oriented
interventionssuch as academic assistive systems havebeen built for alleviating the expertise
gap between experts and novices in terms of research execution (Raamkumar et al., 2016).
As a part of technology-oriented interventions, recommender systems (RS) techniques have
been proposed to recommend scholarly information objects for a wide variety of tasks in the
research lifecycle. Among the scholarly information objects, research papers are the most
recommendedobjects in prior studies as papers arerequired for performing LR and for ad-hoc
information needs. Other information objects that have been recommended to researchers in
prior RS studies include collaborators (Gunawardena and Weber, 2009), co-authors (Sie et al.,
2014) and publication venues (Beierle et al., 2016; Chen et al., 2015). In the research lifecycle,
RS studies have majorly focussed on the search tasks conducted during the LR stage. Such
tasks include building an initial reading list (IRL) at the start of LR (Ekstrand et al., 2010;
Jardine, 2014) and finding similar papers for seed paper(s) (Küçüktunç et al., 2015;
Huynh et al., 2012; Liang et al., 2011).
During the dissemination stage of the research lifecycle, researchers engage in
manuscript preparation (MP) and writing for corresponding publication venues. For this
phase, one unexplored area is helping researchers in identifying the important and unique
papers that can be potentially cited in the manuscript. This identification is influenced by
two factors. The first factor is the type of research for which citation of a particular paper
makes sense due to the particular citation context[1]. The second factor is the type of
article (e.g. conference full paper (cfp), journal paper and demo paper) that the author is
intending to writ e. For the first fac tor, there have been previous studies (He et al., 2011;
Huang et al., 2014; Shaoping, 2010). The second factor represents a task that can be
explored since the article-type places a constraintonthecitationsthatcanbemadeina
manuscript, through dimensions such as recency, quantity, to name a few. This factor
forms the focal point of the current study.
In this study, we seek to address the manuscript preparatory task of shortlisting papers
(SP) from the final reading list (FRL)[2] of researchers based on article-type preference.
We first describe the task and then propose a shortlisting technique for operationalizing the
task. By the term “shortlisting”, we allude to the nature of the task in identifying important
papers from the reading list (RL). We implemented a recommendation technique for this SP
task as part of the functionality provided by an assistive system called Rec4LRW.
This system was developed for providing recommendations for two other LR tasks along
with the SP task. The two LR tasks are building an IRL of research papers and finding
similar papers based on a set of papers. The system’s corpus of 103,739 articles and
2,320,345 references was built using an extract of ACM Digital Library (ACM DL).
A user evaluation studywas conducted to evaluate the recommendations of the tasks and
the overall Rec4LRW system, from the researcher’s perspective. In this paper, we report the
findings for the SP task from the evaluati on study. The study was conducted with 119
participants comprising of graduate research students, academic staff and research staff.
The participants,who had experience in publishing researcherpapers, were selected through
a screening survey.Participants had to select one of the provided 43 research topics and run
741
User
evaluation
To continue reading
Request your trial