Usability evaluation with tasks characterized by the information search process. The China National Knowledge Infrastructure
Date | 01 August 2016 |
Pages | 572-587 |
DOI | https://doi.org/10.1108/EL-05-2015-0074 |
Published date | 01 August 2016 |
Author | Cuichang Ma,Shujin Cao,Tinghua Gu |
Usability evaluation with tasks
characterized by the information
search process
The China National Knowledge Infrastructure
Cuichang Ma
Library, Sun Yat-sen University, Guangzhou, China, and
Shujin Cao and Tinghua Gu
School of Information Management, Sun Yat-sen University,
Guangzhou, China
Abstract
Purpose – The purpose of this study is to present an approach to evaluating the usability of digital
libraries in terms of knowledge states within the information search process (ISP) by investigating the
connections among components with which users interact, perceived usability and task stages. The
China National Knowledge Infrastructure (CNKI), the most popular digital library in China, was chosen
to be studied.
Design/methodology/approach – With the research framework, measurements were generated for
perceived usability in task stages for efciency, effectiveness and satisfaction. Two usability
experiments and usability tests were administered to 22 subjects in two groups. Three information
retrieval tasks were designed according to the characteristics of knowledge states within the ISP for
each experiment. The transaction logs were captured and the experimental results were recorded. The
users completed a pre-search questionnaire and post-search questionnaire, and interviews were
conducted.
Findings – The study showed that both interactions on components of the digital library and usability
scores differed with sequential tasks characterized by knowledge states within the ISP; new usability
problems relating to the CNKI were found based on the empirical results.
Originality/value – Many studies on the usability evaluation of digital libraries focused on the
outcome rather than on the ISP. This study proposed an effective approach to usability evaluation
based on users’ knowledge states within the ISP.
Keywords Evaluation, Digital libraries, Information seeking behaviours, Usability engineering
Paper type Research paper
1. Introduction
To date, much of the research evaluating the usability of digital libraries (Jeng, 2006;Joo
and Lee, 2011) has focused on the development of frameworks that describe, model and
evaluate the users’ interaction with digital libraries as a whole, across a set of measures.
This paper is supported by the National Social Science Fund of China (Grant No. 14CTQ015 and
Grant No. 12&ZD222) in the entire research process. The authors are indebted to Associate
Professor Yuxiang Zhao from Nanjing University and Penghui Lu from Wuhan University for
academic support.
The current issue and full text archive of this journal is available on Emerald Insight at:
www.emeraldinsight.com/0264-0473.htm
EL
34,4
572
Received 13 May 2015
Revised 17 June 2015
Accepted 30 July 2015
TheElectronic Library
Vol.34 No. 4, 2016
pp.572-587
©Emerald Group Publishing Limited
0264-0473
DOI 10.1108/EL-05-2015-0074
Such frameworks, however, have focused on the outcome of digital libraries as a whole
rather than on the interaction process, and have not considered the components of
digital libraries; in so doing, little attention has been given to the value of bottom-up
evaluation (Balatsoukas et al., 2009).
Some researchers have attempted to synthesize theories concerning both the
information search process (ISP) and human–computer interaction to conceptualize the
usability of digital libraries, such as Ferreira and Pithan (2005) and Makri et al. (2008).
Interaction studies in the context of ISP, however, are always longitudinal because of the
time span of the ISP, and so are inefcient at evaluating usability in terms of system
design or improvement (Fuhr et al., 2007).
Characterizing usability by how a digital library supports the interactions involved
in the ISP, this paper aims to present an approach to evaluating the usability of the
components of digital libraries, with tasks characterized by knowledge states in the ISP.
Here, components are classied according to the interaction objects, including the
navigation system, information organization system, search system and general
components of the digital library as a whole. The following research questions were
explored:
RQ1. Does the distribution of interactions on components of the digital library differ
with the tasks characterized by the knowledge states in the ISP?
RQ2. Does the distribution of perceived usability differ among the tasks
characterized by the knowledge states in the ISP?
RQ3. What are the usability problems with the research system?
With this aim, the China National Knowledge Infrastructure (CNKI) (www.cnki.net) was
chosen as the focus of this study. It is the most popular digital library in China,
encompassing a wide range of disciplines with a variety of genre types. Also, usability
studies on CNKI are scarce in publications in China as well as worldwide, except for
studies on user-centred design-related issues (Qiu, 2011), which would be comparable to
the present study.
2. Literature review
2.1 Usability evaluation in the context of the information search process
Information-seeking theories, such as Kuhlthau’s process model, Ellis’s feature set and
Dervin and Nilan’s (1986) sense-making approach, could be integrated with information
retrieval in an information storage and retrieval model, as Ingwersen and Järvelin (2005)
proposed. On the operational level, Vakkari (2001) rened Kuhlthau’s model of the
information search process in the eld of information retrieval (IR) and demonstrated
that the information sought, search tactics, term choices and relevance judgements, as
well as the contributing types of information in documents, depend systematically on
the stage of the task performance process and the mental model of the searcher.
In terms of usability evaluation, Makri et al. (2008) even proposed the information
behaviour (IB) method as an approach to evaluating functionality and usability of a
digital library based on Ellis’ information behaviour theory. Several other types of
information behaviour research have been used to explore the relationship between the
user’s ISP and the digital library itself – or components of it – for usability evaluation.
For example, Kim (2002) developed the information-seeking process model as a
573
Information
search process
To continue reading
Request your trial