Subjective and objective evaluation of interactive and automatic query expansion
| Published date | 01 August 2005 |
| Date | 01 August 2005 |
| DOI | https://doi.org/10.1108/14684520510617820 |
| Pages | 374-390 |
| Author | Bracha Shapira,Meirav Taieb‐Maimon,Yael Nemeth |
| Subject Matter | Information & knowledge management,Library & information science |
Subjective and objective
evaluation of interactive and
automatic query expansion
Bracha Shapira and Meirav Taieb-Maimon
Department of Information Systems Engineering, Ben-Gurion University,
Beer-Sheva, Israel, and
Yael Nemeth
Department of Industrial Engineering & Management, Ben-Gurion University,
Beer-Sheva, Israel
Abstract
Purpose – Query expansion and query limitation are two known techniques for assisting users to
define efficient queries. The purpose of this article is to examine the effectiveness of the two methods.
Design/methodology/approach – The research entailed an objective and subjective evaluation of
the effectiveness of automatic and interactive query expansion and of two query limit options. The
evaluation included both lab simulations and large-scale user studies. The objective aspects were
evaluated in lab simulations with experts judging user performance. The subjective analysis was
carried out by having the participants evaluate the quality of, and express their satisfaction with, the
retrieval process and its results, thus employing perceived-value analysis.
Findings – The main findings reveal a difference between the perceived and real values of these
techniques. While users expressed their satisfaction with interactive query expansion and its
performance, the real-value analysis of their performance did not show any significant difference
between the retrieval modes.
Originality/value – The article evaluates the objective and subjective effectiveness of automatic and
interactive query expansion and two query limit options.
Keywords Information retrieval, User studies, Projectevaluation
Paper type Research paper
Introduction
A current trend in the information retrieval (IR) field is to improve search results by
assisting users in expressing their information needs accurately. Many users,
especially on the internet, submit very short queries that do not clearly express their
actual needs (Spink et al., 2002). A retrieved set of documents might be relevant to a
submitted query but not particularly relevant to a user’s need. In addition, users who
are non-experts in the searching domain might use out-of-context keyword terms,
resulting in unsatisfactory search results (Ho
¨lscher and Strube, 2000; Hsieh-Yee, 1993).
A vast amount of research has been undertaken in developing methods to assist users
in expressing their information needs. One such method is query expansion (QE),
which consists of adding relevant query terms to an existing query in order to improve
retrieval results (Buckley et al., 1994; Robertson and Walker, 2000; Billerbeck and
The Emerald Research Register for this journal is available at The current issue and full text archive of this journal is available at
www.emeraldinsight.com/researchregister www.emeraldinsight.com/1468-4527.htm
A two-page, short version of this paper was presented at SIGIR 2004 (Nemeth et al., 2004)
OIR
29,4
374
Refereed article received
17 February 2005
Accepted 21 April 2005
Online Information Review
Vol. 29 No. 4, 2005
pp. 374-390
qEmerald Group Publishing Limited
1468-4527
DOI 10.1108/14684520510617820
Zobel, 2004). Query expansion can be carried out automatically or interactively. In
automatic query expansion (AQE), the retrieval system automatically adds query
terms to a user’s original query without their intervention. Interactive query expansion
(IQE) involves feedback from the user on the documents retrieved or on the expansion
terms.
Query limitation is an alternative query-improvement method (quite opposite to
query expansion), where users are provided with options to limit their search in
order to receive more focused results. These options are usually offered by search
engines as “advanced search options”. In this paper we present a study that
examined the effectiveness of the above two methods, namely query expansion
(automatic and interactive) and search limit options. Most previous studies, in
evaluating these methods, carried out lab simulations to compare retrieval results
with and without query expansion, and investigated different query-expansion
methods (Billerbeck and Zobel, 2004; Robertson and Walker, 2000; Sakai and
Robertson, 2001). Some of the research projects did carry out user studies, but
involved only a few actual users (Magennis and van Rijsbergen, 1997; Ruthven,
2003; Belkin et al., 2000). In this study, a thorough evaluation considering various
points of view was undertaken, and included lab simulations and actual users,
examining objective system and user performance, as well as subjective user
opinion of these methods. In applying both objective and subjective aspects, we
were also able to compare the real (objective) and perceived (subjective) value of
these methods.
In the following sections we review related work, present our evaluation framework,
describe the user studies we performed, and detail the lab simulations carried out. In
the final section we discuss our findings and conclude with further research issues.
Related work
There are two main known query-expansion methods: automatic and interactive
(semi-automatic). In automatic query expansion, the expansion terms are added to the
query by the retrieval system without user intervention, while in interactive query
expansion the user is involved in the process of adding the expansion terms.
Automatic global techniques examine term occurrences and relationships in the
entire collection, and use this information to expand any particular query. Some
known automatic global techniques include term clustering (Sparck-Jones and
Jackson, 1970), dimensionality reduction (Caid et al., 1995) and phrasefinder (Jing and
Croft, 1994). In global analysis, the system obtains the global context of each concept
in order to define similarities between the concepts. The simplest definition of a
concept is a word. The context of a word might be simply defined as the list of words
that co-occur with it in other documents in the entire collection. A system employing
global analysis would compute similarities between words based on their
co-occurrences in the collection. The expansion algorithm expands the query using
the concepts most similar (also called pseudo-documents) to those appearing in the
original query.
Automatic local expansion involves only the top ranked documents retrieved for
the original query to calculate expansion terms (Rocchio, 1971; Attar and Fraenkel,
1977; Buckley et al., 1994; Robertson and Walker, 2000). The local context analysis
suggested by Xu and Croft (1996) integrates global and local automatic expansion
Subjective and
objective
evaluation
375
To continue reading
Request your trial