A method to assess search engine results

DOIhttps://doi.org/10.1108/14684521111193166
Pages854-868
Published date29 November 2011
Date29 November 2011
AuthorJudit Bar‐Ilan,Mark Levene
Subject MatterInformation & knowledge management,Library & information science
A method to assess search engine
results
Judit Bar-Ilan
Department of Information Science, Bar-Ilan University, Ramat Gan, Israel, and
Mark Levene
Department of Computer Science and Information Systems, Birkbeck College,
University of London, London, UK
Abstract
Purpose – The aim of this paper is to develop a methodology for assessing search results retrieved
from different sources.
Design/methodology/approach – This is a two phase method, where in the first stage users select
and rank the ten best search results from a randomly ordered set. In the second stage they are asked to
choose the best pre-ranked result from a set of possibilities. This two-stage method allows users to
consider each search result separately (in the first stage) and to express their views on the rankings as
a whole, as they were retrieved by the search provider. The method was tested in a user study that
compared different country-specific search results of Google and Live Search (now Bing). The users
were Israelis and the search results came from six sources: Google Israel, Google.com, Google UK, Live
Search Israel, Live Search US and Live Search UK. The users evaluated the results of nine pre-selected
queries, created their own preferred ranking and picked the best ranking from the six sources.
Findings – The results indicate that the group of users in this study preferred their local Google
interface, i.e. Google succeeded in its country-specific customisation of search results. Live Search was
much less successful in this aspect.
Research limitations/implications Search engines are highly dynamic, thus the findings of the
case study have to be viewed cautiously.
Originality/value – The main contribution of the paper is a two-phase methodology for comparing
and evaluating search results from different sources.
Keywords Search engines,User assessment, Country-specificresults, User interfaces,
Information retrieval, Informationsearches
Paper type Research paper
Introduction
Searching for information on the web has become a major activity. According to the
Pew Internet and American Life surveys (PEW, 2009) 88 percent of American internet
users have used search engines to find information. Searching for information is the
second most frequently mentioned internet activity after e-mail. By far the most
popular search engine is Google (see for example ComScore, 2010) followed by Yahoo!
and Microsoft sites. In Israel Google is by far the most popular site, its exposure was
91.1 percent as of July 2010 (Walla, 2010).
Thus it seems that search engines are very heavily used and their quality taken for
granted by internet users. The question is: are they really satisfied with the search
The current issue and full text archive of this journal is available at
www.emeraldinsight.com/1468-4527.htm
A preliminary version of this work was presented at the 2009 ASIST Conference, and is available
at: http://www.asis.org/Conferences/AM09/open-proceedings/papers/46.xml
OIR
35,6
854
Received 1 September 2010
Accepted 27 March 2011
Online Information Review
Vol. 35 No. 6, 2011
pp. 854-868
qEmerald Group Publishing Limited
1468-4527
DOI 10.1108/14684521111193166

To continue reading

Request your trial

VLEX uses login cookies to provide you with a better browsing experience. If you click on 'Accept' or continue browsing this site we consider that you accept our cookie policy. ACCEPT