Automatic performance evaluation of web search engines using judgments of metasearch engines
| DOI | https://doi.org/10.1108/14684521111193229 |
| Published date | 29 November 2011 |
| Date | 29 November 2011 |
| Pages | 957-971 |
| Author | Hamid Sadeghi |
| Subject Matter | Information & knowledge management,Library & information science |
Automatic performance
evaluation of web search engines
using judgments of metasearch
engines
Hamid Sadeghi
Department of Computer Engineering, Islamic Azad University, Hashtgerd, Iran
Abstract
Purpose – The purpose of this paper is to introduce two new automatic methods for evaluating the
performance of search engines. The reported study uses the methods to experimentally investigate
which search engine among three popular search engines (Ask.com, Bing and Google) gives the best
performance.
Design/methodology/approach – The study assesses the performance of three search engines. For
each one the weighted average of similarity degrees between its ranked result list and those of its
metasearch engines is measured. Next these measures are compared to establish which search engine
gives the best performance. To compute the similarity degree between the lists two measures called
the “tendency degree” and “coverage degree” are introduced; the former assesses a search engine in
terms of results presentation and the latter evaluates it in terms of retrieval effectiveness. The
performance of the search engines is experimentally assessed based on the 50 topics of the 2002 TREC
web track. The effectiveness of the methods is also compared with human-based ones.
Findings – Google outperformed the others, followed by Bing and Ask.com. Moreover significant
degrees of consistency – 92.87 percent and 91.93 percent – were found between automatic and
human-based approaches.
Practical implications – The findings of this work could help users to select a truly effective search
engine. The results also provide motivation for the vendors of web search engines to improve their
technology.
Originality/value – The paper focuses on two novel automatic methods to evaluate the performance
of search engines and provides valuable experimental results on three popular ones.
Keywords Search engines,Performance evaluation, Metasearchengines, Information retrieval,
Automation, Information searches, Functionevaluation
Paper type Research paper
Introduction
The amount of information on the web is enormous and finding relevant information
among an overwhelming amount of irrelevant information is one of the major concerns
in the electronic environment. The web search engine is an effective and efficient web
information search tool that has been developed to help users solve this problem. As
the volume of information on the internet increases dramatically, the web search
engine has become an indispensable tool to search and locate the required information.
However it is virtually impossible for any single search engine to index the entire web
The current issue and full text archive of this journal is available at
www.emeraldinsight.com/1468-4527.htm
The author would like to thank the Islamic Azad University, Hashtgerd Campus, for the financial
support of this research.
Automatic
evaluation of web
search
957
Received 14 September 2010
Accepted 27 March 2011
Online Information Review
Vol. 35 No. 6, 2011
pp. 957-971
qEmerald Group Publishing Limited
1468-4527
DOI 10.1108/14684521111193229
To continue reading
Request your trial