Exploring fact‐focused relevance and novelty detection
| Date | 25 July 2008 |
| DOI | https://doi.org/10.1108/00220410810884057 |
| Pages | 496-510 |
| Published date | 25 July 2008 |
| Author | Jahna Otterbacher,Dragomir Radev |
| Subject Matter | Information & knowledge management,Library & information science |
Exploring fact-focused relevance
and novelty detection
Jahna Otterbacher
Department of Public and Business Administration, University of Cyprus,
Nicosia, Cyprus, and
Dragomir Radev
School of Information, Departments of EECS and Linguistics,
University of Michigan, Ann Arbor, Michigan, USA
Abstract
Purpose – Automated sentence-level relevance and novelty detection would be of direct benefit to
many information retrieval systems. However, the low level of agreement between human judges
performing the task is an issue of concern. In previous approaches, annotators were asked to identify
sentences in a document set that are relevant to a given topic, and then to eliminate sentences that do
not provide novel information. This paper aims to explore a new approach in which relevance and
novelty judgments are made within the context of specific, factual information needs, rather than with
respect to a broad topic.
Design/methodology/approach – An experiment is conducted in which annotators perform the
novelty detection task in both the topic-focused and fact-focused settings.
Findings – Higher levels of agreement between judges are found on the task of identifying relevant
sentences in the fact-focused approach. However, the new approach does not improve agreement on
novelty judgments.
Originality/value – The analysis confirms the intuition that making sentence-level relevance
judgments is likely to be the more difficult of the two tasks in the novelty detection framework.
Keywords Information retrieval, Text retrieval,Information searches, Semantics
Paper type Research paper
Introduction
A core challenge for future information retrieval (IR) systems is to find information
that is not only relevant to a user’s need, but is also novel (Allan et al., 2005). To this
end, “novelty detection,” the task of identifying units in a text or set of texts that
express interesting and previously unseen information, has been introduced. In
contrast to systems that retrieve all relevant items given a user’s topic of interest,
systems incorporating novelty detection aim to reduce the amount of redundant
information presented to the user.
The current issue and full text archive of this journal is available at
www.emeraldinsight.com/0022-0418.htm
This work was partially supported by the US National Science Foundation under the following
grant: 0329043 “Probabilistic and link-based methods for exploiting very large textual
repositories” administered through the IDM program. All opinions, findings, conclusions, and
recommendations in this paper are made by the authors and do not necessarily reflect the views
of the National Science Foundation. The authors would like to thank the members of the CLAIR
research group at the University of Michigan and the anonymous Journal of Documentation
reviewers for their feedback and comments on this work.
JDOC
64,4
496
Received 15 February 2007
Revised 1 August 2007
Accepted 4 August 2007
Journal of Documentation
Vol. 64 No. 4, 2008
pp. 496-510
qEmerald Group Publishing Limited
0022-0418
DOI 10.1108/00220410810884057
To continue reading
Request your trial