A WEIGHTED RANK CORRELATION COEFFICIENT FOR THE COMPARISON OF RELEVANCE JUDGEMENTS

Pages380-389
DOIhttps://doi.org/10.1108/eb026564
Published date01 April 1973
Date01 April 1973
AuthorD. KAYE
Subject MatterInformation & knowledge management,Library & information science
A WEIGHTED RANK CORRELATION COEFFICIENT
FOR THE COMPARISON OF RELEVANCE
JUDGEMENTS
D.
KAYE
Department of Librarianship,
Manchester
Polytechnic
The application of conventional rank correlation tests to the comparison of
relevance judgements is outlined and their disadvantages are noted. A
weighted rank correlation coefficient rw
is
proposed, which assigns variable
weights to exchanges of rank on different parts of the
scale.
Examples of its
computation are given, both for tied and untied
ranks,
and the values of rw
for n=3 and n=4 are tabulated. The chief mathematical properties of the
coefficient are investigated.
MOST RELEVANCE MEASURES at present in use are based on the
assumption that the judgement of relevance
is
dichotomous, that is, that a
document is either relevant to a question or it is not. In practice, there are
many situations in which degrees of relevance are apparent, and in conven-
tional testing it may be that
less
relevant documents are unnaturally forced
into
yes
or
no
categories, with consequent
loss
of sensitivity in the measure.
Moreover two measures (recall and precision) are usually needed to charac-
terize a system and various attempts have been made to combine these into
a single measure.
RANK CORRELATION TESTING
Pollock1 and other writers have suggested the use of rank correlation
testing for systems where the output from a search is in ranked order, for
example as a result of a weighted term search, or in cases where the raw
output is screened and ranked by the system operator. Measures which
have been proposed for this purpose include Kendall's Τ statistic and
Spearman's p (here called rs). The latter is outlined below, as it forms the
basis of the weighted coefficient proposed in this paper. Kendall's statistic,
however, will not be described
here;
those interested may consult Kendall,3
pp.
3-8.
If there are n pairs of corresponding ranks (X1, Y1), (X2, Y2) . . . (Xi, Yi)
. . . (Xn, Yn) where the Xi take the values 1, 2, 3.. . i,... n in succession
and the Yi take the same set of values generally in some other order, then
Spearman's coefficient of rank correlation is defined as:
380

To continue reading

Request your trial

VLEX uses login cookies to provide you with a better browsing experience. If you click on 'Accept' or continue browsing this site we consider that you accept our cookie policy. ACCEPT