AN EXTENSION OF THE SIGNAL DETECTION MODEL OF INFORMATION RETRIEVAL
| Published date | 01 January 1978 |
| Pages | 51-54 |
| DOI | https://doi.org/10.1108/eb026652 |
| Date | 01 January 1978 |
| Author | T.P. HUTCHINSON |
| Subject Matter | Information & knowledge management,Library & information science |
DOCUMENTATION NOTE
AN EXTENSION OF THE SIGNAL DETECTION MODEL OF
INFORMATION RETRIEVAL
T. P. HUTCHINSON
Transport Studies Group, University College London
It is suggested that a possible reason for receiver-operating characteristics
(ROC) curves having a slope other than unity when plotted on Normal-
deviate axes is that the dichotomy into relevant and non-relevant items is
artificial, and that there
is
really
a
bivariate distribution of objective relevance
and relevance as assessed by the IR system, with positive correlation. This
correlation
is
the appropriate measure of the effectiveness of the IR system,
and can be calculated from the shape of the ROC curve.
THE SIGNAL DETECTION theory (SDT) of psychophysics1, 2 has recently
been applied to measuring the performance of information retrieval (IR) sys-
tems.3,4
The advantage lies in giving a measure of the ability of the system to
discriminate between relevant and non-relevant items that is independent of the
overall propensity of the system to retrieve items irrespective of their relevance.
The problem is: we ask an IR system for references on a certain topic. As well as
giving us some of these, it will usually retrieve some references that are not
relevant, and it will fail to retrieve some that are. We may consider that the IR
system has a decision axis—notionally, we can envisage it calculating a 'rele-
vance score' for each item—and that those items passing a certain threshold are re-
trieved, and those falling below are not. The user of the system decides which items
are relevant and which are not. These two populations will each have a certain
distribution of relevance scores, that for the relevant items having a higher mean.
If the threshold for retrieval can be altered, the proportion of relevant items
retrieved can be plotted against the proportion of non-relevant items retrieved.
This is known as a receiver-operating characteristic (ROC) curve. If the two
distributions of relevance score are Normal in shape, with the same variance,
then the ROC curve becomes a straight line of unit slope when the axes are
transformed to the Normal deviates of the proportions. The obvious generaliza-
tion of this is to relax the assumption of equal variances, and this gives rise to a
slope other than unity. The objection to this is that then the same distribution
(relevant or non-relevant items) dominates both ends of the relevance score axis.
Thus we should find ourselves either selecting items with a very low relevance
score in addition to those with a high one (if the distribution for relevant items
had the larger variance) or excluding items with a very high relevance score (if
the distribution for non-relevant items Had the larger variance). For further dis-
cussion of this in the psychophysical context, see reference 5, pp.
102-3,
336-9.
Next, we remark that instead of plotting Prob {retrieval│relevant} against
Prob {retrieval | non-relevant} as relevance score varies, we could consider
(objective) relevance to be itself an ordered variable with different distributions
in two cases which might differ in specification of the query, the formula used
for comparison with a threshold, or the system
itself.
This would suggest a plot
of proportion of cases exceeding a given degree of objective relevance under one
Journal
of
Documentation,
Vol. 34, No. 1, March 1978, pp. 51-54.
51
To continue reading
Request your trial