Modern large retrieval environments tend to overwhelm their users by their large output. Since all documents are not of equal relevance to their users, highly relevant documents should be identified and ranked first for presentation. In order to develop IR techniques in this direction, it is necessary to develop evaluation approaches and methods that credit IR methods for their ability to retrieve highly relevant documents. This can be done by extending traditional evaluation methods, that is, recall and precision based on binary relevance judgments, to graded relevance judgments. Alternatively, novel measures based on graded relevance judgments may be developed. This article proposes several novel measures that compute the cumulative gain the user obtains by examining the retrieval result up to a given ranked position. The first one accumulates the relevance scores of retrieved documents along the ranked result list. The second one is similar but applies a discount factor to the relevance scores in order to devaluate late-retrieved documents. The third one computes the relative-to-the-ideal performance of IR techniques, based on the cumulative gain they are able to yield. These novel measures are defined and discussed and their use is demonstrated in a case study using TREC data: sample system run results for 20 queries in TREC-7. As a relevance base we used novel graded relevance judgments on a four-point scale. The test results indicate that the proposed measures credit IR methods for their ability to retrieve highly relevant documents and allow testing of statistical significance of effectiveness differences. The graphs based on the measures also provide insight into the performance IR techniques and allow interpretation, for example, from the user point of view.

https://www.cc.gatech.edu/~zha/CS8803WST/dcg.pdf

Cumulated gain-based evaluation of IR techniques

The principle that, for optimal retrieval, documents should be ranked in order of the probability of relevance or usefulness has been brought into question by Cooper. It is shown that the principle can be justified under certain assumptions, but that in cases where these assumptions do not hold, the principle is not valid. The major problem appears to lie in the way the principle considers each document independently of the rest. The nature of the information on the basis of which the system decides whether or not to retrieve the documents determines whether the document‐by‐document approach is valid.

The probability ranking principle in IR

KEY BENEFIT: Written by a leader in the field of information retrieval, this text provides the background and tools needed to evaluate, compare and modify search engines. KEY TOPICS: Coverage of the underlying IR and mathematical models reinforce key concepts. Numerous programming exercises make extensive use of Galago, a Java-based open source search engine. MARKET: A valuable tool for search engine and information retrieval professionals.

Search Engines: Information Retrieval in Practice

While numerous metrics for information retrieval are available in the case of binary relevance, there is only one commonly used metric for graded relevance, namely the Discounted Cumulative Gain (DCG). A drawback of DCG is its additive nature and the underlying independence assumption: a document in a given position has always the same gain and discount independently of the documents shown above it. Inspired by the "cascade" user model, we present a new editorial metric for graded relevance which overcomes this difficulty and implicitly discounts documents which are shown below very relevant documents. More precisely, this new metric is defined as the expected reciprocal length of time that the user will take to find a relevant document. This can be seen as an extension of the classical reciprocal rank to the graded relevance case and we call this metric Expected Reciprocal Rank (ERR). We conduct an extensive evaluation on the query logs of a commercial search engine and show that ERR correlates better with clicks metrics than other editorial metrics.

Expected reciprocal rank for graded relevance

There are many applications in which it is desirable to order rather than classify instances. Here we consider the problem of learning how to order instances given feedback in the form of preference judgments, i.e., statements to the effect that one instance should be ranked ahead of another. We outline a two-stage approach in which one first learns by conventional means a binary preference function indicating whether it is advisable to rank one instance before another. Here we consider an on-line algorithm for learning preference functions that is based on Freund and Schapire's "Hedge" algorithm. In the second stage, new instances are ordered so as to maximize agreement with the learned preference function. We show that the problem of finding the ordering that agrees best with a learned preference function is NP-complete. Nevertheless, we describe simple greedy algorithms that are guaranteed to find a good approximation. Finally, we show how metasearch can be formulated as an ordering problem, and present experimental results on learning a combination of "search experts," each of which is a domain-specific query expansion strategy for a web search engine.

/pdf/learning-to-order-things-3znw7cn2qb.pdf

Learning to order things

A measure of document retrieval system performance called the “expected search length reduction factor” is defined and compared with indicators, such as precision and recall, that have been suggested by other workers. The new measure is based on calculations of the expected number of irrelevant documents in the collection which would have to be searched through before the desired number of relevant documents could be found. Its advantages are: (1) it provides a single index for the property it attempts to measure; (2) it allows for gradations of retrieval status, through the mathematical concept of a “weak ordering”; (3) it evaluates retrieval performance relative to random searching; and (4) it takes into account the amount of relevant material desired by the requester.

Expected search length: A single measure of retrieval effectiveness based on the weak ordering action of retrieval systems

It is often assumed that the amount of interindexer consistency experienced under a given method of indexing is somehow indicative of the quality of the indexing. To explore this assumption, two hypotheses are stated concerning the possible connection between interindexer consistency and indexing quality. A specific counter-example is then exhibited which shows both hypotheses to be invalid. Although a mathematical analysis of the counterexample yields certain insights, the general relationship between interindexer consistency and successful retrieval is more subtle than might have been expected. It is concluded that until equations describing this relationship have been derived, measurements of interindexer consistency will have little meaning as clues to indexing quality.

Is interindexer consistency a hobgoblin

Library patrons who wish to obtain a known document with which they have had prior personal contact often cannot remember the standard author-title-subject information about it with sufficient accuracy to be able to look it up easily in present-day card catalogs. However, they may remember a surprising amount of such “nonstandard” information as the color of the document's cover or its approximate length. Could this type of “nonstandard” information be profitably exploited in computerized catalogs of the future? Such facilities, if available, would surely be used, for according to catalog usage surveys a majority of today's library patrons seek known documents, and of these, a substantial minority possess nonstandard information. In this report, the results of a memory experiment to test the memorability of various types of nonstandard information are described and analyzed. A ranking according to relative memorability and potential retrieval usefulness of various nonstandard book features is given. It is estimated that if the average patron's nonstandard information were exploited by appropriate retrieval strategies, he would only have to search through roughly one five-hundredth as many documents as in a random search. This reduction factor is great enough to make nonstandard information potentially useful in many situations.

William S. Cooper

Papers

Expected search length: A single measure of retrieval effectiveness based on the weak ordering action of retrieval systems

Is interindexer consistency a hobgoblin

The potential usefulness of catalog access points other than author, title, and subject