Journal ArticleDOI
IR evaluation methods for retrieving highly relevant documents
Kalervo Järvelin,Jaana Kekäläinen +1 more
- Vol. 51, Iss: 2, pp 41-48
Reads0
Chats0
TLDR
The novel evaluation methods and the case demonstrate that non-dichotomous relevance assessments are applicable in IR experiments, may reveal interesting phenomena, and allow harder testing of IR methods.Abstract:
This paper proposes evaluation methods based on the use of non-dichotomous relevance judgements in IR experiments It is argued that evaluation methods should credit IR methods for their ability to retrieve highly relevant documents This is desirable from the user point of view in modem large IR environments The proposed methods are (1) a novel application of P-R curves and average precision computations based on separate recall bases for documents of different degrees of relevance, and (2) two novel measures computing the cumulative gain the user obtains by examining the retrieval result up to a given ranked position We then demonstrate the use of these evaluation methods in a case study on the effectiveness of query types, based on combinations of query structures and expansion, in retrieving documents of various degrees of relevance The test was run with a best match retrieval system (In- Query I) in a text database consisting of newspaper articles The results indicate that the tested strong query structures are most effective in retrieving highly relevant documents The differences between the query types are practically essential and statistically significant More generally, the novel evaluation methods and the case demonstrate that non-dichotomous relevance assessments are applicable in IR experiments, may reveal interesting phenomena, and allow harder testing of IR methodsread more
Citations
More filters
Journal ArticleDOI
Cumulated gain-based evaluation of IR techniques
TL;DR: This article proposes several novel measures that compute the cumulative gain the user obtains by examining the retrieval result up to a given ranked position, and test results indicate that the proposed measures credit IR methods for their ability to retrieve highly relevant documents and allow testing of statistical significance of effectiveness differences.
Proceedings ArticleDOI
Learning to rank using gradient descent
Chris J.C. Burges,Tal Shaked,Erin L. Renshaw,Ari Lazier,Matt Deeds,Nicole A. Hamilton,Greg Hullender +6 more
TL;DR: RankNet is introduced, an implementation of these ideas using a neural network to model the underlying ranking function, and test results on toy data and on data from a commercial internet search engine are presented.
Book
Learning to Rank for Information Retrieval
TL;DR: Three major approaches to learning to rank are introduced, i.e., the pointwise, pairwise, and listwise approaches, the relationship between the loss functions used in these approaches and the widely-used IR evaluation measures are analyzed, and the performance of these approaches on the LETOR benchmark datasets is evaluated.
Book
Foundations of Machine Learning
TL;DR: This graduate-level textbook introduces fundamental concepts and methods in machine learning, and provides the theoretical underpinnings of these algorithms, and illustrates key aspects for their application.
Proceedings ArticleDOI
Learning to rank: from pairwise approach to listwise approach
TL;DR: It is proposed that learning to rank should adopt the listwise approach in which lists of objects are used as 'instances' in learning, and introduces two probability models, respectively referred to as permutation probability and top k probability, to define a listwise loss function for learning.
References
More filters
Book
Practical Nonparametric Statistics
TL;DR: Probability Theory. Statistical Inference. Contingency Tables. Appendix Tables. Answers to Odd-Numbered Exercises and Answers to Answers to Answer Questions as discussed by the authors.
Journal ArticleDOI
Practical Nonparametric Statistics (2nd ed).
Journal ArticleDOI
An evaluation of retrieval effectiveness for a full-text document-retrieval system
David C. Blair,M. E. Maron +1 more
TL;DR: An evaluation of a large, operational full-text document-retrieval system shows the system to be retrieving less than 20 percent of the documents relevant to a particular search.
Journal ArticleDOI
Inference Networks for Document Retrieval
Howard R. Turtle,W. Bruce Croft +1 more
TL;DR: The use of inference networks to support document retrieval and a network-basead retrieval model is described and compared to conventional probabilistic and Boolean models.
Journal ArticleDOI
Practical Nonparametric Statistics (2nd ed.)
TL;DR: In this paper, the authors present the Practical Nonparametric Statistics (2nd ed.) for nonparametric statistics and show that it is NP-hard to compute the probability of a node in a graph.