This tutorial is concerned with a comprehensive introduction to the research area of learning to rank for information retrieval. In the first part of the tutorial, we will introduce three major approaches to learning to rank, i.e., the pointwise, pairwise, and listwise approaches, analyze the relationship between the loss functions used in these approaches and the widely-used IR evaluation measures, evaluate the performance of these approaches on the LETOR benchmark datasets, and demonstrate how to use these approaches to solve real ranking applications. In the second part of the tutorial, we will discuss some advanced topics regarding learning to rank, such as relational ranking, diverse ranking, semi-supervised ranking, transfer ranking, query-dependent ranking, and training data preprocessing. In the third part, we will briefly mention the recent advances on statistical learning theory for ranking, which explain the generalization ability and statistical consistency of different ranking methods. In the last part, we will conclude the tutorial and show several future research directions.

/pdf/learning-to-rank-for-information-retrieval-2q6t3viyug.pdf

Learning to Rank for Information Retrieval

The Probabilistic Relevance Framework (PRF) is a formal framework for document retrieval, grounded in work done in the 1970—1980s, which led to the development of one of the most successful text-retrieval algorithms, BM25. In recent years, research in the PRF has yielded new retrieval models capable of taking into account document meta-data (especially structure and link-graph information). Again, this has led to one of the most successful Web-search and corporate-search algorithms, BM25F. This work presents the PRF from a conceptual point of view, describing the probabilistic modelling assumptions behind the framework and the different ranking algorithms that result from its application: the binary independence model, relevance feedback models, BM25 and BM25F. It also discusses the relation between the PRF and other statistical models for IR, and covers some related topics, such as the use of non-textual features, and parameter optimisation for models with free parameters.

/pdf/the-probabilistic-relevance-framework-290mics7xy.pdf

The Probabilistic Relevance Framework

We present the Word Mover's Distance (WMD), a novel distance function between text documents. Our work is based on recent results in word embeddings that learn semantically meaningful representations for words from local cooccurrences in sentences. The WMD distance measures the dissimilarity between two text documents as the minimum amount of distance that the embedded words of one document need to "travel" to reach the embedded words of another document. We show that this distance metric can be cast as an instance of the Earth Mover's Distance, a well studied transportation problem for which several highly efficient solvers have been developed. Our metric has no hyperparameters and is straight-forward to implement. Further, we demonstrate on eight real world document classification data sets, in comparison with seven state-of-the-art baselines, that the WMD metric leads to unprecedented low k-nearest neighbor document classification error rates.

/pdf/from-word-embeddings-to-document-distances-2qevvhrrb0.pdf

From Word Embeddings To Document Distances

We explore the relation between classical probabilistic models of information retrieval and the emerging language modeling approaches. It has long been recognized that the primary obstacle to effective performance of classical models is the need to estimate a relevance model: probabilities of words in the relevant class. We propose a novel technique for estimating these probabilities using the query alone. We demonstrate that our technique can produce highly accurate relevance models, addressing important notions of synonymy and polysemy. Our experiments show relevance models outperforming baseline language modeling systems on TREC retrieval and TDT tracking tasks. The main contribution of this work is an effective formal method for estimating a relevance model with no training data.

/pdf/relevance-based-language-models-493mnssg4n.pdf

Relevance-Based Language Models

Language modeling approaches to information retrieval are attractive and promising because they connect the problem of retrieval with that of language model estimation, which has been studied extensively in other application areas such as speech recognition. The basic idea of these approaches is to estimate a language model for each document, and then rank documents by the likelihood of the query according to the estimated language model. A core problem in language model estimation is smoothing, which adjusts the maximum likelihood estimator so as to correct the inaccuracy due to data sparseness. In this paper, we study the problem of language model smoothing and its influence on retrieval performance. We examine the sensitivity of retrieval performance to the smoothing parameters and compare several popular smoothing methods on different test collections.

/pdf/a-study-of-smoothing-methods-for-language-models-applied-to-1d4kkxcs09.pdf

A Study of Smoothing Methods for Language Models Applied to Ad Hoc Information Retrieval

City submitted two runs each for the automatic ad hoc, very large collection track, automatic routing and Chinese track; and took part in the interactive and filtering tracks. The method used was : expansion using terms from the top documents retrieved by a pilot search on topic terms. Additional runs seem to show that we would have done better without expansion. Twor runs using the method of city96al were also submitted for the Very Large Collection track. The training database and its relevant documents were partitioned into three parts. Working on a pool of terms extracted from the relevant documents for one partition, an iterative procedure added or removed terms and/or varied their weights. After each change in query content or term weights a score was calculated by using the current query to search a second protion of the training database and evaluating the results against the corresponding set of relevant documents. Methods were compared by evaluating queries predictively against the third training partition. Queries from different methods were then merged and the results evaluated in the same way. Two runs were submitted, one based on character searching and the other on words or phrases. Much of the work involved investigating plausible methods of applying Okapi-style weighting to phrases

Okapi at TREC

The paper presents two novel approaches to query expansion with long-span collocates—words, significantly co-occurring in topic-size windows with query terms. In the first approach—global collocation analysis—collocates of query terms are extracted from the entire collection, in the second—local collocation analysis—from a subset of retrieved documents. The significance of association between collocates was estimated using modified Mutual Information and Z score. The techniques were tested using the Okapi IR system. The effect of different parameters on performance was evaluated: window size, number of expansion terms, measures of collocation significance and types of expansion terms. We present performance results of these techniques and provide comparison with related approaches.

/pdf/query-expansion-with-long-span-collocates-33npmlbjxb.pdf

Query Expansion with Long-Span Collocates

We discuss whether it is feasible to build intelligence rule- or weight-based algorithms into general-purpose software for interactive thesaurus navigation. We survey some approaches to the problem reported in the literature, particularly those involving the assignment of “link weights” in a thesaurus network, and point out some problems of both principle and practice. We then describe investigations which entailed logging the behavior of thesaurus users and testing the effect of thesaurus-based query enhancement in an IR system using term weighting, in an attempt to identify successful strategies to incorporate into automatic procedures. The results cause us to question many of the assumptions made by previous researchers in this area. © 1995 John Wiley & Sons, Inc.

Interactive thesaurus navigation: intelligence rules ok?

This paper demonstrates the application of conven tional database design techniques to thesaurus representation The thesaurus is considered as a printed document, as a semantic net, and as a relational database to be used in conjunction with an intelligent information retrieval system Some issues raised by analysis of two standard thesaun in clude the prevalence of compound terms and the representa tion of term structure: thesaurus redundancy and the extent to which it can be eliminated in machine-readable versions: the difficulty of exploiting thesaurus knowledge originally designed for human rather than automatic interpretation: deriving "strength of association" measures between terms in a thesaurus considered as a semantic net, facet representation and the need for variations in the data model to cater for structural differences between thesauri A complete schema of database tables is presented, with an outline suggestion for using the stored information when matching one or more thesaurus terms wit

A thesaurus data model for an intelligent retrieval system

There are several recent studies that propose search output clustering as an alternative representation method to ranked output. Users are provided with cluster representations instead of lists of titles and invited to make decisions on groups of documents. This paper discusses the difficulties involved in representing clusters for users’ evaluation in a concise but easily interpretable form. The discussion is based on findings and user feedback from a user study investigating the effectiveness of search output clustering. The overall impression created by the experiment results and users’ feedback is that clusters cannot be relied on to consistently produce meaningful document groups that can easily be recognised by the users. They also seem to lead to unrealistic user expectations.

Susan Jones

Papers

Okapi at TREC

Query Expansion with Long-Span Collocates

Interactive thesaurus navigation: intelligence rules ok?

A thesaurus data model for an intelligent retrieval system

Deciphering cluster representations