scispace - formally typeset
Search or ask a question
Proceedings ArticleDOI

Fusion in Information Retrieval: SIGIR 2018 Half-Day Tutorial

TL;DR: The goal of this half day, intermediate-level, tutorial is to provide a methodological view of the theoretical foundations of fusion approaches, the numerous fusion methods that have been devised and a variety of applications for which fusion techniques have been applied.
Abstract: Fusion is an important and central concept in Information Retrieval. The goal of fusion methods is to merge different sources of information so as to address a retrieval task. For example, in the adhoc retrieval setting, fusion methods have been applied to merge multiple document lists retrieved for a query. The lists could be retrieved using different query representations, document representations, ranking functions and corpora. The goal of this half day, intermediate-level, tutorial is to provide a methodological view of the theoretical foundations of fusion approaches, the numerous fusion methods that have been devised and a variety of applications for which fusion techniques have been applied.
Citations
More filters
Proceedings ArticleDOI
01 Jul 2020
TL;DR: A fully unsupervised method is presented that exploits the FAQ pairs to train two BERT models that match user queries to FAQ answers and questions, respectively and shows that the model is on par and even outperforms supervised models on existing datasets.
Abstract: We focus on the task of Frequently Asked Questions (FAQ) retrieval. A given user query can be matched against the questions and/or the answers in the FAQ. We present a fully unsupervised method that exploits the FAQ pairs to train two BERT models. The two models match user queries to FAQ answers and questions, respectively. We alleviate the missing labeled data of the latter by automatically generating high-quality question paraphrases. We show that our model is on par and even outperforms supervised models on existing datasets.

35 citations


Cites methods from "Fusion in Information Retrieval: SI..."

  • ...The first one, CombSUM (Kurland and Culpepper, 2018), calculates a combined score by summing for each candidate pair the scores that were assigned to it by the three re-ranking methods....

    [...]

  • ...As mentioned above, BERT-Q-q has a significantly better performance on FAQIR than the other two individual rankers, thus a simple fusion method such as CombSUM can not handle such cases well....

    [...]

  • ...PoolRank first ranks the candidate pairs using CombSUM....

    [...]

  • ...Table 1 reports the results for the two datasets.13 We compare the base BM25 retrieval (bm25(q+a)), our three proposed unsupervised re-ranking methods (bm25-maxpsg, BERT-Q-a and BERT-Q-q) and their fusion-based combinations (CombSUM and PoolRank) with the state-of-the-art unsupervised and supervised baselines....

    [...]

  • ...The first one, CombSUM (Kurland and Culpepper, 2018), calculates a combined score by summing for each candidate pair the scores that were assigned to it by the three re-ranking methods.9 Following (Roitman, 2018), as a second alternative, we implement the PoolRank method....

    [...]

Journal ArticleDOI
TL;DR: The authors propose Demonstrate-Search-Predict (DSP), a framework that relies on passing natural language texts in sophisticated pipelines between an LMs and an RMs, which can express high-level programs that bootstrap pipeline-aware demonstrations and generate grounded predictions.
Abstract: Retrieval-augmented in-context learning has emerged as a powerful approach for addressing knowledge-intensive tasks using frozen language models (LM) and retrieval models (RM). Existing work has combined these in simple"retrieve-then-read"pipelines in which the RM retrieves passages that are inserted into the LM prompt. To begin to fully realize the potential of frozen LMs and RMs, we propose Demonstrate-Search-Predict (DSP), a framework that relies on passing natural language texts in sophisticated pipelines between an LM and an RM. DSP can express high-level programs that bootstrap pipeline-aware demonstrations, search for relevant passages, and generate grounded predictions, systematically breaking down problems into small transformations that the LM and RM can handle more reliably. We have written novel DSP programs for answering questions in open-domain, multi-hop, and conversational settings, establishing in early evaluations new state-of-the-art in-context learning results and delivering 37-120%, 8-39%, and 80-290% relative gains against the vanilla LM (GPT-3.5), a standard retrieve-then-read pipeline, and a contemporaneous self-ask pipeline, respectively. We release DSP at https://github.com/stanfordnlp/dsp

33 citations

Proceedings ArticleDOI
20 Apr 2020
TL;DR: This work makes a novel use of intrinsic and extrinsic table similarities for enhanced retrieval via a simple, yet an effective, cascade re-ranking approach that results in a significantly better table retrieval quality, which even transcends that of strong semantically-rich baselines.
Abstract: Given a keyword query, the ad hoc table retrieval task aims at retrieving a ranked list of the top-k most relevant tables in a given table corpus. Previous works have primarily focused on designing table-centric lexical and semantic features, which could be utilized for learning-to-rank (LTR) tables. In this work, we make a novel use of intrinsic (passage-based) and extrinsic (manifold-based) table similarities for enhanced retrieval. Using the WikiTables benchmark, we study the merits of utilizing such similarities for this task. To this end, we combine both similarity types via a simple, yet an effective, cascade re-ranking approach. Overall, our proposed approach results in a significantly better table retrieval quality, which even transcends that of strong semantically-rich baselines.

14 citations


Cites methods from "Fusion in Information Retrieval: SI..."

  • ...To this end, we combine the two similarities by first applying the manifold-based ranking approach on the pool of candidate tables; and only then we combine its score with the intrinsic (passage-based) score using a simple CombMult approach [11]....

    [...]

Proceedings ArticleDOI
17 Oct 2022
TL;DR: This paper presents ranx.fuse, a Python library for Metasearch, built following a user-centered design, that provides easy-to-use tools for combining the results of multiple search engines and offers a convenient functionality for their optimization that evaluates pre-defined hyper-parameters configurations via grid search.
Abstract: This paper presents ranx.fuse, a Python library for Metasearch. Built following a user-centered design, it provides easy-to-use tools for combining the results of multiple search engines. ranx.fuse comprises 25 Metasearch algorithms implemented with Numba, a just-in-time compiler for Python code, for efficient vector operations and automatic parallelization. Moreover, in conjunction with the Metasearch algorithms, our library implements six normalization strategies that transform the search engines' result lists to make them comparable, a mandatory step for Metasearch. Finally, as many Metasearch algorithms require a training or optimization step, ranx.fuse offers a convenient functionality for their optimization that evaluates pre-defined hyper-parameters configurations via grid search. By relying on the provided functions, the user can optimally combine the results of multiple search engines in very few lines of code. ranx.fuse can also serve as a user-friendly tool for fusing the rankings computed by a first-stage retriever and a re-ranker, as a library providing several baselines for Metasearch, and as a playground to test novel normalization strategies.

7 citations

Journal Article
TL;DR: In this paper, the authors evaluate four linear score normalization methods, namely the fitting method, zero-one, Sum, and ZMUV, through extensive experiments and show that fitting method and Zero-one appear to be the two leading methods.
Abstract: In data fusion, score normalization is a step to make scores, which are obtained from different component systems for all documents, comparable to each other It is an indispensable step for effective data fusion algorithms such as CombSum and CombMNZ to combine them In this paper, we evaluate four linear score normalization methods, namely the fitting method, Zero-one, Sum, and ZMUV, through extensive experiments The experimental results show that the fitting method and Zero-one appear to be the two leading methods

6 citations

References
More filters
Book ChapterDOI
21 Jun 2000
TL;DR: Some previous studies comparing ensemble methods are reviewed, and some new experiments are presented to uncover the reasons that Adaboost does not overfit rapidly.
Abstract: Ensemble methods are learning algorithms that construct a set of classifiers and then classify new data points by taking a (weighted) vote of their predictions. The original ensemble method is Bayesian averaging, but more recent algorithms include error-correcting output coding, Bagging, and boosting. This paper reviews these methods and explains why ensembles can often perform better than any single classifier. Some previous studies comparing ensemble methods are reviewed, and some new experiments are presented to uncover the reasons that Adaboost does not overfit rapidly.

5,679 citations


"Fusion in Information Retrieval: SI..." refers background in this paper

  • ...fusion and ensembles of classi€ers in supervised learning [36, 82, 116]....

    [...]

  • ..., [6, 9, 29, 40, 107]) – Ensembles [36, 82, 116] • Applications – ‹ery Performance Prediction [7, 35, 75, 81, 85, 92, 95, 111]...

    [...]

  • ...3 FORMAT AND PLANNED SCHEDULE – Retrieval Score Normalization and Rank-to-Score Transformations [4, 5, 29, 39, 57, 73, 74, 78, 104, 108] – Content-based [15, 30, 49–51, 62, 64, 87, 91] – Selecting Retrieved Lists for Fusion [43, 45, 46] – Query Variations [11, 16–18, 22, 28, 52, 113] – Failure Analysis / Risk [18, 37] – Efficiency Considerations [44, 59] • Learning & Fusion [55, 88] – Models over Permutations (e.g., [1, 38, 48, 54, 83]) – Supervised (e.g., [3, 55, 65–67, 85, 88, 89, 102, 105, 106, 110, 112]) vs Unsupervised (e.g., [6, 9, 29, 40, 107]) – Ensembles [36, 82, 116] • Applications – Query Performance Prediction [7, 35, 75, 81, 85, 92, 95, 111] – Diversification [60, 63, 109] – Relevance Feedback [8, 84] – Selecting a Ranker [2, 12, 33, 58] – Blog and Microblog Retrieval [60, 61, 64, 101] – Pooling and Evaluation [8, 21, 25, 68–71, 86, 97, 98] • Conclusions & Future Directions...

    [...]

Proceedings ArticleDOI
01 Apr 2001
TL;DR: A set of techniques for the rank aggregation problem is developed and compared to that of well-known methods, to design rank aggregation techniques that can be used to combat spam in Web searches.
Abstract: We consider the problem of combining ranking results from various sources. In the context of the Web, the main applications include building meta-search engines, combining ranking functions, selecting documents based on multiple criteria, and improving search precision through word associations. We develop a set of techniques for the rank aggregation problem and compare their performance to that of well-known methods. A primary goal of our work is to design rank aggregation techniques that can e ectively combat \spam," a serious problem in Web searches. Experiments show that our methods are simple, e cient, and e ective.

1,982 citations


"Fusion in Information Retrieval: SI..." refers background in this paper

  • ..., [11, 29, 38, 41, 77, 79, 80, 103]) – Retrieval Score Normalization and Rank-to-Score Transformations [4, 5, 29, 39, 57, 73, 74, 78, 104, 108] – Content-based [15, 30, 49–51, 62, 64, 87, 91] – Selecting Retrieved Lists for Fusion [43, 45, 46] – Query Variations [11, 16–18, 22, 28, 52, 113] – Failure Analysis / Risk [18, 37] – Efficiency Considerations [44, 59] • Learning & Fusion [55, 88] – Models over Permutations (e....

    [...]

23 Jun 2010
TL;DR: RankNet, LambdaRank, and LambdaMART have proven to be very successful algorithms for solving real world ranking problems and the details are spread across several papers and reports, so here is a self-contained, detailed and complete description of them.
Abstract: LambdaMART is the boosted tree version of LambdaRank, which is based on RankNet. RankNet, LambdaRank, and LambdaMART have proven to be very successful algorithms for solving real world ranking problems: for example an ensemble of LambdaMART rankers won Track 1 of the 2010 Yahoo! Learning To Rank Challenge. The details of these algorithms are spread across several papers and reports, and so here we give a self-contained, detailed and complete description of them.

1,114 citations


"Fusion in Information Retrieval: SI..." refers methods in this paper

  • ...[42]), a LambdaMART learning-to-rank (LTR) model [23, 26] (here lightGBM is used with 459 features), and double unsupervised...

    [...]

Proceedings Article
01 Jan 1994
TL;DR: This paper describes one method that has been shown to increase performance by combining the similarity values from five different retrieval runs using both vector space and P-norm extended boolean retrieval methods.
Abstract: The TREC-2 project at Virginai Tech focused on methods for combining the evidence from multiple retrieval runs to improve performance over any single retrieval method. This paper describes one such method that has been shown to increase performance by combining the similarity values from five different retrieval runs using both vector space and P-norm extended boolean retrieval methods

1,106 citations


"Fusion in Information Retrieval: SI..." refers background in this paper

  • ...Risk-reward Trade-offs in Rank Fusion....

    [...]

  • ..., [6, 9, 29, 40, 107]) – Ensembles [36, 82, 116] • Applications – Query Performance Prediction [7, 35, 75, 81, 85, 92, 95, 111] – Diversification [60, 63, 109] – Relevance Feedback [8, 84] – Selecting a Ranker [2, 12, 33, 58] – Blog and Microblog Retrieval [60, 61, 64, 101] – Pooling and Evaluation [8, 21, 25, 68–71, 86, 97, 98] • Conclusions & Future Directions...

    [...]

  • ...– Social Choice Theory and Voting Schemes [20] ∗ Condorcet, Borda, Kemeny [13, 34, 115] – TREC and Rank Fusion [40] – Federated Search [24, 90] • Theoretical Foundations...

    [...]

Proceedings ArticleDOI
15 Aug 2005
TL;DR: A novel approach is developed to train the model that directly maximizes the mean average precision rather than maximizing the likelihood of the training data, and significant improvements are possible by modeling dependencies, especially on the larger web collections.
Abstract: This paper develops a general, formal framework for modeling term dependencies via Markov random fields. The model allows for arbitrary text features to be incorporated as evidence. In particular, we make use of features based on occurrences of single terms, ordered phrases, and unordered phrases. We explore full independence, sequential dependence, and full dependence variants of the model. A novel approach is developed to train the model that directly maximizes the mean average precision rather than maximizing the likelihood of the training data. Ad hoc retrieval experiments are presented on several newswire and web collections, including the GOV2 collection used at the TREC 2004 Terabyte Track. The results show significant improvements are possible by modeling dependencies, especially on the larger web collections.

996 citations


"Fusion in Information Retrieval: SI..." refers methods in this paper

  • ...tems being compared are BM25, a €eld-based SDM model [76] (the exact con€guration is identical to the one described by Gallagher...

    [...]