The naive Bayes classifier, currently experiencing a renaissance in machine learning, has long been a core technique in information retrieval. We review some of the variations of naive Bayes models used for text retrieval and classification, focusing on the distributional assumptions made about word occurrences in documents.

/pdf/naive-bayes-at-forty-the-independence-assumption-in-1ht5r4ylap.pdf

Naive (Bayes) at forty: the independence assumption in information retrieval

The Probabilistic Relevance Framework (PRF) is a formal framework for document retrieval, grounded in work done in the 1970—1980s, which led to the development of one of the most successful text-retrieval algorithms, BM25. In recent years, research in the PRF has yielded new retrieval models capable of taking into account document meta-data (especially structure and link-graph information). Again, this has led to one of the most successful Web-search and corporate-search algorithms, BM25F. This work presents the PRF from a conceptual point of view, describing the probabilistic modelling assumptions behind the framework and the different ranking algorithms that result from its application: the binary independence model, relevance feedback models, BM25 and BM25F. It also discusses the relation between the PRF and other statistical models for IR, and covers some related topics, such as the use of non-textual features, and parameter optimisation for models with free parameters.

/pdf/the-probabilistic-relevance-framework-290mics7xy.pdf

The Probabilistic Relevance Framework

The term‐weighting function known as IDF was proposed in 1972, and has since been extremely widely used, usually as part of a TF*IDF function. It is often described as a heuristic, and many papers have been written (some based on Shannon's Information Theory) seeking to establish some theoretical basis for it. Some of these attempts are reviewed, and it is shown that the Information Theory approaches are problematic, but that there are good theoretical justifications of both IDF and TF*IDF in the traditional probabilistic model of information retrieval.

/pdf/understanding-inverse-document-frequency-on-theoretical-r4qm06i7n4.pdf

Understanding inverse document frequency: on theoretical arguments for IDF

The 2-Poisson model for term frequencies is used to suggest ways of incorporating certain variables in probabilistic models for information retrieval. The variables concerned are within-document term frequency, document length, and within-query term frequency. Simple weighting functions are developed, and tested on the TREC test collection. Considerable performance improvements (over simple inverse collection frequency weighting) are demonstrated.

Some simple effective approximations to the 2-Poisson model for probabilistic weighted retrieval

The 2-Poisson model for term frequencies is used to suggest ways of incorporating certain variables in probabilistic models for information retrieval. The variables concerned are within-document term tkequency, document length, and within-query term frequency. Simple weighting functions are developed, and tested on the TREC test collection. Considerable performance improvements (over simple inverse collection frequency weighting) are demonstrated.

/pdf/some-simple-effective-approximations-to-the-2-poisson-model-1byccvajwy.pdf

Some Simple Effective Approximations to the 2-Poisson Model

The problem studied in this research is that of developing a set of formal statistical rules for the purpose of identifying the keywords of a document-words likely to be useful as index terms for that document. The research was prompted by the observation, made by a number of writers, that non-specialty words, words which possess little value for indexing purposes, tend to be distributed at random in a collection of documents. In contrast, specialty words are not so distributed.



In Part I of the study, a mixture of two Poisson distributions is examined in detail as a model of specialty word distribution, and formulas expressing the three parameters of the model in terms of empirical frequency statistics are derived. The fit of the model is tested on an experimental document collection and found to be acceptable for the purposes of the study. A measure intended to identify specialty words, consistent with the 2-Poisson model, is proposed and evaluated.

A probabilistic approach to automatic keyword indexing. Part I. On the Distribution of Specialty Words in a Technical Literature

In Part I of this study,* a mixture of two Poisson distributions was examined as a model of specialty word distribution. Formulas expressing the three parameters of the model in terms of empirical frequency statistics were derived, and a statistical measure intended to identify specialty words, consistent with the model, was proposed.



In the present paper, Part II of the study, a probabilistic model of keyword indexing is outlined, and some of the consequences of the model are examined. An algorithm defining a measure of indexability is developed-a measure intended to reflect the relative significance of words in documents. The measure is evaluated and is found to consistently produce indexes superior to those produced by another measure which had previously been identified in the literature as producing the best results.

A probabilistic approach to automatic keyword indexing. Part II. An algorithm for probabilistic indexing

The results of a survey of accredited library schools indicate that instruction provided in on-line searching is widespread and growing but that there is great diversity among library schools in terms of the goals and objectives of the instruction provided. The University of South Florida's new course, Atomated Information Sources and Services, is described.

An assessment of instruction provided by library schools in on-line searching

Evaluating Information: A Guide for Users of Social Science Research. Jeffry Katzer, Kenneth H. Cook, and Wayne W. Crouch. Reading, MA: Addison-Wesley; 1978: 191 p.

The use of on-line search in teaching: An assessment of projects carried out by UK schools of library and information studies: B.C. Vickery. British Library Research and Development Report No. 5390. The British Library, Research and Development Department, London, 1977. 43pp., $5.00

Stephen P. Harter

Papers

A probabilistic approach to automatic keyword indexing. Part I. On the Distribution of Specialty Words in a Technical Literature

A probabilistic approach to automatic keyword indexing. Part II. An algorithm for probabilistic indexing

An assessment of instruction provided by library schools in on-line searching

Evaluating Information: A Guide for Users of Social Science Research. Jeffry Katzer, Kenneth H. Cook, and Wayne W. Crouch. Reading, MA: Addison-Wesley; 1978: 191 p.

The use of on-line search in teaching: An assessment of projects carried out by UK schools of library and information studies: B.C. Vickery. British Library Research and Development Report No. 5390. The British Library, Research and Development Department, London, 1977. 43pp., $5.00