Modern large retrieval environments tend to overwhelm their users by their large output. Since all documents are not of equal relevance to their users, highly relevant documents should be identified and ranked first for presentation. In order to develop IR techniques in this direction, it is necessary to develop evaluation approaches and methods that credit IR methods for their ability to retrieve highly relevant documents. This can be done by extending traditional evaluation methods, that is, recall and precision based on binary relevance judgments, to graded relevance judgments. Alternatively, novel measures based on graded relevance judgments may be developed. This article proposes several novel measures that compute the cumulative gain the user obtains by examining the retrieval result up to a given ranked position. The first one accumulates the relevance scores of retrieved documents along the ranked result list. The second one is similar but applies a discount factor to the relevance scores in order to devaluate late-retrieved documents. The third one computes the relative-to-the-ideal performance of IR techniques, based on the cumulative gain they are able to yield. These novel measures are defined and discussed and their use is demonstrated in a case study using TREC data: sample system run results for 20 queries in TREC-7. As a relevance base we used novel graded relevance judgments on a four-point scale. The test results indicate that the proposed measures credit IR methods for their ability to retrieve highly relevant documents and allow testing of statistical significance of effectiveness differences. The graphs based on the measures also provide insight into the performance IR techniques and allow interpretation, for example, from the user point of view.

https://www.cc.gatech.edu/~zha/CS8803WST/dcg.pdf

Cumulated gain-based evaluation of IR techniques

The naive Bayes classifier, currently experiencing a renaissance in machine learning, has long been a core technique in information retrieval. We review some of the variations of naive Bayes models used for text retrieval and classification, focusing on the distributional assumptions made about word occurrences in documents.

/pdf/naive-bayes-at-forty-the-independence-assumption-in-1ht5r4ylap.pdf

Naive (Bayes) at forty: the independence assumption in information retrieval

This paper presents an outline of models of information seeking and other aspects of information behaviour, showing the relationship between communication and information behaviour in general with information seeking and information searching in information retrieval systems. It is suggested that these models address issues at various levels of information behaviour and that they can be related by envisaging a ‘nesting’ of models. It is also suggested that, within both information seeking research and information searching research, alternative models address similar issues in related ways and that the models are complementary rather than conflicting. Finally, an alternative, problem‐solving model is presented, which, it is suggested, provides a basis for relating the models in appropriate research strategies.

Models in information behaviour research

This paper proposes evaluation methods based on the use of non-dichotomous relevance judgements in IR experiments It is argued that evaluation methods should credit IR methods for their ability to retrieve highly relevant documents This is desirable from the user point of view in modem large IR environments The proposed methods are (1) a novel application of P-R curves and average precision computations based on separate recall bases for documents of different degrees of relevance, and (2) two novel measures computing the cumulative gain the user obtains by examining the retrieval result up to a given ranked position We then demonstrate the use of these evaluation methods in a case study on the effectiveness of query types, based on combinations of query structures and expansion, in retrieving documents of various degrees of relevance The test was run with a best match retrieval system (In- Query I) in a text database consisting of newspaper articles The results indicate that the tested strong query structures are most effective in retrieving highly relevant documents The differences between the query types are practically essential and statistically significant More generally, the novel evaluation methods and the case demonstrate that non-dichotomous relevance assessments are applicable in IR experiments, may reveal interesting phenomena, and allow harder testing of IR methods

IR evaluation methods for retrieving highly relevant documents

Abbreviated Contents Figures and Tables Preface Introduction and Examples Concepts Relevant to Information Behavior Models, Paradigms, and Theories in the Study of Information Behavior Methods for Studying Information Behavior Research Results and Reflections Appendix: Glossary Appendix: Questions for Discussion and Application References Index

Looking for Information: A Survey of Research on Information Seeking, Needs and Behavior

Information may be defined as the characteristics of the output of a process, these being informative about the process and the input. This discipline independent definition may be applied to all domains, from physics to epistemology. Hierarchies of processes, linked together, provide a communication channel between each of the corresponding functions and layers in the hierarchies. Models of communication (Shannon), perception, observation, belief, and knowledge are suggested that are consistent with this conceptual framework of information as the value of the output of any process in a hierarchy of processes. Misinformation and errors are considered. © 1997 John Wiley & Sons, Inc.

A discipline independent definition of information

Cet article propose une discussion concernant les differentes definitions de la notion de feedback qui ont emergees avec le developpement de la recherche d'informations et il etudie les relations des ces differentes vues avec la cybernetique et les modeles sociaux du feedback. Apres une definition du feedback face a la recherche d'information, les concepts et les modeles de feedback dans les domaines de la cybernetique et des sciences sociales sont discutes. Puis les approches traditionnelles et interactives de la recherche d'information face au feedback sont examinees. Et enfin, des recherches supplementaires concernant le feedback sont proposees

Feedback in Information Retrieval.

Preface. 1. Introduction. 2. Quantitative Reasoning. 3. Similarity and Retrieval Decisions. 4. Measuring Performance. 5. The Quality of A Ranking Method. 6. Performance with One Term. 7. Multivariate Probabilities. 8. Performance with Multiple Terms. 9. Logics and Rules. 10. Linguistic Knowledge. Bibliography. Index.

Text Retrieval and Filtering: Analytic Models of Performance

A probabilistic document-retrieval system may be seen as a sequential learning process, in which the system learns the characteristics of relevant documents, or more formally, it learns the parameters of probability distributions describing the frequencies of feature occurrences in relevant and nonrelevant documents. Probability distributions that may be used to describe the distribution of features include binary and Poisson distributions. Techniques for estimating the parameters of distributions are suggested. We have tested a proposal that parameters of distributions describing the distribution of features in nonrelevant documents be estimated from the parameters of the corresponding distributions of the entire database; the confidence parameter of such an estimate resulting in the highest average precision is given. Tests of several methods for estimating the parameters of distributions describing the distribution of features in relevant documents suggest that small values of the confidence parameter be used in our initial estimates of parameters for relevant documents. © 1988 John Wiley & Sons, Inc.

Parameter Estimation for Probabilistic Document-Retrieval Models.

The grammars of natural languages may be learned by using genetic algorithms that reproduce and mutate grammatical rules and part-of-speech tags, improving the quality of later generations of grammatical components. Syntactic rules are randomly generated and then evolve; those rules resulting in improved parsing and occasionally improved retrieval and filtering performance are allowed to further propagate. The LUST system learns the characteristics of the language or sublanguage used in document abstracts by learning from the document rankings obtained from the parsed abstracts. Unlike the application of traditional linguistic rules to retrieval and filtering applications, LUST develops grammatical structures and tags without the prior imposition of some common grammatical assumptions (e.g. part-of-speech assumptions), producing grammars that are empirically based and are optimized for this particular application.

/pdf/learning-syntactic-rules-and-tags-with-genetic-algorithms-4zaqtvuf64.pdf

Robert M. Losee

Papers

A discipline independent definition of information

Feedback in Information Retrieval.

Text Retrieval and Filtering: Analytic Models of Performance

Parameter Estimation for Probabilistic Document-Retrieval Models.

Learning syntactic rules and tags with genetic algorithms for information retrieval and filtering: an empirical basis for grammatical rules