Journal ArticleDOI
The Simple Scalability of Documents.
Reads0
Chats0
TLDR
The test results indicate that, with some reservations, this theory of scaling is applicable to documents, and this finding is further applied to the construction of test collections for Information Retrieval research that could more sensitively measure retrieval system alterations through the use of documents scaled not merely by relevance, but rather, by preference.Abstract:
The relationship between scaling practice and scaling theory remains a controversial problem in Information Retrieval research and experimentation. This article reports a test of a general theory of scaling, i.e., Simple Scalability, applied to the stimulus domain of documents represented as abstracts. The significance of Simple Scalability is that it implies three important properties of scales: transitivity, substitutibility, and independence. The test results indicate that, with some reservations, this theory of scaling is applicable to documents. This finding is further applied to the construction of test collections for Information Retrieval research that could more sensitively measure retrieval system alterations through the use of documents scaled not merely by relevance, but rather, by preference. © 1990 John Wiley & Sons, Inc.read more
Citations
More filters
Proceedings ArticleDOI
Retrieval evaluation with incomplete information
Chris Buckley,Ellen M. Voorhees +1 more
TL;DR: It is shown that current evaluation measures are not robust to substantially incomplete relevance judgments, and a new measure is introduced that is both highly correlated with existing measures when complete judgments are available and more robust to incomplete judgment sets.
Journal ArticleDOI
A re-examination of relevance: toward a dynamic, situational definition
TL;DR: This paper reviews literature over the last 30 years that presents various views of relevance as topical, user-oriented, multidimensional, cognitive, and dynamic to build a case for an approach to the problem of definition based on alternative assumptions.
Book
Test Collection Based Evaluation of Information Retrieval Systems
TL;DR: This tutorial and review shows that despite its age, this long-standing evaluation method is still a highly valued tool for retrieval research.
Here or there: preference judgments for relevance
TL;DR: This work hypothesizes that preference judgments of the form "document A is more relevant than document B" are easier for assessors to make than absolute judgments, and investigates methods to evaluate search engines using preference judgments.
Proceedings ArticleDOI
System effectiveness, user models, and user utility: a conceptual framework for investigation
TL;DR: A conceptual framework for analyzing effectiveness measures based on classifying members of this broad family of measures into four distinct families, each of which reflects a different notion of system utility.