The Turn: Integration of Information Seeking and Retrieval in Context (The Information Retrieval Series)

In the Web, making judgments of information quality and authority is a difficult task for most users because overall, there is no quality control mechanism. This study examines the problem of the judgment of information quality and cognitive authority by observing people's searching behavior in the Web. Its purpose is to understand the various factors that influence people's judgment of quality and authority in the Web, and the effects of those judgments on selection behaviors. Fifteen scholars from diverse disciplines participated, and data were collected combining verbal protocols during the searches, search logs, and postsearch interviews. It was found that the subjects made two distinct kinds of judgment: predictive judgment, and evaluative judgment. The factors influencing each judgment of quality and authority were identified in terms of characteristics of information objects, characteristics of sources, knowledge, situation, ranking in search output, and general assumption. Implications for Web design that will effectively support people's judgments of quality and authority are also discussed.

/pdf/judgement-of-information-quality-and-cognitive-authority-in-17ekf2rxg2.pdf

Judgement of information quality and cognitive authority in the Web

This article introduces the concept of relevance as viewed and applied in the context of IR evaluation, by presenting an overview of the multidimensional and dynamic nature of the concept. The literature on relevance reveals how the relevance concept, especially in regard to the multidimensionality of relevance, is many faceted, and does not just refer to the various relevance criteria users may apply in the process of judging relevance of retrieved information objects. From our point of view, the multidimensionality of relevance explains why some will argue that no consensus has been reached on the relevance concept. Thus, the objective of this article is to present an overview of the many different views and ways by which the concept of relevance is used--leading to a consistent and compatible understanding of the concept. In addition, special attention is paid to the type of situational relevance. Many researchers perceive situational relevance as the most realistic type of user relevance, and therefore situational relevance is discussed with reference to its potential dynamic nature, and as a requirement for interactive information retrieval (IIR) evaluation.

The concept of relevance in IR

The objective of this study was to describe the criteria mentioned by users evaluating the information within documents as it related to the users' information need situations. Data were collected by asking users in an academic environment to evaluate representations and the full text of documents that had been retrieved specifically for each user's information need situation. Users were asked to mark the portions of the document representations or of the full text of documents that indicated to the users whether they would or would not pursue the information within documents. An open-ended interview technique was then employed to discuss each marked portion with users. The interviews were audiotaped, the tapes transcribed, and the transcriptions were content analyzed in order to identify and describe evaluation criteria.



The results indicate that the criteria employed by users included tangible characteristics of documents (e.g., the information content of the document, the provision of references to other sources of information), subjective qualities (e.g., agreement with the information provided by the document) and situational factors (e.g., the time constraints under which the user was working). The implications of this research for our understanding of the concept of relevance, and for the design and evaluation of information retrieval systems, are discussed. © 1994 John Wiley & Sons, Inc.

User-defined relevance criteria: an exploratory study

Relevant: Having significant and demonstrable bearing on the matter at hand.

A version of this article has been published in 2006 as a chapter in E.G. Abels & D.A. Nitecki (Eds.), Advances in Librarianship (Vol. 30, pp. 3–71). San Diego: Academic Press. (Saracevic, 2006).

Relevance: The ability as of an information retrieval system to retrieve material that satisfies the needs of the user.

—Merriam-Webster Dictionary 2005

Relevance is a, if not even the, key notion in information science in general and information retrieval in particular. This two-part critical review traces and synthesizes the scholarship on relevance over the past 30 years and provides an updated framework within which the still widely dissonant ideas and works about relevance might be interpreted and related. It is a continuation and update of a similar review that appeared in 1975 under the same title, considered here as being Part I. The present review is organized into two parts: Part II addresses the questions related to nature and manifestations of relevance, and Part III addresses questions related to relevance behavior and effects. In Part II, the nature of relevance is discussed in terms of meaning ascribed to relevance, theories used or proposed, and models that have been developed. The manifestations of relevance are classified as to several kinds of relevance that form an interdependent system of relevances. In Part III, relevance behavior and effects are synthesized using experimental and observational works that incorporate data. In both parts, each section concludes with a summary that in effect provides an interpretation and synthesis of contemporary thinking on the topic treated or suggests hypotheses for future research. Analyses of some of the major trends that shape relevance work are offered in conclusions. © 2007 Wiley Periodicals, Inc.

/pdf/relevance-a-review-of-the-literature-and-a-framework-for-3jte83s7x1.pdf

Relevance: A review of the literature and a framework for thinking on the notion in information science. Part II: nature and manifestations of relevance

Experimental research in information retrieval (IR) depends on the idea of relevance. Because of its key role in IR, recent questions about relevance have raised issues of methodological concern and have shaken the philosophical foundations of IR theory development. Despite an existing set of theoretical definitions of this concept, our understanding of relevance from users' perspectives is still limited. Using naturalistic inquiry methodology, this article reports an empirical study of user-based relevance interpretations. A model is presented that reflects the nature of the thought processes of users who are evaluating bibliographic citations produced by a document retrieval system. Three major categories of variables affecting relevance assessments-internal context, external context, and problem context-are identified and described. Users' relevance assessments involve multiple layers of interpretations that are derived from individuals' experiences, perceptions, and private knowledge related to the pa...

/pdf/the-nature-of-relevance-in-information-retrieval-an-1jghwpgln1.pdf

Taemin Kim Park

Papers

The Nature of Relevance in Information Retrieval: An Empirical Study