A new method for automatic indexing and retrieval is described. The approach is to take advantage of implicit higher-order structure in the association of terms with documents (“semantic structure”) in order to improve the detection of relevant documents on the basis of terms found in queries. The particular technique used is singular-value decomposition, in which a large term by document matrix is decomposed into a set of ca. 100 orthogonal factors from which the original matrix can be approximated by linear combination. Documents are represented by ca. 100 item vectors of factor weights. Queries are represented as pseudo-document vectors formed from weighted combinations of terms, and documents with supra-threshold cosine values are returned. initial tests find this completely automatic method for retrieval to be promising.

/pdf/indexing-by-latent-semantic-analysis-4kzjkh4lw5.pdf

Indexing by Latent Semantic Analysis

The experimental evidence accumulated over the past 20 years indicates that textindexing systems based on the assignment of appropriately weighted single terms produce retrieval results that are superior to those obtainable with other more elaborate text representations. These results depend crucially on the choice of effective term weighting systems. This paper summarizes the insights gained in automatic term weighting, and provides baseline single term indexing models with which other more elaborate content analysis procedures can be compared.

/pdf/term-weighting-approaches-in-automatic-text-retrieval-4073ean1e5.pdf

Term Weighting Approaches in Automatic Text Retrieval

Statistical approaches to processing natural language text have become dominant in recent years This foundational text is the first comprehensive introduction to statistical natural language processing (NLP) to appear The book contains all the theory and algorithms needed for building NLP tools It provides broad but rigorous coverage of mathematical and linguistic foundations, as well as detailed discussion of statistical methods, allowing students and researchers to construct their own implementations The book covers collocation finding, word sense disambiguation, probabilistic parsing, information retrieval, and other applications

/pdf/foundations-of-statistical-natural-language-processing-1ou5e6c2q6.pdf

Foundations of Statistical Natural Language Processing

The automated categorization (or classification) of texts into predefined categories has witnessed a booming interest in the last 10 years, due to the increased availability of documents in digital form and the ensuing need to organize them. In the research community the dominant approach to this problem is based on machine learning techniques: a general inductive process automatically builds a classifier by learning, from a set of preclassified documents, the characteristics of the categories. The advantages of this approach over the knowledge engineering approach (consisting in the manual definition of a classifier by domain experts) are a very good effectiveness, considerable savings in terms of expert labor power, and straightforward portability to different domains. This survey discusses the main approaches to text categorization that fall within the machine learning paradigm. We will discuss in detail issues pertaining to three different problems, namely, document representation, classifier construction, and classifier evaluation.

/pdf/machine-learning-in-automated-text-categorization-e4pkjhyj3r.pdf

Machine learning in automated text categorization

Merchants selling products on the Web often ask their customers to review the products that they have purchased and the associated services. As e-commerce is becoming more and more popular, the number of customer reviews that a product receives grows rapidly. For a popular product, the number of reviews can be in hundreds or even thousands. This makes it difficult for a potential customer to read them to make an informed decision on whether to purchase the product. It also makes it difficult for the manufacturer of the product to keep track and to manage customer opinions. For the manufacturer, there are additional difficulties because many merchant sites may sell the same product and the manufacturer normally produces many kinds of products. In this research, we aim to mine and to summarize all the customer reviews of a product. This summarization task is different from traditional text summarization because we only mine the features of the product on which the customers have expressed their opinions and whether the opinions are positive or negative. We do not summarize the reviews by selecting a subset or rewrite some of the original sentences from the reviews to capture the main points as in the classic text summarization. Our task is performed in three steps: (1) mining product features that have been commented on by customers; (2) identifying opinion sentences in each review and deciding whether each opinion sentence is positive or negative; (3) summarizing the results. This paper proposes several novel techniques to perform these tasks. Our experimental results using reviews of a number of products sold online demonstrate the effectiveness of the techniques.

/pdf/mining-and-summarizing-customer-reviews-3kfih6aknk.pdf

Mining and summarizing customer reviews

The exhaustivity of document descriptions and the specificity of index terms are usually regarded as independent. It is suggested that specificity should be interpreted statistically, as a function of term use rather than of term meaning. The effects on retrieval of variations in term specificity are examined, experiments with three test collections showing in particular that frequently‐occurring terms are required for good overall performance. It is argued that terms should be weighted according to collection frequency, so that matches on less frequent, more specific, terms are of greater value than matches on frequent terms. Results for the test collections show that considerable improvements in performance are obtained with this very simple procedure.

A statistical interpretation of term specificity and its application in retrieval

This paper examines statistical techniques for exploiting relevance information to weight search terms. These techniques are presented as a natural extension of weighting methods using information about the distribution of index terms in documents in general. A series of relevance weighting functions is derived and is justified by theoretical considerations. In particular, it is shown that specific weighted search methods are implied by a general probabilistic theory of retrieval. Different applications of relevance weighting are illustrated by experimental results for test collections.

Relevance weighting of search terms

Chapter 1 Overall Introduction Chapter 2 History Chapter 3 Key Concepts Chapter 4 Evaluation Chapter 5 Models Chapter 6 Techniques Chapter 7 Systems Chapter 8 Extensions Chapter 9 Envoi

Readings in information retrieval

What are the key concepts? Healthy Ageing is relevant for everybody. It is de ned as the process of developing and maintaining the functional ability that enables wellbeing in older age. Functional ability is determined by the person’s intrinsic capacity (the combination of all the individual’s physical and mental capacities), relevant environmental factors, and the interaction between the two. Environmental factors include policies, systems, and services related to transport, housing, social protection, streets and parks, social facilities, and health and long-term care; politics; products and technologies; relationships with friends, family, and care givers; and cultural and social attitudes and values. What will the strategy achieve? The vision of the Strategy is a world in which everyone can live a long and healthy life. Over the next ve years, goals include implementing existing evidence to maximize functional ability, lling the evidence gaps and establishing partnerships to ensure a Decade of Healthy Ageing from 2020 – 2030. The strategy will be achieved through a focus on five strategic objectives. The Strategy’s implementation should be underpinned by the following principles: human rights, equity, equality and non-discrimination (particularly on the basis of age), gender equality, and intergenerational solidarity.

Key concepts

This paper reviews research on automatic summarising in the last decade. This work has grown, stimulated by technology and by evaluation programmes. The paper uses several frameworks to organise the review, for summarising itself, for the factors affecting summarising, for systems, and for evaluation.

The review examines the evaluation strategies applied to summarising, the issues they raise, and the major programmes. It considers the input, purpose and output factors investigated in recent summarising research, and discusses the classes of strategy, extractive and non-extractive, that have been explored, illustrating the range of systems built.

The conclusions drawn are that automatic summarisation has made valuable progress, with useful applications, better evaluation, and more task understanding. But summarising systems are still poorly motivated in relation to the factors affecting them, and evaluation needs taking much further to engage with the purposes summaries are intended to serve and the contexts in which they are used.

/pdf/automatic-summarising-the-state-of-the-art-2gd3hob6kg.pdf

Karen Sparck Jones

Papers

A statistical interpretation of term specificity and its application in retrieval

Relevance weighting of search terms

Readings in information retrieval

Key concepts

Automatic summarising: The state of the art