The technology underlying text search engines has advanced dramatically in the past decade. The development of a family of new index representations has led to a wide range of innovations in index storage, index construction, and query evaluation. While some of these developments have been consolidated in textbooks, many specific techniques are not widely known or the textbook descriptions are out of date. In this tutorial, we introduce the key techniques in the area, describing both a core implementation and how the core can be enhanced through a range of extensions. We conclude with a comprehensive bibliography of text indexing literature.

/pdf/inverted-files-for-text-search-engines-1fgfjm96s3.pdf

Inverted files for text search engines

This paper compares text retrieval methods intended for office systems. The operational requirements of the office environment are discussed, and retrieval methods from database systems and from information retrieval systems are examined. We classify these methods and examine the most interesting representatives of each class. Attempts to speed up retrieval with special purpose hardware are also presented, and issues such as approximate string matching and compression are discussed. A qualitative comparison of the examined methods is presented. The signature file method is discussed in more detail.

Access methods for text

Information retrieval systems contain large volumes of text, and currently have typical sizes into the gigabyte range. Inverted indexes are one important method for providing search facilities into these collections, but unless compressed require a great deal of space. In this paper we introduce a new method for compressing inverted indexes that yields excellent compression, fast decoding, and exploits clustering—the tendency for words to appear relatively frequently in some parts of the collection and infrequently in others. We also describe two other quite separate applications for the same compression method: representing the MTF list positions generated by the Burrows-Wheeler Block Sorting transformations and transmitting the codebook for semi-static block-based minimum-redundancy coding.

Binary Interpolative Coding for Effective Index Compression

One of the first projects in computer analysis of natural language was to devise procedures for representing the subject content of a document by a few text-derived terms, a process called automatic indexing. Although many developments have taken place over the years, the essential techniques of automatic indexing continue to focus on answering three basic questions: 1. How can automatic indexing be made to more adequately represent the subject content of a document? 2. How can automatic indexing improve recall by increasing the number of relevant documents retrieved? 3. How can automatic indexing improve precision by decreasing the number of non-relevant documents retrieved? During this tutorial session, we will review the advances that have been made, and the procedures that have been devised, to improve the effectiveness of automatic indexing. We will then consider the question of whether online retrieval systems can rely entirely on automatic indexing to achieve adequate retrieval effectiveness or whether some manual pre-indexing is still necessary.

Automatic indexing

Instead of using citations or marketing academics™ perceptual ranking of journals, this article examines the ranking of marketing journals using Australian university library holdings, in either hard copy or full-text electronic format. This measure was used as a proxy for broad-based accessibility of marketing journals. The study found that the accessibility rankings differed significantly from the most recent U.S. perceptual rankings, and it is suggested that in some situations, the accessibility ranking may be a more appropriate measure than other approaches. An examination of journal characteristics and their relationship to holdings in Australian university libraries was also undertaken. It was found that the year in which the journal started publication and its perceived importance within the United States (i.e., perceptual ranking) had a statistical impact on the proportion of Australian university libraries holding the journal.

Accessibility: An Alternative Method of Ranking Marketing Journals?:

The storage requirements for retrieval systems utilizing inverted files are calculated assuming different storage modes. Various methods for compression of these large files are analyzed. Binary vectors compressed by run-length coding as well as lists of document numbers were found to be suitable. The problem of minimal storage requirements for the inverted file is solved for different assumptions about index term distributions. A representation combining run-length coded binary vectors with list of document numbers was found to be the most economical. Parameter values for this minimum storage form are calculated and specified in tables as well as displayed graphically.

Compression of large inverted files with hyperbolic term distribution

The structure of a retrospective information retrieval system that uses equifrequent word or text fragments is described, and its advantages over word oriented systems are mentioned briefly. Word fragments are proposed as retrieval elements, and a discussion is given of the changes required in order to process a query. Some necessary modifications in the treatment of logical operators are described, and two conditions are postulated as necessary for the successful operation of the system. Aspects of query processing are illustrated by experimental results obtained from single and two-term queries applied to a portion of the MARC tapes.

Query processing in a retrospective document retrieval system that uses word fragments as language elements

An Automatic Document Indexing System Based on Cooperating Expert Systems: Design and Development.

Early and current approaches to automatic indexing of document databases are reviewed, together with some attempts at incorporating artificial intelligence techniques into the indexing process. A fully automatic indexing system currently under development is described. It combines pre-indexing by statistical techniques with knowledge based indexing carried out by two cooperating expert systems. The architecture of the system, databases and functions of each module are outlined. Reasons that justify the approach are given.

Automatic Indexing of Document Databases by Cooperating Expert Systems

This article describes a study of personal journal subscriptions of university faculty teaching in arts and professional programs at an undergraduate liberal arts university. Based on personal interviews, this study examines subscriptions with regard to reason for subscription, classification of journal, and importance to the discipline and the faculty member. Overlap with the holdings of the university library, financing of personal subscriptions, and style of reading are also investigated. Results from the two groups are presented separately and compared with the results that were described in part I for the faculty of science. © 1994 John Wiley & Sons, Inc.

Ernst J. Schuegraf

Papers

Compression of large inverted files with hyperbolic term distribution

Query processing in a retrospective document retrieval system that uses word fragments as language elements

An Automatic Document Indexing System Based on Cooperating Expert Systems: Design and Development.

Automatic Indexing of Document Databases by Cooperating Expert Systems

An analysis of personal journal subscriptions of university faculty. Part II: arts and professional programs