We introduce (1) a novel stochastic inversion transduction grammar formalism for bilingual language modeling of sentence-pairs, and (2) the concept of bilingual parsing with a variety of parallel corpus analysis applications. Aside from the bilingual orientation, three major features distinguish the formalism from the finite-state transducers more traditionally found in computational linguistics: it skips directly to a context-free rather than finite-state base, it permits a minimal extra degree of ordering flexibility, and its probabilistic formulation admits an efficient maximum-likelihood bilingual parsing algorithm. A convenient normal form is shown to exist. Analysis of the formalism's expressiveness suggests that it is particularly well suited to modeling ordering shifts between languages, balancing needed flexibility against complexity constraints. We discuss a number of examples of how stochastic inversion transduction grammars bring bilingual constraints to bear upon problematic corpus analysis tasks such as segmentation, bracketing, phrasal alignment, and parsing.

Stochastic inversion transduction grammars and bilingual parsing of parallel corpora

The dramatic growth of the Internet has created a new problem for users: location of the relevant sources of documents. This article presents a framework for (and experimentally analyzes a solution to) this problem, which we call the text-source discovery problem. Our approach consists of two phases. First, each text source exports its contents to a centralized service. Second, users present queries to the service, which returns an ordered list of promising text sources. This article describes GlOSS, Glossary of Servers Server, with two versions: bGlOSS, which provides a Boolean query retrieval model, and vGlOSS, which provides a vector-space retrieval model. We also present hGlOSS, which provides a decentralized version of the system. We extensively describe the methodology for measuring the retrieval effectiveness of these systems and provide experimental evidence, based on actual data, that all three systems are highly effective in determining promising text sources for a given query.

/pdf/gloss-text-source-discovery-over-the-internet-3cywiuxkzz.pdf

GlOSS: text-source discovery over the Internet

The initial stage of text analysis for any NLP task usually involves the tokenization of the input into words. For languages like English one can assume, to a first approximation, that word boundaries are given by whitespace or punctuation. In various Asian languages, including Chinese, on the other hand, whitespace is never used to delimit words, so one must resort to lexical information to "reconstruct" the word-boundary information. In this paper we present a stochastic finite-state model wherein the basic workhorse is the weighted finite-state transducer. The model segments Chinese text into dictionary entries and words derived by various productive lexical processes, and---since the primary intended application of this model is to text-to-speech synthesis---provides pronunciations for these words. We evaluate the system's performance by comparing its segmentation "judgments" with the judgements of a pool of human segmenters, and the system is shown to perform quite well.

A stochastic finite-state word-segmentation algorithm for Chinese

urgent need to promote Chinese in this paper we will raise the significance of keyword extraction using a new PAT-treebased approach, which is efficient in automatic keyword extraction from a set of relevant Chinese documents. This approach has been successfully applied in several IR researches, such as document classification, book indexing and relevance feedback. Many Chinese language processing applications therefore step ahead from character level to word/phrase level,

PAT-tree-based keyword extraction for Chinese information retrieval

This article presents a pragmatic approach to Chinese word segmentation. It differs from most previous approaches mainly in three respects. First, while theoretical linguists have defined Chinese words using various linguistic criteria, Chinese words in this study are defined pragmatically as segmentation units whose definition depends on how they are used and processed in realistic computer applications. Second, we propose a pragmatic mathematical framework in which segmenting known words and detecting unknown words of different types (i.e., morphologically derived words, factoids, named entities, and other unlisted words) can be performed simultaneously in a unified way. These tasks are usually conducted separately in other systems. Finally, we do not assume the existence of a universal word segmentation standard that is application-independent. Instead, we argue for the necessity of multiple segmentation standards due to the pragmatic fact that different natural language processing applications might require different granularities of Chinese words.These pragmatic approaches have been implemented in an adaptive Chinese word segmenter, called MSRSeg, which will be described in detail. It consists of two components: (1) a generic segmenter that is based on the framework of linear mixture models and provides a unified approach to the five fundamental features of word-level Chinese language processing: lexicon word processing, morphological analysis, factoid detection, named entity recognition, and new word identification; and (2) a set of output adaptors for adapting the output of (1) to different application-specific standards. Evaluation on five test sets with different standards shows that the adaptive system achieves state-of-the-art performance on all the test sets.

/pdf/chinese-word-segmentation-and-named-entity-recognition-a-4je2hliymz.pdf

Chinese Word Segmentation and Named Entity Recognition: A Pragmatic Approach

Present text retrieval systems are generally built on the reductionist basis that words in texts (keywords) are used as indexing terms to represent the texts. A necessary precursor to these systems is word extraction which, for English texts, can be achieved automatically by using spaces and punctuations as word delimiters. This cannot be readily applied to Chinese texts because they do not have obvious word boundaries. A Chinese text consists of a linear sequence of nonspaced or equally spaced ideographic characters, which are similar to morphemes in English. Researchers of Chinese text retrieval have been seeking methods of text segmentation to divide Chinese texts automatically into words. First, a review of these methods is provided in which the various different approaches to Chinese text segmentation have been classified in order to provide a general picture of the research activity in this area. Some of the most important work is described. There follows a discussion of the problems of Chinese text segmentation with examples to illustrate. These problems include morphological complexities, segmentation ambiguity, and parsing problems, and demonstrate that text segmentation remains one of the most challenging and interesting areas for Chinese text retrieval. © 1993 John Wiley & Sons, Inc.

Chinese text segmentation for text retrieval: achievements and problems

Text segmentation is a prerequisite for text retrieval systems. Chinese texts cannot be readily segmented into words because they do not contain word boundaries. ACTS is an automatic Chinese text segmentation proto-type for Chinese full text retrieval. It applies partial syntactic analysis—the analysis of morphemes, words, and phrases. The idea was originally largely inspired by experiments on English morpheme and phrase-analysis-based text retrieval, which are particularly germane to Chinese, because neither Chinese nor English texts have morpheme and phrase boundaries. ACTS is built on the hypothesis that Chinese words and phrases exceeding two characters can be characterized by a grammar that describes the concatenation behavior of the morphological and syntactic categories of their formatives. This is examined through three procedures: (1) Segmentation—texts are divided into one and two character segments by matching against a dictionary; (2) Category disambiguation—the syntactic categories of segments are determined according to context; (3) Parsing—the segments are analyzed based on the grammar, and subsequently combined into compound and complex words for indexing and retrieval. The experimental results, based on a small sample of 30 texts, show that most significant words and phrases in these texts can be extracted with a high degree of accuracy. © 1995 John Wiley & Sons, Inc.

ACTS: an automatic Chinese text segmentation system for full text retrieval

Although many books on the Internet are available, this one concentrates on aspects relevant to library and information professionals. For those wishing to participate in the professional communication taking place on the Net, it takes the mystique out of networking and provides a manageable guide to what is on offer. Set out clearly and concisely, it should help users find their way into and about the Internet from a library and information perspective. The third edition has been comprehensively revised to keep up with rapid change in this area, including advances in technology for: connecting to the Internet; free e-mail services; dealing with viruses; real-time services such as video conferencing; Java, multimedia plug-ins; web-cams; portals; online shopping; meta search engines; and metadata. The selective and updated resource guide section provides evaluative guidance and advice on how to locate and retrieve key resources that clients and customers might need. The resource guide is regularly updated and available on the Web, providing ongoing support for the book. Materials covered in the resource guide include: discussion lists and electronic conferences; electronic journals, newsletters and alerting services; general reference works; software; networking guides and training materials; information servers and gateways including bulletin boards, gophers and world wide web sites; subject listing and indexing services; and OPACs and library-based information servers. This guide is designed to be of use to all students and practitioners of library and information services. It serves as both a training tool and a desk book. It assumes no prior knowledge, but goes beyond the basics to offer advanced techniques for getting the best out of the Internet.

The Library and Information Professional's Guide to the Internet

Courses in library and information studies (LIS) departments in universities and polytechnics in the UK began to involve computers early, and now make heavy use of them. However, only a very limited amount of specifically pedagogic software is used, and the amount of this in use has actually fallen as pes have replaced the use of institutional mainframes. Most of the schools now concentrate on giving students hands-on experience of packages that they are likely to encounter in their future careers, the packages falling into two groups: general office systems, such as word processors, spreadsheets and DBMS; and commercial library and information systems, such as online information retrieval systems and library housekeeping packages. There is potential for the introduction of pedagogic software developed for other disciplines, and of LIS teaching software developed abroad, especially in the USA.

Computer Methods in the Teaching of Library and Information Studies.

The development and testing of an expert system to assist with the selection of online business databases for UK company information is described. The Company Information Database Adviser (CIDA) was designed for novice and end- user searchers and covers seven categories of company infor mation : factual (directory) details, financial data, company news, company structure, mergers and acquisitions (M & A) activity, market information and company profiles. The devel opment tool was the Leonardo expert system shell. Knowledge acquisition included three knowledge elicita tion sessions with each of four experienced business informa tion intermediaries. Outputs of knowledge acquisition were: (i) factual and evaluative information for over 200 database/ host combinations, and (ii) rules, expressed diagrammatically, for database selection within each category of company infor mation. These were used to build a prototype which was examined by intermediaries and further refined. The design is outlined and the rules...

Gwyneth Tseng

Papers

Chinese text segmentation for text retrieval: achievements and problems

ACTS: an automatic Chinese text segmentation system for full text retrieval

The Library and Information Professional's Guide to the Internet

Computer Methods in the Teaching of Library and Information Studies.

CIDA: the expert Company Information Database Adviser