This introductory text to statistical machine translation (SMT) provides all of the theories and methods needed to build a statistical machine translator, such as Google Language Tools and Babelfish. In general, statistical techniques allow automatic translation systems to be built quickly for any language-pair using only translated texts and generic software. With increasing globalization, statistical machine translation will be central to communication and commerce. Based on courses and tutorials, and classroom-tested globally, it is ideal for instruction or self-study, for advanced undergraduates and graduate students in computer science and/or computational linguistics, and researchers in natural language processing. The companion website provides open-source corpora and tool-kits.

/pdf/statistical-machine-translation-1yr1jtzcjp.pdf

Statistical Machine Translation

The technology underlying text search engines has advanced dramatically in the past decade. The development of a family of new index representations has led to a wide range of innovations in index storage, index construction, and query evaluation. While some of these developments have been consolidated in textbooks, many specific techniques are not widely known or the textbook descriptions are out of date. In this tutorial, we introduce the key techniques in the area, describing both a core implementation and how the core can be enhanced through a range of extensions. We conclude with a comprehensive bibliography of text indexing literature.

/pdf/inverted-files-for-text-search-engines-1fgfjm96s3.pdf

Inverted files for text search engines

コンピュータ・サイエンス : ACM computing surveys

We present a syntax-based statistical translation model. Our model transforms a source-language parse tree into a target-language string by applying stochastic operations at each node. These operations capture linguistic differences such as word order and case marking. Model parameters are estimated in polynomial time using an EM algorithm. The model produces word alignments that are better than those produced by IBM Model 5.

https://faculty.washington.edu/fxia/courses/LING580MT/yamada_2001.pdf

A Syntax-based Statistical Translation Model

Predictive modeling of consumer financial behavior, including determination of likely responses to particular marketing efforts, is provided by application of consumer transaction data to predictive models associated with merchant segments. The merchant segments are derived from the consumer transaction data based on co-occurrences of merchants in sequences of transactions. Merchant vectors represent specific merchants, and are aligned in a vector space as a function of the degree to which the merchants co-occur more or less frequently than expected. Supervised segmentation is applied to merchant vectors to form the merchant segments. Merchant segment predictive models provide predictions of spending in each merchant segment for any particular consumer, based on previous spending by the consumer. Consumer profiles describe summary statistics of each consumer's spending in the merchant segments, and across merchant segments. The consumer profiles include consumer vectors derived as summary vectors of selected merchants patronized by the consumer. Predictions of consumer behavior are made by applying nearest-neighbor analysis to consumer vectors, thus facilitating the targeting of promotional offers to consumers most likely to respond positively.

Predictive modeling of consumer financial behavior using supervised segmentation and nearest-neighbor matching

In this paper we review studies of the growth of the Internet and technologies that are useful for information search and retrieval on the Web. We present data on the Internet from several different sources, e.g., current as well as projected number of users, hosts, and Web sites. Although numerical figures vary, overall trends cited by the sources are consistent and point to exponential growth in the past and in the coming decade. Hence it is not surprising that about 85% of Internet users surveyed claim using search engines and search services to find specific information. The same surveys show, however, that users are not satisfied with the performance of the current generation of search engines; the slow retrieval speed, communication delays, and poor quality of retrieved results (e.g., noise and broken links) are commonly cited problems. We discuss the development of new techniques targeted to resolve some of the problems associated with Web-based information retrieval and speculate on future trends.

Information retrieval on the web

This paper describes the application of IBM TAKMI® for Biomedical Documents to facilitate knowledge discovery from the very large text databases characteristic of life science and healthcare applications. This set of tools, designated MedTAKMI, is an extension of the TAKMI (Text Analysis and Knowledge MIning) system originally developed for text mining in customer-relationship-management applications. MedTAKMI dynamically and interactively mines a collection of documents to obtain characteristic features within them. By using multifaceted mining of these documents together with biomedically motivated categories for term extraction and a series of drill-down queries, users can obtain knowledge about a specific topic after seeing only a few key documents. In addition, the use of natural language techniques makes it possible to extract deeper relationships among biomedical concepts. The MedTAKMI system is capable of mining the entire MEDLINE® database of 11 million biomedical journal abstracts. It is currently running at a customer site.

/pdf/a-text-mining-system-for-knowledge-discovery-from-biomedical-4bkdc3zfr7.pdf

A text-mining system for knowledge discovery from biomedical documents

This paper introduces an overview of the RITE (Recognizing Inference in TExt) task in NTCIR-9. We evaluate systems that automatically recognize entailment, paraphrase, and contradiction between two texts written in Japanese, Simplified Chinese, or Traditional Chinese. The task consists of four subtasks: Binary classification of entailment (BC); Multi-class classification including paraphrase and contradiction (MC); and two extrinsic application-oriented datasets: Entrance Exam and RITE4QA. This paper also describes how we built the test collection, evaluation metrics, and evaluation results of the submitted runs.

/pdf/overview-of-ntcir-9-rite-recognizing-inference-in-text-4hy1mi85ob.pdf

Overview of NTCIR-9 RITE : Recognizing Inference in TExt

In a machine translation method and apparatus, an input text analyzer analyzes an input text in a first language to form a graph structure of nodes and edges. An equivalent selector introduces variables to ambiguous components of the graph structure by a variable introducer. A constraint solver propagates constraints on the variables to other components. Solutions are then found for equivalent determination and locally consistent solutions are also found for the respective components. A variable realizer finds solutions to the respective locally consistent components according to the patterns of the variables. An output text generator then generates text in a second language from a set of these solutions. Thus, the method and apparatus enable avoidance of an explosion of combinations between source words and equivalents capable of occurring in the analysis phase and also enables the optimum use of information through deferring the determination of words or phrases in the second language of ambiguous components to the text generation phase.

Machine translation method and apparatus

A machine translation method and system for improving the accuracy of the selection of appropriate words, without incurring any deterioration of the processing efficiency, includes registering, during the translation of a document by using a compound word dictionary, elemental word information of an applied compound word in a discourse dictionary, and employing a plurality of dictionaries, including the discourse dictionary, to translate the document without using a compound word dictionary.

Koichi Takeda

Papers

Information retrieval on the web

A text-mining system for knowledge discovery from biomedical documents

Overview of NTCIR-9 RITE : Recognizing Inference in TExt

Machine translation method and apparatus

Translation method and system