The concept of maximum entropy can be traced back along multiple threads to Biblical times. Only recently, however, have computers become powerful enough to permit the widescale application of this concept to real world problems in statistical estimation and pattern recognition. In this paper, we describe a method for statistical modeling based on maximum entropy. We present a maximum-likelihood approach for automatically constructing maximum entropy models and describe how to implement this approach efficiently, using as examples several problems in natural language processing.

/pdf/a-maximum-entropy-approach-to-natural-language-processing-57psgipa7w.pdf

A maximum entropy approach to natural language processing

Speech recognition is formulated as a problem of maximum likelihood decoding. This formulation requires statistical models of the speech production process. In this paper, we describe a number of statistical models for use in speech recognition. We give special attention to determining the parameters for such models from sparse data. We also describe two decoding methods, one appropriate for constrained artificial languages and one appropriate for more realistic decoding tasks. To illustrate the usefulness of the methods described, we review a number of decoding results that have been obtained with them.

A Maximum Likelihood Approach to Continuous Speech Recognition

The use of hidden Markov models for speech recognition has become predominant in the last several years, as evidenced by the number of published papers and talks at major speech conferences. The reasons this method has become so popular are the inherent statistical (mathematically precise) framework; the ease and availability of training algorithms for cstimating the parameters of the models from finite training sets of speech data; the flexibility of the resulting recognition system in which one can easily change the size, type, or architecture of the models to suit particular words, sounds, and so forth; and the ease of implementation of the overall recognition system. In this expository article, we address the role of statistical methods in this powerful technology as applied to speech recognition and discuss a range of theoretical and practical issues that are as yet unsolved in terms of their importance and their effect on performance for different system implementations.

/pdf/hidden-markov-models-for-speech-recognition-1vqk5sdtvy.pdf

Hidden Markov models for speech recognition

An intelligent automated assistant system engages with the user in an integrated, conversational manner using natural language dialog, and invokes external services when appropriate to obtain information or perform various actions. The system can be implemented using any of a number of different platforms, such as the web, email, smartphone, and the like, or any combination thereof. In one embodiment, the system is based on sets of interrelated domains and tasks, and employs additional functionally powered by external services with which the system can interact.

Intelligent Automated Assistant

The Hearsay-II system, developed during the DARPA-sponsored five-year speech-understanding research program, represents both a specific solution to the speech-understanding problem and a general framework for coordinating independent processes to achieve cooperative problem-solving behavior. As a computational problem, speech understanding reflects a large number of intrinsically interesting issues. Spoken sounds are achieved by a long chain of successive transformations, from intentions, through semantic and syntactic structuring, to the eventually resulting audible acoustic waves. As a consequence, interpreting speech means effectively inverting these transformations to recover the speaker's intention from the sound. At each step in the interpretive process, ambiguity and uncertainty arise.

The Hearsay-II problem-solving framework reconstructs an intention from hypothetical interpretations formulated at various levels of abstraction. In addition, it allocates limited processing resources first to the most promising incremental actions. The final configuration of the Hearsay-II system comprises problem-solving components to generate and evaluate speech hypotheses, and a focus-of-control mechanism to identify potential actions of greatest value. Many of these specific procedures reveal novel approaches to speech problems. Most important, the system successfully integrates and coordinates all of these independent activities to resolve uncertainty and control combinatorics. Several adaptations of the Hearsay-II framework have already been undertaken in other problem domains, and it is anticipated that this trend will continue; many future systems necessarily will integrate diverse sources of knowledge to solve complex problems cooperatively.

Discussed in this paper are the characteristics of the speech problem in particular, the special kinds of problem-solving uncertainty in that domain, the structure of the Hearsay-II system developed to cope with that uncertainty, and the relationship between Hearsay-II's structure and those of other speech-understanding systems. The paper is intended for the general computer science audience and presupposes no speech or artificial intelligence background.

/pdf/the-hearsay-ii-speech-understanding-system-integrating-4upd3ocgip.pdf

The Hearsay-II Speech-Understanding System: Integrating Knowledge to Resolve Uncertainty

Preliminary results have been obtained with a system for recognizing continuously read sentences from a naturally-occurring corpus (Laser Patents), restricted to a 1000-word vocabulary. Our model of the task language has an entropy of about 4.8 bits/word and a perplexity of 21.11 words. Many new problems arise in recognition of a substantial natural corpus (compared to recognition of an artificially constrained language). Some techniques are described for treating these problems. On a test set consisting of 20 sentences having a total of 486 words, there was a word error rate of 33.1%.

Recognition of continuously read natural corpus

A method and system for reducing perplexity in a speech recognition system based upon determined geographic location. In a mobile speech recognition system which processes input frames of speech against stored templates representing speech, a core library of speech templates is created and stored representing a basic vocabulary of speech. Multiple location-specific libraries of speech templates are also created and stored, each library containing speech templates representing a specialized vocabulary for a specific geographic location. The geographic location of the mobile speech recognition system is then periodically determined utilizing a cellular telephone system, a geopositioning satellite system or other similar systems and a particular one of the location-specific libraries of speech templates is identified for the current location of the system. Input frames of speech are then processed against the combination of the core library and the particular location-specific library to greatly enhance the accuracy and efficiency of speech recognition by the system. Each location-specific library preferably includes speech templates representative of location place names, proper names, and business establishments within a specific geographic location.

Method and system for location-specific speech recognition

A method and system are disclosed for reducing perplexity in a speech recognition system within a telephonic network based upon determined caller identity. In a speech recognition system which processes input frames of speech against stored templates representing speech, a core library of speech templates is created and stored representing a basic vocabulary of speech. Multiple caller-specific libraries of speech templates are also created and stored, each library containing speech templates which represent a specialized vocabulary and pronunciations for a specific geographic location and a particular individual. Additionally, the caller-specific libraries of speech templates are preferably processed to reflect the reduced bandwidth, transmission channel variations and other signal variations introduced into the system via a telephonic network. The identification of a caller is determined upon connection to the network via standard caller identification circuitry and upon detection of a spoken utterance, that utterance is processed against the core library, if the caller's identity cannot be determined, or against a particular caller-specific library, if the caller's identity can be determined, thereby greatly enhancing the efficiency and accuracy of speech recognition by the system.

Method and system for reducing perplexity in speech recognition via caller identification

Further results have been obtained on the recognition of continuously read sentences from a natural language corpus of laser patents. The vocabulary is limited to the 1000 most frequently occurring words in the corpus. Our model of the task language has a perplexity of 24.1 words (corresponding to an entropy of 4.6 bits/word). This paper describes modifications and improvements to the system which have resulted in the lowering of the word error rate from the previously reported 33.1% to 8.9%.

Further results on the recognition of a continuously read natural corpus

A continuous speech recognition system having a speech processor and a word recognition computer subsystem, characterized by an element for developing a graph for confluent links between confluent nodes; an element for developing a graph of boundary links between adjacent words; an element for storing an inventory of confluent links and boundary links as a coding inventory; an element for converting an unknown utterance into an encoded sequence of confluent links and boundary links corresponding to recognition sequences stored in the word recognition subsystem recognition vocabulary for speech recognition. The invention also includes a method for achieving continouous speech recognition by characterizing speech as a sequence of confluent links which are matched with candidate words. The invention also applies to isolated word speech recognition as with continuous speech recognition, except that in such case there are no boundary links.

Paul S. Cohen

Papers

Recognition of continuously read natural corpus

Method and system for location-specific speech recognition

Method and system for reducing perplexity in speech recognition via caller identification

Further results on the recognition of a continuously read natural corpus

Speech recognition system with efficient storage and rapid assembly of phonological graphs