This tutorial provides an overview of the basic theory of hidden Markov models (HMMs) as originated by L.E. Baum and T. Petrie (1966) and gives practical details on methods of implementation of the theory along with a description of selected applications of the theory to distinct problems in speech recognition. Results from a number of original sources are combined to provide a single source of acquiring the background required to pursue further this area of research. The author first reviews the theory of discrete Markov chains and shows how the concept of hidden states, where the observation is a probabilistic function of the state, can be used effectively. The theory is illustrated with two simple examples, namely coin-tossing, and the classic balls-in-urns system. Three fundamental problems of HMMs are noted and several practical techniques for solving these problems are given. The various types of HMMs that have been studied, including ergodic as well as left-right models, are described. >

/pdf/a-tutorial-on-hidden-markov-models-and-selected-applications-1vlzbxbk68.pdf

A tutorial on hidden Markov models and selected applications in speech recognition

Statistical approaches to processing natural language text have become dominant in recent years This foundational text is the first comprehensive introduction to statistical natural language processing (NLP) to appear The book contains all the theory and algorithms needed for building NLP tools It provides broad but rigorous coverage of mathematical and linguistic foundations, as well as detailed discussion of statistical methods, allowing students and researchers to construct their own implementations The book covers collocation finding, word sense disambiguation, probabilistic parsing, information retrieval, and other applications

/pdf/foundations-of-statistical-natural-language-processing-1ou5e6c2q6.pdf

Foundations of Statistical Natural Language Processing

The concept of maximum entropy can be traced back along multiple threads to Biblical times. Only recently, however, have computers become powerful enough to permit the widescale application of this concept to real world problems in statistical estimation and pattern recognition. In this paper, we describe a method for statistical modeling based on maximum entropy. We present a maximum-likelihood approach for automatically constructing maximum entropy models and describe how to implement this approach efficiently, using as examples several problems in natural language processing.

/pdf/a-maximum-entropy-approach-to-natural-language-processing-57psgipa7w.pdf

A maximum entropy approach to natural language processing

Recurrent neural networks are powerful sequence learners. They are able to incorporate context information in a flexible way, and are robust to localised distortions of the input data. These properties make them well suited to sequence labelling, where input sequences are transcribed with streams of labels. The aim of this thesis is to advance the state-of-the-art in supervised sequence labelling with recurrent networks. Its two main contributions are (1) a new type of output layer that allows recurrent networks to be trained directly for sequence labelling tasks where the alignment between the inputs and the labels is unknown, and (2) an extension of the long short-term memory network architecture to multidimensional data, such as images and video sequences.

https://mediatum.ub.tum.de/doc/673554/file.pdf

Supervised Sequence Labelling with Recurrent Neural Networks

1. Introduction to text mining 2. Core text mining operations 3. Text mining preprocessing techniques 4. Categorization 5. Clustering 6. Information extraction 7. Probabilistic models for Information extraction 8. Preprocessing applications using probabilistic and hybrid approaches 9. Presentation-layer considerations for browsing and query refinement 10. Visualization approaches 11. Link analysis 12. Text mining applications Appendix Bibliography.

/pdf/the-text-mining-handbook-advanced-approaches-in-analyzing-ge55wa29fm.pdf

The Text Mining Handbook: Advanced Approaches in Analyzing Unstructured Data

In this paper we present some experiments on the use of a probabilistic model to tag English text, i.e. to assign to each word the correct tag (part of speech) in the context of the sentence. The main novelty of these experiments is the use of untagged text in the training of the model. We have used a simple triclass Markov model and are looking for the best way to estimate the parameters of this model, depending on the kind and amount of training data provided. Two approaches in particular are compared and combined:using text that has been tagged by hand and computing relative frequency counts,using text without tags and training the model as a hidden Markov process, according to a Maximum Likelihood principle.Experminents show that the best training is obtained by using as much tagged text as possible. They also show that Maximum Likelihood training, the procedure that is routinely used to estimate hidden Markov models parameters from training data, will not necessarily improve the tagging accuracy. In fact, it will generally degrade this accuracy, except when only a limited amount of hand-tagged text is available.

Tagging English text with a probabilistic model

In the case of a trigram language model, the probability of the next word conditioned on the previous two words is estimated from a large corpus of text. The resulting static trigram language model (STLM) has fixed probabilities that are independent of the document being dictated. To improve the language model (LM), one can adapt the probabilities of the trigram language model to match the current document more closely. The partially dictated document provides significant clues about what words are more likely to be used next. Of many methods that can be used to adapt the LM, we describe in this paper a simple model based on the trigram frequencies estimated from the partially dictated document. We call this model a cache trigram language model (CTLM) since we are caching the recent history of words. We have found that the CTLM reduces the perplexity of a dictated document by 23%. The error rate of a 20,000-word isolated word recognizer decreases by about 5% at the beginning of a document and by about 24% after a few hundred words.

A dynamic language model for speech recognition

In this paper, we study the automatic construction of personalized TV News programs, where we want to build a program with predefined duration and maximum content value for a specific user. We combine video indexing techniques to parse TV News recordings into stories, and information filtering techniques to select stories which are most adequate given the user profile. We formalize the selection process as an optimization problem, and we study how to take into account duration in the selection of stories. Experiments show that a simple heuristic can provide high quality selection with little computation. We also describe two prototypes, which implement two different mechanisms for the construction of user profiles: explicit specification, using a category-based model,implicit specification, using a keyword-based model.

Automatic construction of personalized TV news programs

This paper relates different kinds of language modeling methods that can be applied to the linguistic decoding part of a speech recognition system with a very large vocabulary. These models are studied experimentally on a pseudophonetic input arising from French stenotypy. We propose a model which combines the advantages of a statistical modeling with information theoretic tools, and those of a grammatical approach.

Natural Language Modeling for Phoneme-to-Text Transcription

This paper presents a novel and effective approach for multi-video summarization: Video Maximal Marginal Relevance (Video-MMR), which extends a classical algorithm of text summarization, Maximal Marginal Relevance. Video-MMR rewards relevant keyframes and penalizes redundant keyframes, as MMR does with text fragments. Two variants of Video-MMR are suggested, and we propose a criterion to select the best combination of parameters for Video-MMR. Then, we compare two summarization strategies: Global Summarization, which summarizes all the individual videos at the same time, and Individual Summarization, which summarizes each individual video independently and concatenates the results. Finally, Video-MMR algorithm is compared with popular K-means algorithm, supported by user-made summary.

Bernard Merialdo

Papers

Tagging English text with a probabilistic model

A dynamic language model for speech recognition

Automatic construction of personalized TV news programs

Natural Language Modeling for Phoneme-to-Text Transcription

Multi-video summarization based on Video-MMR