Multilayer neural networks trained with the back-propagation algorithm constitute the best example of a successful gradient based learning technique. Given an appropriate network architecture, gradient-based learning algorithms can be used to synthesize a complex decision surface that can classify high-dimensional patterns, such as handwritten characters, with minimal preprocessing. This paper reviews various methods applied to handwritten character recognition and compares them on a standard handwritten digit recognition task. Convolutional neural networks, which are specifically designed to deal with the variability of 2D shapes, are shown to outperform all other techniques. Real-life document recognition systems are composed of multiple modules including field extraction, segmentation recognition, and language modeling. A new learning paradigm, called graph transformer networks (GTN), allows such multimodule systems to be trained globally using gradient-based methods so as to minimize an overall performance measure. Two systems for online handwriting recognition are described. Experiments demonstrate the advantage of global training, and the flexibility of graph transformer networks. A graph transformer network for reading a bank cheque is also described. It uses convolutional neural network character recognizers combined with global training techniques to provide record accuracy on business and personal cheques. It is deployed commercially and reads several million cheques per day.

Gradient-based learning applied to document recognition

Most current speech recognition systems use hidden Markov models (HMMs) to deal with the temporal variability of speech and Gaussian mixture models (GMMs) to determine how well each state of each HMM fits a frame or a short window of frames of coefficients that represents the acoustic input. An alternative way to evaluate the fit is to use a feed-forward neural network that takes several frames of coefficients as input and produces posterior probabilities over HMM states as output. Deep neural networks (DNNs) that have many hidden layers and are trained using new methods have been shown to outperform GMMs on a variety of speech recognition benchmarks, sometimes by a large margin. This article provides an overview of this progress and represents the shared views of four research groups that have had recent successes in using DNNs for acoustic modeling in speech recognition.

Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups

Recurrent neural networks can be used to map input sequences to output sequences, such as for recognition, production or prediction problems. However, practical difficulties have been reported in training recurrent neural networks to perform tasks in which the temporal contingencies present in the input/output sequences span long intervals. We show why gradient based learning algorithms face an increasingly difficult problem as the duration of the dependencies to be captured increases. These results expose a trade-off between efficient learning by gradient descent and latching on information for long periods. Based on an understanding of this problem, alternatives to standard gradient descent are considered. >

http://cslt.riit.tsinghua.edu.cn/mediawiki/images/0/03/Learning_Long-Term_Dependencies_with_Gradient_Descent_is_Difficult.pdf

Learning long-term dependencies with gradient descent is difficult

SRILM is a collection of C++ libraries, executable programs, and helper scripts designed to allow both production of and experimentation with statistical language models for speech recognition and other applications. SRILM is freely available for noncommercial purposes. The toolkit supports creation and evaluation of a variety of language model types based on N-gram statistics, as well as several related tasks, such as statistical tagging and manipulation of N-best lists and word lattices. This paper summarizes the functionality of the toolkit and discusses its design and implementation, highlighting ease of rapid prototyping, reusability, and combinability of tools.

SRILM – An Extensible Language Modeling Toolkit

From the Publisher:
This book takes an empirical approach to language processing, based on applying statistical and other machine-learning algorithms to large corpora.Methodology boxes are included in each chapter. Each chapter is built around one or more worked examples to demonstrate the main idea of the chapter. Covers the fundamental algorithms of various fields, whether originally proposed for spoken or written language to demonstrate how the same algorithm can be used for speech recognition and word-sense disambiguation. Emphasis on web and other practical applications. Emphasis on scientific evaluation. Useful as a reference for professionals in any of the areas of speech and language processing.

/pdf/speech-and-language-processing-an-introduction-to-natural-atzaecqfem.pdf

Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition

A knowledge representation formalism for SLU is introduced. It is used for incremental and partially automated annotation of the Media corpus in terms of semantic structures. An automatic interpretation process is described for composing semantic structures from basic semantic constituents using patterns involving constituents and words. The process has procedures for obtaining semantic compositions and for generating frame hypotheses by inference. This process is evaluated on a dialogue corpus manually annotated at the word and semantic constituent levels.

https://hal.archives-ouvertes.fr/hal-01158578/document

Semantic composition process in a speech understanding system

Augmenting standard speech recognition features with energy gravity centres

A search technique incorporating the automatic modeling of lexical variability is introduced for medium or large-vocabulary speaker-independent speech recognition. Current state-of-art systems depend on being able to model the entire language based on acoustic features and the constraints of syntax or inter-word probabilities. These methods often fail in the presence of multiple speakers, new vocabulary, noise, and spontaneous speech phenomena. A new approach for word hypothesization is proposed, based on an acoustic-phonetic unit called the pseudo-syllable segment. An algorithm is described for transforming a sequence of syllables into words. Techniques are suggested for controlling the accuracy of the syllabic hypothesis set, and learning the phonotactics of syllables automatically in a statistical framework.

The use of syllable phonotactics for word hypothesization

Automatic speech understanding and automatic speech recognition extract different kinds of information from the input signal. The result of the former must be evaluated on the basis of the response of the system while the result of the latter is the word sequence which best matches the input signal. In both cases search has to be performed based on scores of interpretation hypotheses.

A scoring method is presented based on stochastic context-free grammars. The method gives optimal upper-bounds for the computation of the "best" derivation trees of a sentence. This method allows language models to be built based on stochastic context-free grammars and their use with an admissible search algorithm that interprets a speech signal with left-to-right or middle-out strategies. Theoretical and computational aspects are discussed.

Computation of upper-bounds for stochastic context-free languages

The paper provides a theoretical justification that gravity centers (GC) in frequency bands computed from zero-crossing information are far more robust to additive telephone noise than GCs computed from FFT spectra. Experiments on two different corpora confirm the theoretical results when GCs are added to standard mel frequency-scaled cepstral coefficients (MFCC) and their time derivatives. A 20.1% word error reduction is observed on a large telephone corpus of Italian cities, with an average signal-to-noise ratio (SNR) of 15 dB, if GCs are computed from zero-crossings, while performance deteriorates when GCs are computed from FFT spectra.

R. De Mori

Papers

Semantic composition process in a speech understanding system

Augmenting standard speech recognition features with energy gravity centres

The use of syllable phonotactics for word hypothesization

Computation of upper-bounds for stochastic context-free languages

Ear-model derived features for automatic speech recognition