Several variants of the long short-term memory (LSTM) architecture for recurrent neural networks have been proposed since its inception in 1995. In recent years, these networks have become the state-of-the-art models for a variety of machine learning problems. This has led to a renewed interest in understanding the role and utility of various computational components of typical LSTM variants. In this paper, we present the first large-scale analysis of eight LSTM variants on three representative tasks: speech recognition, handwriting recognition, and polyphonic music modeling. The hyperparameters of all LSTM variants for each task were optimized separately using random search, and their importance was assessed using the powerful functional ANalysis Of VAriance framework. In total, we summarize the results of 5400 experimental runs ( $\approx 15$ years of CPU time), which makes our study the largest of its kind on LSTM networks. Our results show that none of the variants can improve upon the standard LSTM architecture significantly, and demonstrate the forget gate and the output activation function to be its most critical components. We further observe that the studied hyperparameters are virtually independent and derive guidelines for their efficient adjustment.

LSTM: A Search Space Odyssey

This paper introduces WaveNet, a deep neural network for generating raw audio
waveforms. The model is fully probabilistic and autoregressive, with the
predictive distribution for each audio sample conditioned on all previous ones;
nonetheless we show that it can be efficiently trained on data with tens of
thousands of samples per second of audio. When applied to text-to-speech, it
yields state-of-the-art performance, with human listeners rating it as
significantly more natural sounding than the best parametric and concatenative
systems for both English and Mandarin. A single WaveNet can capture the
characteristics of many different speakers with equal fidelity, and can switch
between them by conditioning on the speaker identity. When trained to model
music, we find that it generates novel and often highly realistic musical
fragments. We also show that it can be employed as a discriminative model,
returning promising results for phoneme recognition.

WaveNet: A Generative Model for Raw Audio

We propose a novel context-dependent (CD) model for large-vocabulary speech recognition (LVSR) that leverages recent advances in using deep belief networks for phone recognition. We describe a pre-trained deep neural network hidden Markov model (DNN-HMM) hybrid architecture that trains the DNN to produce a distribution over senones (tied triphone states) as its output. The deep belief network pre-training algorithm is a robust and often helpful way to initialize deep neural networks generatively that can aid in optimization and reduce generalization error. We illustrate the key components of our model, describe the procedure for applying CD-DNN-HMMs to LVSR, and analyze the effects of various modeling choices on performance. Experiments on a challenging business search dataset demonstrate that CD-DNN-HMMs can significantly outperform the conventional context-dependent Gaussian mixture model (GMM)-HMMs, with an absolute sentence accuracy improvement of 5.8% and 9.2% (or relative error reduction of 16.0% and 23.2%) over the CD-GMM-HMMs trained using the minimum phone error rate (MPE) and maximum-likelihood (ML) criteria, respectively.

/pdf/context-dependent-pre-trained-deep-neural-networks-for-large-14o17jhkzd.pdf

Context-Dependent Pre-Trained Deep Neural Networks for Large-Vocabulary Speech Recognition

In this paper, we present bidirectional Long Short Term Memory (LSTM) networks, and a modified, full gradient version of the LSTM learning algorithm. We evaluate Bidirectional LSTM (BLSTM) and several other network architectures on the benchmark task of framewise phoneme classification, using the TIMIT database. Our main findings are that bidirectional networks outperform unidirectional ones, and Long Short Term Memory (LSTM) is much faster and also more accurate than both standard Recurrent Neural Nets (RNNs) and time-windowed Multilayer Perceptrons (MLPs). Our results support the view that contextual information is crucial to speech processing, and suggest that BLSTM is an effective architecture with which to exploit it'.

Framewise phoneme classification with bidirectional LSTM and other neural network architectures

Automated analysis of human affective behavior has attracted increasing attention from researchers in psychology, computer science, linguistics, neuroscience, and related disciplines. However, the existing methods typically handle only deliberately displayed and exaggerated expressions of prototypical emotions despite the fact that deliberate behaviour differs in visual appearance, audio profile, and timing from spontaneously occurring behaviour. To address this problem, efforts to develop algorithms that can process naturally occurring human affective behaviour have recently emerged. Moreover, an increasing number of efforts are reported toward multimodal fusion for human affect analysis including audiovisual fusion, linguistic and paralinguistic fusion, and multi-cue visual fusion based on facial expressions, head movements, and body gestures. This paper introduces and surveys these recent advances. We first discuss human emotion perception from a psychological perspective. Next we examine available approaches to solving the problem of machine understanding of human affective behavior, and discuss important issues like the collection and availability of training and test data. We finally outline some of the scientific and engineering challenges to advancing human affect sensing technology.

A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions

Since 1984, the European Commission has supported projects related to spoken language processing within various programs (particularly ESPRIT, ESPRIT-BRA and TELEMATICSLANGUAGE ENGINEERING) The projects address various aspects of spoken language processing including basic research, technology development, infrastructure (language resources and evaluation) and applications in different areas (telecommunications, language training, office systems, games, cars, information access and retrieval, banking, aids to the handicapped) The next program, the 5th Framework Program is now being prepared, and should unite the activities in the area of Human Language Technology and its components in a single sector The three sub-areas which have been identified are “Multilinguality”, “Natural Interactivity” and “Active Content” Several on-going projects are addressing the design of conversational systems Three example applications are: the ESPRIT Mask project which aims to develop a Multimodal-Multimedia Automated Service Kiosk providing access to rail travel information; the LE Arise project which aims to provide similar information via the telephone for the Dutch, French, and Italian railways; and the LE Vodis project whose goal is to develop a vocal interface to enhance the usability and functionality of in-car driver assistance and information services

An Overview of EU Programs Related to Conversational/Interactive Systems

This paper addresses recent progress in speaker-independent, large vocabulary, continuous speech recognition, which has opened up a wide range of near and mid-term applications. One rapidly expanding application area is the processing of broadcast audio for information access. At LIMSI, broadcast news transcription systems have been developed for English, French, German, Mandarin and Portuguese, and systems for other languages are under development. Audio indexation must take into account the specificities of audio data, such as needing to deal with the continuous data stream and an imperfect word transcription. Some near-term applications areas are audio data mining, selective dissemination of information and media monitoring.

Invited Talk: Processing Broadcast Audio for Information Access

In this work the random forest language modeling approach is applied with the aim of improving the performance of the LIMSI, highly competitive, Mandarin Chinese speech-to-text system. The experimental setup is that of the GALE Phase 4 evaluation. This setup is characterized by a large amount of available language model training data (over 3.2 billion segmented words). A conventional unpruned 4-gram language model with a vocabulary of 56K words serves as a baseline that is challenging to improve upon. However moderate perplexity and CER improvements over this model were obtained with a random forest language model. Different random forest training strategies were explored so as to attain the maximal gain in performance and Forest of Random Forest language modeling scheme is introduced.

Large-scale language modeling with random forests for mandarin Chinese speech-to-text

The 2004 20xRT BBN/LIMSI English Conversational Telephone Speech System

Le luxembourgeois est une langue germano-franconique et l’une des langues europeennes sous-decrites. Cet article etudie la similitude entre les segments phonetiques en luxembourgeois avec leurs equivalents en allemand, francais et anglais via des techniques d’alignement forces. En utilisant les modeles acoustiques monolingues d’amorcage de ces trois langues, ainsi que des modeles "multilingues" entraines sur un corpus de parole obtenu par concatenation, nous avons examine si le luxembourgeois etait mieux represente par l’une des langues prises individuellement ou par le modele multilingue. Au niveau global, les modeles allemands fournissent la meilleure correspondance, mais une analyse par segments montre des preferences specifiques. Les premiers resultats en transcriptions illustrent les performances des differents jeux de modeles acoustiques monolingues et multilingues, ainsi que les modeles luxembourgeois construits a partir de 1200 heures de parole non transcrites en luxembourgeois, et des methodes non supervisees.

Lori Lamel

Papers

An Overview of EU Programs Related to Conversational/Interactive Systems

Invited Talk: Processing Broadcast Audio for Information Access

Large-scale language modeling with random forests for mandarin Chinese speech-to-text

The 2004 20xRT BBN/LIMSI English Conversational Telephone Speech System

Modélisation acoustico-phonétique de langues peu dotées : Études phonétiques et travaux de reconnaissance automatique en luxembourgois