We propose a novel context-dependent (CD) model for large-vocabulary speech recognition (LVSR) that leverages recent advances in using deep belief networks for phone recognition. We describe a pre-trained deep neural network hidden Markov model (DNN-HMM) hybrid architecture that trains the DNN to produce a distribution over senones (tied triphone states) as its output. The deep belief network pre-training algorithm is a robust and often helpful way to initialize deep neural networks generatively that can aid in optimization and reduce generalization error. We illustrate the key components of our model, describe the procedure for applying CD-DNN-HMMs to LVSR, and analyze the effects of various modeling choices on performance. Experiments on a challenging business search dataset demonstrate that CD-DNN-HMMs can significantly outperform the conventional context-dependent Gaussian mixture model (GMM)-HMMs, with an absolute sentence accuracy improvement of 5.8% and 9.2% (or relative error reduction of 16.0% and 23.2%) over the CD-GMM-HMMs trained using the minimum phone error rate (MPE) and maximum-likelihood (ML) criteria, respectively.

/pdf/context-dependent-pre-trained-deep-neural-networks-for-large-14o17jhkzd.pdf

Context-Dependent Pre-Trained Deep Neural Networks for Large-Vocabulary Speech Recognition

Recurrent neural networks are powerful sequence learners. They are able to incorporate context information in a flexible way, and are robust to localised distortions of the input data. These properties make them well suited to sequence labelling, where input sequences are transcribed with streams of labels. The aim of this thesis is to advance the state-of-the-art in supervised sequence labelling with recurrent networks. Its two main contributions are (1) a new type of output layer that allows recurrent networks to be trained directly for sequence labelling tasks where the alignment between the inputs and the labels is unknown, and (2) an extension of the long short-term memory network architecture to multidimensional data, such as images and video sequences.

https://mediatum.ub.tum.de/doc/673554/file.pdf

Supervised Sequence Labelling with Recurrent Neural Networks

We show that an end-to-end deep learning approach can be used to recognize either English or Mandarin Chinese speech-two vastly different languages. Because it replaces entire pipelines of hand-engineered components with neural networks, end-to-end learning allows us to handle a diverse variety of speech including noisy environments, accents and different languages. Key to our approach is our application of HPC techniques, enabling experiments that previously took weeks to now run in days. This allows us to iterate more quickly to identify superior architectures and algorithms. As a result, in several cases, our system is competitive with the transcription of human workers when benchmarked on standard datasets. Finally, using a technique called Batch Dispatch with GPUs in the data center, we show that our system can be inexpensively deployed in an online setting, delivering low latency when serving users at scale.

/pdf/end-to-end-speech-recognition-in-english-and-mandarin-3d91x54v7w.pdf

End to end speech recognition in English and Mandarin

We present a state-of-the-art speech recognition system developed using end-to-end deep learning. Our architecture is significantly simpler than traditional speech systems, which rely on laboriously engineered processing pipelines; these traditional systems also tend to perform poorly when used in noisy environments. In contrast, our system does not need hand-designed components to model background noise, reverberation, or speaker variation, but instead directly learns a function that is robust to such effects. We do not need a phoneme dictionary, nor even the concept of a "phoneme." Key to our approach is a well-optimized RNN training system that uses multiple GPUs, as well as a set of novel data synthesis techniques that allow us to efficiently obtain a large amount of varied data for training. Our system, called Deep Speech, outperforms previously published results on the widely studied Switchboard Hub5'00, achieving 16.0% error on the full test set. Deep Speech also handles challenging noisy environments better than widely used, state-of-the-art commercial speech systems.

Deep Speech: Scaling up end-to-end speech recognition

/pdf/deep-speech-2-end-to-end-speech-recognition-in-english-and-ey5zvxui4p.pdf

Deep speech 2: end-to-end speech recognition in English and mandarin

The authors are concerned with integrating connectionist networks into a hidden Markov model (HMM) speech recognition system. This is achieved through a statistical interpretation of connectionist networks as probability estimators. They review the basis of HMM speech recognition and point out the possible benefits of incorporating connectionist networks. Issues necessary to the construction of a connectionist HMM recognition system are discussed, including choice of connectionist probability estimator. They describe the performance of such a system using a multilayer perceptron probability estimator evaluated on the speaker-independent DARPA Resource Management database. In conclusion, they show that a connectionist component improves a state-of-the-art HMM system. >

/pdf/connectionist-probability-estimators-in-hmm-speech-1qc9ca14gn.pdf

Connectionist probability estimators in HMM speech recognition

A linguistic model of words uses a state transition diagram (10). The diagram connects states (12, 14, 16, 18, 20) with arcs (22, 24, 26, 28, 30, 32) that represent phoneme transitions within a word. Different pronunciations of the same word may be modelled using alternate transitions (26 or 28 and 30 or 32) from single states (16 and 18, respectively). This effectively models alternate pronunciations using similar phonemes such as substituting a ''D'' (28) phoneme for the ''T'' (26) phoneme in the word ''WATER'' (10). Such models could be extended for longer words or linked together for multiple words.

Method for recognizing speech using linguistically-motivated hidden markov models

SRI is developing a system that uses real time speech recognition to diag nose, evaluate and provide training in spoken English. The paper first describes the methods and results of a study of the feasibility of automati cally grading the performance of Japanese students when reading English aloud. Utterances recorded from Japanese speakers were independently rated by expert listeners. Speech grading software was developed from a speaker independent hidden-Markov-model speech recognition system. The auto matic grading procedure first aligned the speech with a model and then com pared the segments of the speech signal with models of those segments that have been developed from a database of speech from native speakers of English. The evaluation study showed that ratings of speech quality by experts are very reliable and automatic grades correlate well (r > 0.8) with those expert ratings. SRI is now extending this technology and integrating it in a spoken-language training system. This effort involves (1) porting SRI's DECIPHER speech recognition system to a microcomputer platform, and (2) extending the speech-evaluation software to more exactly diagnose a learner's pronuncia tion deficits and lead the learner through an appropriate regimen of exer cises.

Automatic evaluation and training in English pronunciation.

In a speaker-independent, large-vocabulary continuous speech recognition systems, recognition accuracy varies considerably from speaker to speaker, and performance may be significantly degraded for outlier speakers such as nonnative talkers. In this paper, we explore supervised speaker adaptation and normalization in the MLP component of a hybrid hidden Markov model/ multilayer perceptron version of SRI's DECIPHERTM speech recognition system. Normalization is implemented through an additional transformation network that preprocesses the cepstral input to the MLP. Adaptation is accomplished through incremental retraining of the MLP weights on adaptation data. Our approach combines both adaptation and normalization in a single, consistent manner, works with limited adaptation data, and is text-independent. We show significant improvement in recognition accuracy.

Connectionist speaker normalization and adaptation.

A speaker-independent, continuous-speech, large-vocabulary speech recognition system, DECIPHER, has been developed. It provides state-of-the-art performance on the DARPA standard speaker-independent resource management training and testing materials. The approach is to integrate speech and linguistic knowledge into the HMM (hidden Markov model) framework. Performance improvements arising from detailed phonological modeling and from the incorporation of cross-word coarticulatory constraints are described. It is concluded that speech and linguistic knowledge sources can be used to improve the performance of HMM-based speech recognition systems provided that care is taken to incorporate these knowledge sources appropriately. >

Michael Cohen

Papers

Connectionist probability estimators in HMM speech recognition

Method for recognizing speech using linguistically-motivated hidden markov models

Automatic evaluation and training in English pronunciation.

Connectionist speaker normalization and adaptation.

Linguistic constraints in hidden Markov model based speech recognition