In recent years, deep neural networks have been successful in both industry and academia, especially for computer vision tasks. The great success of deep learning is mainly due to its scalability to encode large-scale data and to maneuver billions of model parameters. However, it is a challenge to deploy these cumbersome deep models on devices with limited resources, e.g., mobile phones and embedded devices, not only because of the high computational complexity but also the large storage requirements. To this end, a variety of model compression and acceleration techniques have been developed. As a representative type of model compression and acceleration, knowledge distillation effectively learns a small student model from a large teacher model. It has received rapid increasing attention from the community. This paper provides a comprehensive survey of knowledge distillation from the perspectives of knowledge categories, training schemes, teacher-student architecture, distillation algorithms, performance comparison and applications. Furthermore, challenges in knowledge distillation are briefly reviewed and comments on future research are discussed and forwarded.

Knowledge Distillation: A Survey

https://hal.archives-ouvertes.fr/hal-00499180/document

Automatic speech recognition and speech variability: A review

Overcoming catastrophic forgetting in neural networks

An automatic text classification system is provided which extracts words and word sequences from a text or texts to be analyzed. The extracted words and word sequences are compared with training data comprising words and word sequences together with a measure of probability with respect to the plurality of qualities. Each plurality of qualities may be represented by an axis whose two end points correspond to mutually exclusive characteristics. Based on the comparison, the texts to be analyzed are then classified in terms of the plurality of qualities. In addition, a fuzzy logic retrieval system and a system for generating the training data are provided.

Automatic text classification system

An automatic speech recognition (ASR) system and method is provided for controlling the recognition of speech utterances generated by an end user operating a communications device. The ASR system and method can be used with a communications device that is used in a communications network. The ASR system can be used for ASR of speech utterances input into a mobile device, to perform compensating techniques using at least one characteristic and for updating an ASR speech recognizer associated with the ASR system by determined and using a background noise value and a distortion value that is based on the features of the mobile device. The ASR system can be used to augment a limited data input capability of a mobile device, for example, caused by limited input devices physically located on the mobile device.

Systems and methods for automatic speech recognition

Constructing deep neural network (DNN) acoustic models from limited training data is an important issue for the development of automatic speech recognition (ASR) applications that will be used in various application-specific acoustic environments. To this end, domain adaptation techniques that train a domain-matched model without overfitting by lever-aging pre-constructed source models are widely used. In this paper, we propose a novel domain adaptation method for DNN acoustic models based on the knowledge distillation framework. Knowledge distillation transfers the knowledge of a teacher model to a student model and offers better generalizability of the student model by controlling the shape of posterior probability distribution of the teacher model, which was originally proposed for model compression. We apply this framework to model adaptation. Our domain adaptation method avoids overfitting of the adapted model trained on limited data by transferring the knowledge of the source model to the adapted model by distillation. Experiments show that the proposed method can effectively avoid the overfitting of convolutional neural network based acoustic models and yield lower error rates than conventional adaptation methods.

Domain adaptation of DNN acoustic models using knowledge distillation

A model adaptation scheme in the pattern recognition, which is capable of realizing a fast, real time model adaptation and improving the recognition performance This model adaptation scheme determines a change in a parameter expressing a condition of pattern recognition and probabilistic model training between an initial condition at a time of acquiring training data used in obtaining a model parameter of each probabilistic model and a current condition at a time of actual recognition Then, the probabilistic models are adapted by obtaining a model parameter after a condition change by updating a model parameter before a condition change according to the determined change, when the initial condition and the current condition are mismatching The adaptation processing uses a Taylor expansion expressing a change in the model parameter in terms of a change in the parameter expressing the condition

Scheme for model adaptation in pattern recognition based on Taylor expansion

Acoustic-to-word speech recognition based on attention-based encoder-decoder models achieves better accuracies with much lower latency than the conventional speech recognition systems. However, acoustic-to-word models require a very large amount of training data and it is difficult to prepare one for a new domain such as elderly speech. To address the problem, we propose domain adaptation based on transfer learning with layer freezing. Layer freezing first pre-trains a network with the source domain data, and then a part of parameters is re-trained for the target domain while the rest is fixed. In the attentionbased acoustic-to-word model, the encoder part is frozen to maintain the generality, and only the decoder part is re-trained to adapt to the target domain. This substantially allows for adaptation of the latent linguistic capability of the decoder to the target domain. Using a large-scale Japanese spontaneous speech corpus as source, the proposed method is applied to three target domains: a call center task and two voice search tasks by adults and by elderly. The models trained with the proposed method achieved better accuracy than the baseline models, which are trained from scratch or entirely re-trained with the target domain.

Encoder Transfer for Attention-based Acoustic-to-word Speech Recognition.

This paper presents a novel deep neural network architecture for transfer learning in acoustic models. A well-known approach for transfer leaning is using target domain data to fine-tune a pre-trained model with source model. The model is trained so as to raise its performance in the target domain. However, this approach may not fully utilize the knowledge of the pre-trained model because the pre-trained knowledge is forgotten when the target domain is updated. To solve this problem, we propose a new architecture based on progressive neural networks (PNN) that can transfer knowledge; it does not forget and can well utilize pre-trained knowledge. In addition, we introduce an enhanced PNN that uses feature augmentation to better leverage pre-trained knowledge. The proposed architecture is challenged in experiments on three different recorded Japanese speech recognition tasks (one source and two target domain tasks). In a comparison with various transfer learning approaches, our proposal achieves the lowest error rate in the target tasks.

Progressive Neural Network-based Knowledge Transfer in Acoustic Models

PROBLEM TO BE SOLVED: To provide a speech segment speaker classification device and the method therefore, a speech recognition device using the same and the method therefore, a program and a recording medium, capable of performing speech recognition for an arbitrary speaker, without registering the speaker beforehand. SOLUTION: The speech segment speaker classification device includes; a sound volume voice period dividing section; a feature value analysis section; a representative feature value extracting section; a segment classification section; and a segment integration section. The sound volume voice period dividing section outputs a speech segment by detecting a voice period of a voice signal which is made discrete value. The feature value analysis section outputs a sound feature value by performing sound feature value analysis of a speech segment. The representative feature value extracting section extracts a representative feature value of the speech segment from the sound feature value. The segment classification section calculates a distance between each representative feature value and classifies the speech segment into clusters on the basis of the distance. The segment integration section integrates adjoining speech segments to one segment, when the adjoining speech segments belong to the same cluster. COPYRIGHT: (C)2010,JPO&INPIT

Speech segment speaker classification device and method therefore, speech recognition device using the same and method therefore, program and recording medium

Yoshikazu Yamaguchi

Papers

Domain adaptation of DNN acoustic models using knowledge distillation

Scheme for model adaptation in pattern recognition based on Taylor expansion

Encoder Transfer for Attention-based Acoustic-to-word Speech Recognition.

Progressive Neural Network-based Knowledge Transfer in Acoustic Models

Speech segment speaker classification device and method therefore, speech recognition device using the same and method therefore, program and recording medium