The Journal of the Acoustical Society of America

https://hal.archives-ouvertes.fr/hal-00499180/document

Automatic speech recognition and speech variability: A review

We present a machine learning technique for driving 3D facial animation by audio input in real time and with low latency. Our deep neural network learns a mapping from input waveforms to the 3D vertex coordinates of a face model, and simultaneously discovers a compact, latent code that disambiguates the variations in facial expression that cannot be explained by the audio alone. During inference, the latent code can be used as an intuitive control for the emotional state of the face puppet. We train our network with 3--5 minutes of high-quality animation data obtained using traditional, vision-based performance capture methods. Even though our primary goal is to model the speaking style of a single actor, our model yields reasonable results even when driven with audio from other speakers with different gender, accent, or language, as we demonstrate with a user study. The results are applicable to in-game dialogue, low-cost localization, virtual reality avatars, and telepresence.

Audio-driven facial animation by joint end-to-end learning of pose and emotion

The present capabilities of speech recognition algorithms will be surveyed. The application of IC technology to the implementation of these algorithms will be explored and potential future directions will be determined.

Speech recognition

Dynamic Bayesian networks (DBNs) are a powerful and flexible methodology for representing and computing with probabilistic models of stochastic processes. In the past decade, there has been increasing interest in applying them to practical problems, and this thesis shows that they can be used effectively in the field of automatic speech recognition. 
A principle characteristic of dynamic Bayesian networks is that they can model an arbitrary set of variables as they evolve over time. Moreover, an arbitrary set of conditional independence assumptions can be specified, and this allows the joint distribution to be represented in a highly factored way. Factorization allows for models with relatively few parameters, and computational efficiency. Standardized inference and learning routines allow a variety of model structures to be tested without deriving new formulae, or writing new code. 
The contribution of this thesis is to show how DBNs can be used in automatic speech recognition. This involves solving problems related to both representation and inference. Representationally, the thesis shows how to encode stochastic finite-state word models as DBNs, and how to construct DBNs that explicitly model the speech-articulators, accent, gender, speaking-rate, and other important phenomena. Technically, the thesis presents inference routines that are especially tailored to the requirements of speech recognition: efficient inference with deterministic constraints, variable-length utterances, and online inference. Finally, the thesis presents experimental results that indicate that real systems can be built, and that modeling important phenomena with DBNs results in higher recognition accuracy.

Speech recognition with Dynamic Bayesian Networks

Phonetic alignment: speech synthesis-based vs. viterbi-based

In this paper, we evaluate multi-Gaussian HMM systems and hybrid HMM/ANN systems in the framework of task independent training for small size (75 words) and medium size (600 words) vocabularies. To do this, we use the Phonebook database (Pitrelli et al., 1995) which is particularly well suited to this kind of experiment since (1) it is a very large telephone database and (2) the size and content of the test vocabulary is very flexible. For each system, different HMM topologies are compared to test the influence of state tying (with a number of parameters approximately kept constant) on the recognition performance. Two lexica (Phonebook and CMU) are also compared and it is shown that the CMU lexicon leads to significantly better performance. Finally, it is shown that with a quite simple system and a few adaptations to the basic HMM/ANN scheme, recognition performance of 98.5% and 94.7% can easily be achieved, respectively on a lexicon of 75 and 600 words (isolated words, telephone speech and lexicon words not present in the training data).

/pdf/hybrid-hmm-ann-systems-for-training-independent-tasks-1maaf5qarq.pdf

Hybrid HMM/ANN systems for training independent tasks: experiments on Phonebook and related improvements

Cultural expression is not limited to architecture, monuments or collections of artifacts. It also includes fragile intangible live expressions, which involve knowledge and skills such as music, dance, singing, theatre, human skills and craftsmanship. These manifestations of human intelligence and creativeness constitute our Intangible Cultural Heritage (ICH), a basic factor of local cultural identity and a guaranty for sustainable development. In this paper, we briefly introduce the i-Treasures research project, which aims at developing an open and extendable platform to provide access to ICH resources, enable knowledge exchange and contribute to the transmission of rare know-how. The project goes beyond digitization of cultural content; it creates new knowledge that has never been analysed or studied before through novel methodologies for the analysis and modelling of ICH based on multisensory technology. High-level semantics are extracted enabling researchers to identify possible implicit or hidden correlations between different ICH expressions or interpretation styles and study the evolution of a specific ICH. Four different ICH cases are studied: traditional songs, dance interactions, pottery and contemporary music composition Combining conventional learning procedures and sensorimotor learning through an interactive 3D environment, i-Treasures breaks new ground in education and knowledge transfer of ICH.

Capturing the intangible an introduction to the i-Treasures project

In this paper we compare two different methods for phonetically labeling a speech database. The first approach is based on the alignment of the speech signal on a high quality synthetic speech pattern, and the second one uses a hybrid HMM/ANN system. Both systems have been evaluated on French read utterances from a speaker never seen in the training stage of the HMM/ANN system and manually segmented. This study outlines the advantages and drawbacks of both methods. The high quality speech synthetic system has the great advantage that no training stage is needed, while the classical HMM/ANN system easily allows multiple phonetic transcriptions. We deduce a method for the automatic constitution of phonetically labeled speech databases based on using the synthetic speech segmentation tool to bootstrap the training process of our hybrid HMM/ANN system. The importance of such segmentation tools will be a key point for the development of improved speech synthesis and recognition systems.

Olivier Deroo

Papers

Automatic speech recognition and speech variability: A review

Phonetic alignment: speech synthesis-based vs. viterbi-based

Hybrid HMM/ANN systems for training independent tasks: experiments on Phonebook and related improvements

Capturing the intangible an introduction to the i-Treasures project

Phonetic alignment: speech synthesis based vs. hybrid HMM/ANN.