Predicting articulatory movements from audio or text has diverse applications, such as speech visualization. Various approaches have been proposed to solve the acoustic-articulatory mapping problem. However, their precision is not high enough with only acoustic features available. Recently, deep neural network (DNN) has brought tremendous success in various fields, like speech recognition and image processing. To increase the accuracy, we propose a new network architecture for articulatory movement prediction with both text and audio inputs, called a bottleneck long-term recurrent convolutional neural network (BLTRCNN). To the best of our knowledge, it is the first time to predict articulatory movements based on DNN by fusing text and audio inputs. Our BLTRCNN consists of two networks. The first is the bottleneck network, generating a compact bottleneck features of text information for each frame independently. The second, including convolutional neural network, long short-term memory and skip connection, is called the long-term recurrent convolutional neural network (LTRCNN). LTRCNN is used for articulatory movement prediction when bottleneck features, acoustic features, and text features are integrated as inputs together. Experiments show that the proposed BLTRCNN achieves the state-of-the-art root-mean-square error (RMSE) 0.528 mm and the correlation coefficient 0.961. Moreover, we also demonstrate how text information complements acoustic features in this prediction task.

BLTRCNN-Based 3-D Articulatory Movement Prediction: Learning Articulatory Synchronicity From Both Text and Audio Inputs

This paper proposes a text-driven 3D visual pronunciation system of Mandarin Chinese. Firstly, an articulatory speech corpus is collected through Electro-Magnetic Articulography, and then articulatory model and acoustic model are trained by multi-stream Hidden Semi-Markov Model together; secondly, based on an accurate anatomical 3D articulatory mesh model constructed by Magnetic Resonance Imaging, the Hidden Semi-Markov Model synthesized result is combined with articulatory anatomy to produce realistic articulatory animation synchronized with speech. The experimental results show the effectiveness of the system for instructing language learners to articulate.

3D visual pronunciation of Mandarine Chinese for language learning

Simulation of the tongue has important applications in biomechanics, medical science, linguistics, and graphics. The accuracy of the geometry, intrinsic structure and dynamic simulation of tongue are crucial for these applications. In this paper, we build a 3D anatomically and biomechanically accurate tongue model. For ensuring anatomical accuracy, the tongue mesh model is constructed based on accurate medical data and an interactive muscle marking method for specifying the muscle geometry and fiber arrangement. For ensuring biomechanical accuracy, a nonlinear, quasi-incompressible, isotropic, hyperelastic constitutive model is applied for describing the tongue tissues. Particularly, tongue muscles are additionally endowed with an anisotropic constitutive model, which reflects the active and passive mechanical behavior of muscle fibers. The dynamic simulation results of tongue movements subjected to certain muscle activations are presented and validated with experimental data, indicating the suitability for visual speech synthesis.

Creating and simulating a realistic physiological tongue model for speech production

The 3D talking head has been developed fast, in which both external and internal articulators were demonstrated. For Mandarin pronunciation, the aspiration airflow is crucial to discriminate confusable Mandarin consonants. In this paper, we present a 3D talking head system for articulatory and aspiration animation with the use of EMA articulation data and airflow data simultaneously. The quantitative analyses of airflow data indicated confusable Mandarin consonants could be distinguished from each other by the mean airflow during voicing, peak expiratory airflow, and airflow duration. An airflow model was then incorporated into the 3D articulatory model to produce the airflow in accordance with articulator movements of Mandarin pronunciation. An audio-visual test was designed to evaluate the current 3D articulation and aspiration system, where minimal pairs were used to recognize the animation. The identification accuracy was significantly improved from 43.9% without airflow to 84.8% with airflow-incorporated information.

Intelligible enhancement of 3D articulation animation by incorporating airflow information

汉语国际教育用音节汉字词汇等级划分. 国家标准・应用解读本 = The graded Chinese syllables, characters and words for the application of teaching Chinese to the speakers of other languages. National standard : application and interpretation

In the framework of intelligent aided language learning, a real-time data-driven 3D visual pronunciation system of Chinese IPA is proposed. First, a high quality articulatory speech corpus including speech and 3D articulatory data of lips, tongue and jaw movements is collected through Electro-Magnetic Articulograph; second, the 3D articulatory modeling including shape design and motion synthesis is conducted. The articulatory shape is obtained by designing a precise 3D facial model including internal and external articulators. The articulatory motion synthesis is obtained combining parameterized model and anatomical model. The system can thus illustrate the slight differences among phonemes by synthesizing both internal and external articulatory movements. The perceptual evaluation shows the suitability of the system for instructing language learners to articulate.

Qiang Fang

Papers

Data-driven 3D visual pronunciation of Chinese IPA for language learning