A morphable model for the synthesis of 3D faces

This paper presents a new learning-based approach to speech synthesis that achieves mouth movements with rich and expressive articulation for novel audio input. From a database of 3D triphone motions, our algorithm picks the optimal sequences based on a triphone similarity measure, and concatenates them to create new utterances that include coarticulation effects. By using a Locally Linear Embedding (LLE) representation of feature points on 3D scans, we propose a model that defines a measure of similarity among visemes, and a system of viseme categories, which are used to define triphone substitution rules and a cost function. Moreover, we compute deformation vectors for several facial expressions, allowing expression variation to be smoothly added to the speech animation. In an entirely data-driven approach, our automated procedure for defining viseme categories closely reproduces the groups of related visemes that are defined in the phonetics literature. The structure of our selection method is intrinsic to the nature of speech and generates a substitution table that can be reused as-is in different speech animation systems.

/pdf/a-nonlinear-viseme-model-for-triphone-based-speech-synthesis-5ja2w8qav6.pdf

A nonlinear viseme model for triphone-based speech synthesis

Lot of deaf hearing impaired people is using lips reading as a main communication form. They prefer Virtual speaking head in crisis situation, as is fire, flood, bombs, different evacuation... A viseme is a representational unit used to classify speech sounds in the visual domain. A “viseme” describes the particular facial and oral positions and movements that occur alongside the voicing of phonemes. Design tool for creating correct speech visemes is designed. Its composed from 5 modules; module for creating phonemes, module for creating 3D speech visemes, module for facial expression and module for synchronization between phonemes and visemes and last one module to generate speech triphones. We are testing correctness of visemes on Slovak speech domains. Up to now, have not been developed the regular Slovak speech visemes. The paper describe developed tool. Slovak speech visemes and Virtual speaking head [16] as a helpful for hearing impaired people in crisis situations to orientate him.

Virtual Speaking Head for Hearing Impaired People in Crisis Time

Usefulness of new one teaching technologies and presentation technologies, as well as a variety of audiovisual communications software technologies is almost unlimited. Currently we are working in creating the final education tool for students, including the students with delimited ability as are also hearing impaired students The tool is building on developing the virtual Slovak speaking head and in using the virtual head in new education technologies. The paper describe root of the tool and functionality all tools modules and usefulness the final education tool.

New 3D Education Technology for Students with Delimited Ability

Audio-visual speech recognition and visual speech synthesisers are used as interfaces between humans and machines. Such interactions specifically rely on the analysis and synthesis of both audio and visual information, which humans use for face-to-face communication. Currently, there is no global standard to describe these interactions nor is there a standard mathematical tool to describe lip movements. Furthermore, the visual lip movement for each phoneme is considered in isolation rather than a continuation from one to another. Consequently, there is no globally accepted standard method for representing lip movement during articulation. This thesis addresses these issues by designing a transcribed group of words, by mathematical formulas, and so introducing the concept of a visual word, allocating signatures to visual words and finally building a visual speech vocabulary database. In addition, visual speech information has been analysed in a novel way by considering both lip movements and phonemic structure of the English language. In order to extract the visual data, three visual features on the lip have been chosen; these are on the outer upper, lower and corner of the lip. The extracted visual data during articulation is called the visual speech sample set. The final visual data is obtained after processing the visual speech sample sets to correct experimented artefacts such as head tilting, which happened during articulation and visual data extraction. The ‘Barycentric Lagrange Interpolation’ (BLI) formulates the visual speech sample sets into visual speech signals. The visual word is defined in this work and consists of the variation of three visual features. Further processing on relating the visual speech signals to the uttered word leads to the allocation of signatures that represent the visual word. This work suggests the visual word signature can be used either as a ‘visual word barcode’, a ‘digital visual word’ or a ‘2D/3D representations’. The 2D version of the visual word provides a unique signature that allows the identification of the words being uttered. In addition, identification of visual words has also been performed using a technique called ‘volumetric representations of the visual words’. Furthermore, the effect of altering the amplitudes and sampling rate for BLI has been evaluated. In addition, the performance of BLI in reconstructing the visual speech sample sets has been considered. Finally, BLI has been compared to signal reconstruction approach by RMSE and correlation coefficients. The results show that the BLI is the more reliable method for the purpose of this work according to Section 7.7.

New method for mathematical modelling of human visual speech

In this paper, we present a new approach that generates synthetic mouth articulations from an audio file and that transfers them to different face meshes. It is based on learning articulations from a stream of 3D scans of a real person acquired by a structured light scanner at 40 three-dimensional frames per second. Correspondence between these scans over several speech sequences is established via optical flow. We propose a novel type of Principal Component Analysis that considers variances only in a sub-region of the face, while retaining the full dimensionality of the original vector space of sample scans. Audio is recorded at the same time, so the head scans can be synchronized with phoneme and viseme information for computing viseme clusters. Given a new audio sequence along with text data, we are able to quickly create in a fully automated fashion an animation synchronized with that new sentence by morphing between the visemes along a path in viseme-space. The methods described in the paper include an automated process for data analysis in streams of 3D scans, and a framework that connects the system to existing static face modeling technology for articulation transfer.

Robert Bargmann

Papers

A nonlinear viseme model for triphone-based speech synthesis

Learning-Based Facial Rearticulation Using Streams of 3D Scans