Digital Coding of Waveforms

Perceiving Talking Faces: From Speech Perception to a Behavioral Principle

A method of producing synthetic visual speech according to this invention includes receiving an input containing speech information. One or more visemes that correspond to the speech input are then identified. Next, the weights of those visemes are calculated using a coarticulation engine including viseme deformability information. Finally, a synthetic visual speech output is produced based on the visemes' weights over time (or tracks). The synthetic visual speech output is combined with a synchronized audio output corresponding to the input to produce a multimedia output containing a 3D lipsyncing animation.

Methods and devices for producing and using synthetic visual speech based on natural coarticulation

Seeing and hearing others and oneself talk.

This paper presents the main approaches used to synthesize talking faces, and provides greater detail on a handful of these approaches. An attempt is made to distinguish between facial synthesis itself (i.e. the manner in which facial movements are rendered on a computer screen), and the way these movements may be controlled and predicted using phonetic input. The two main synthesis techniques (model-based vs. image-based) are contrasted and presented by a brief description of the most illustrative existing systems. The challenging issues—evaluation, data acquisition and modeling—that may drive future models are also discussed and illustrated by our current work at ICP.

/pdf/audiovisual-speech-synthesis-385yaqunem.pdf

Audiovisual Speech Synthesis

We describe our Finnish audio-visual speech synthesizer, its evaluation and discuss possible improvements. We have combined a three dimensional facial model with a commercial audio text-to-speech synthesizer. The visual speech is based on a letter-to-viseme mapping and the animation is created by linear interpolation between the visemes. An intelligibility test was run to quantify the benefit of seeing the synthetic and natural face on hearing the synthetic and natural voice presented at different signal to noise ratios. Both natural and synthetic faces improved the intelligibility of both natural and synthetic auditory speech. We examined the confusion patterns of consonants and the identification of the Finnish visemes. We also propose how the viseme repertoire of the talking head can be improved.

Audio-visual speech synthesis for Finnish

We describe how our Finnish talking head was improved by using a new auditory speech synthesis method based on neural networks and optimal synchronization of the facial speech animation and the audio signal. In our first version of the talking head, the user typed in text and synthesized auditory speech and synchronized facial animation were created automatically. We combine a 3D facial model with a commercial auditory text-to-speech synthetizer (TTS). The auditory speech is produced by concatenating pre-recorded samples of natural speech according to a set of rules. The quality of the current speech synthesis is not yet adequate. A new strategy has been developed to improve the TTS and to integrate auditory synthesizer synchronization, especially when hardware capabilities are limited. We are developing a new method to achieve an optimal synchronization, independent of the platform used. This method is based on predictive visual synthesis. The new synchronization method gives us better control over audio-visual speech synthesis in the time domain. Using the diphone duration, we can use a more realistic interpolation function between the visemes. Thus, we can also take into account coarticulation effects.

Towards a high quality Finnish talking head

We have constructed an audio-visual text-to-speech synthesizer for Finnish by combining a dynamic facial model with an acoustic speech synthesizer. The visual speech is based on a letter-to-viseme mapping and the animation is created by linear interpolation between the visemes. A viseme is defined by 12 parameter values. In a recent study we showed that visual speech increases the intelligibility of both natural and synthetic auditory speech [5]. We have upgraded our visual speech synthesis by adding the tongue model and improving the speech parameters on the basis of the intelligibility study. Here we show data from a new intelligibility study demonstrating the improved performance of the synthesizer. Presenting the visual speech in three-dimensional space did not further improve the intelligibility.

Parameterized visual speech synthesis and its evaluation

A Finnish-talking head

We have constructed an audio-visual text-to-speech synthesizer for Finnish by combining a facial model with an acoustic speech synthesizer. The quality of the visual speech synthesizer has been evaluated twice. In addition, we have started to refi ne the facial model taking a more physiologically and anatomically based approach. The synthesizer will be used to produce stimuli for studies of neurocognitive mechanisms of audio-visual speech perception. This sets requirements for flexibility and full controllability for the synthesis. We are also developing applications for the synthesizer .

J. Kulju

Papers

Audio-visual speech synthesis for Finnish

Towards a high quality Finnish talking head

Parameterized visual speech synthesis and its evaluation

A Finnish-talking head

Towards a high-quality and well-controlled Finnish audio-visual speech synthesizer.