http://lxie.nwpu-aslp.org/papers/2007-PR-A1-EI-SCI-JNL.pdf

A coupled HMM approach to video-realistic speech animation

Natural looking lip animation, synchronized with incoming speech, is essential for realistic character animation. In this work, we evaluate the performance of phone and viseme based acoustic units, with and without context information, for generating realistic lip synchronization using HMM based recognition systems. We conclude via objective evaluations that utilization of viseme based units with context information outperforms the other methods.

Comparison of Phoneme and Viseme Based Acoustic Units for Speech Driven Realistic Lip Animation

https://www.comp.hkbu.edu.hk/~ymc/papers/journal/PR_4419_publication_version.pdf

A local region based approach to lip tracking

The advent of media-sharing sites, especially along with the so called Web 2.0 wave, has led to the unprecedented Internet delivery of community-contributed media contents such as images and videos, which have become the primary sources for online advertising. However, conventional ad-networks such as Google Adwords and AdSense treat image and video advertising as general text advertising by displaying the ads either relevant to the queries or the Web page content, without considering automatically monetizing the rich contents of individual images and videos. In this paper, we summarize the trends of online advertising and propose an innovative advertising model driven by the compelling contents of images and videos. We present recently developed ImageSense and VideoSense as two exemplary applications dedicated to images and videos, respectively, in which the most contextually relevant ads are embedded at the most appropriate positions within the images or videos. The ads are selected based on not only textual relevance but also visual similarity so that the ads yield contextual relevance to both the text in the Web page and the visual content. The ad insertion positions are detected based on visual saliency analysis to minimize the intrusiveness to the user. We also envision that the next trend of multimedia advertising would be game-alike advertising.

When multimedia advertising meets the new Internet era

Comparison of Phoneme and Viseme Based Acoustic Units for Speech Driven Realistic lip Animation

This paper presents a robust hierarchical lip tracking approach (RoHiLTA) for lip-reading and audio visual speech recognition (AVSR) applications. Lip regions of interest are subtly detected by motion and facial structure information. Improvements are made on active shape models (ASMs) for extracting lip contours more accurately and efficiently from video sequences of a speaker's talking face in natural lighting conditions and without particular make-ups. Local and global ASM search algorithms are both improved by introducing color information, 2D mouth corner match, and robust estimation. For noise-free features, localization errors are automatically corrected by an interpolating scheme. A fast implementation of the hierarchical approach is also proposed. Extensive experiments show that the improved ASM can effectively reduce the lip locating errors. The fast implementation of RoHiLTA can consistently achieve superior performance to conventional ASMs in lip tracking tasks, and then can be effectively integrated in lip-reading and AVSR systems.

A robust hierarchical lip tracking approach for lipreading and audio visual speech recognition

This paper presents a novel speech driven accurate realistic visual speech synthesis approach. Firstly, an audio visual instance database is built for different viseme context combinations, i.e. diviseme units, using 100 audio visual speech sentences of a female speaker. Then a diviseme instance selection algorithm is introduced to choose the optimal diviseme instances for the viseme contexts in the input speech, considering both the concatenation smoothness of the image sequences, and matching of the mouth movements to the acoustic pronunciation process, as well the intensity of the input speech. Finally mouth image sequences of corresponding viseme segments in the selected diviseme instances are time warped and blended to construct the mouth images of the final animation. Visual speech synthesis experiments and subjective evaluation results show that mouth animations can be obtained which are not only realistic with clear and smooth mouth images, but also in good accordance with the acoustic pronunciation and intensity of the input speech.

Accurate visual speech synthesis based on diviseme unit selection and concatenation

This paper addresses the problem of animating a talking figure, such as an avatar, using speech input only. The proposed system is based on Hidden Markov Models for the acoustic observation vectors of the speech sounds that correspond to each of 16 visually distinct mouth shapes (called visemes). This case study illustrates that it is indeed possible to obtain visually relevant speech segmentation data directly from a purely acoustic observation sequence.

Acoustic viseme modelling for speech driven automation: a case study

In this paper, we present a viseme (the basic speech units In the visual domain) based continuous speech recognition system, which segments speech into viseme sequences with timing boundaries to drive a talking head. In the viseme Hidden Markov Model (HMM) training, the instances of a viseme with different contexts are formulated as trisemes. Based on the mouth shape parameters Liprounding and the defined viseme similarity weight (VSW) from the 3D viseme facial models, 166 questions concerning the viseme contexts are designed to build triseme decision trees to tie the states of the trisemes with similar contexts, so that they can share the same parameters. To evaluate the system performance, the image related measurements are also taken to evaluate the resulting viseme sequences, with 'jerky instances' in Liprounding and VSW graphs evaluating their smoothness. Results show that compared to the phoneme based system, the tied-state triseme based speech recognition system gives talking head animation with smoother and more plausible mouth shapes.

Triseme decision trees in the continuous speech recognition system for a talking head

This paper presents an audio visual bimodal continuous speech recognition system The visual feature extraction of the mouth movements uses the number of granules obtained by applying a datasieve Multi-stream HMMs are introduced for combining audio and visual modalities using time synchronous audio visual features Experimental results show that the recognition system provided by this paper is suitable for continuous speech recognition tasks in noisy environments, and the datasieve based visual features outperform the conventional DCT and DWT features

Dong-Mei Jiang

Papers

A robust hierarchical lip tracking approach for lipreading and audio visual speech recognition

Accurate visual speech synthesis based on diviseme unit selection and concatenation

Acoustic viseme modelling for speech driven automation: a case study

Triseme decision trees in the continuous speech recognition system for a talking head

A multi-stream bimodal continuous speech recognition system using datasieve based features