We present a generative appearance-based method for recognizing human faces under variation in lighting and viewpoint. Our method exploits the fact that the set of images of an object in fixed pose, but under all possible illumination conditions, is a convex cone in the space of images. Using a small number of training images of each face taken with different lighting directions, the shape and albedo of the face can be reconstructed. In turn, this reconstruction serves as a generative model that can be used to render (or synthesize) images of the face under novel poses and illumination conditions. The pose space is then sampled and, for each pose, the corresponding illumination cone is approximated by a low-dimensional linear subspace whose basis vectors are estimated using the generative model. Our recognition algorithm assigns to a test image the identity of the closest approximated illumination cone. Test results show that the method performs almost without error, except on the most extreme lighting directions.

From few to many: illumination cone models for face recognition under variable lighting and pose

https://cs.nju.edu.cn/zhouzh/zhouzh.files/publication/aij02.pdf

Ensembling neural networks: many could be better than all

Visual speech information from the speaker's mouth region has been successfully shown to improve noise robustness of automatic speech recognizers, thus promising to extend their usability in the human computer interface. In this paper, we review the main components of audiovisual automatic speech recognition (ASR) and present novel contributions in two main areas: first, the visual front-end design, based on a cascade of linear image transforms of an appropriate video region of interest, and subsequently, audiovisual speech integration. On the latter topic, we discuss new work on feature and decision fusion combination, the modeling of audiovisual speech asynchrony, and incorporating modality reliability estimates to the bimodal recognition process. We also briefly touch upon the issue of audiovisual adaptation. We apply our algorithms to three multisubject bimodal databases, ranging from small- to large-vocabulary recognition tasks, recorded in both visually controlled and challenging environments. Our experiments demonstrate that the visual modality improves ASR over all conditions and data considered, though less so for visually challenging environments and large vocabulary tasks.

/pdf/recent-advances-in-the-automatic-recognition-of-audiovisual-30c21z0cpc.pdf

Recent advances in the automatic recognition of audiovisual speech

We provide an introduction to theoretical and practical aspects of Boosting and Ensemble learning, providing a useful reference for researchers in the field of Boosting as well as for those seeking to enter this fascinating area of research. We begin with a short background concerning the necessary learning theoretical foundations of weak learners and their linear combinations. We then point out the useful connection between Boosting and the Theory of Optimization, which facilitates the understanding of Boosting and later on enables us to move on to new Boosting algorithms, applicable to a broad spectrum of problems. In order to increase the relevance of the paper to practitioners, we have added remarks, pseudo code, "tricks of the trade", and algorithmic considerations where appropriate. Finally, we illustrate the usefulness of Boosting algorithms by giving an overview of some existing applications. The main ideas are illustrated on the problem of binary classification, although several extensions are discussed.

/pdf/an-introduction-to-boosting-and-leveraging-36mfqregqi.pdf

An introduction to boosting and leveraging

This paper studies the influence of demographics on the performance of face recognition algorithms. The recognition accuracies of six different face recognition algorithms (three commercial, two nontrainable, and one trainable) are computed on a large scale gallery that is partitioned so that each partition consists entirely of specific demographic cohorts. Eight total cohorts are isolated based on gender (male and female), race/ethnicity (Black, White, and Hispanic), and age group (18-30, 30-50, and 50-70 years old). Experimental results demonstrate that both commercial and the nontrainable algorithms consistently have lower matching accuracies on the same cohorts (females, Blacks, and age group 18-30) than the remaining cohorts within their demographic. Additional experiments investigate the impact of the demographic distribution in the training set on the performance of a trainable face recognition algorithm. We show that the matching accuracy for race/ethnicity and age cohorts can be improved by training exclusively on that specific cohort. Operationally, this leads to a scenario, called dynamic face matcher selection, where multiple face recognition algorithms (each trained on different demographic cohorts) are available for a biometric system operator to select based on the demographic information extracted from a probe image. This procedure should lead to improved face recognition accuracy in many intelligence and law enforcement face recognition scenarios. Finally, we show that an alternative to dynamic face matcher selection is to train face recognition algorithms on datasets that are evenly distributed across demographics, as this approach offers consistently high accuracy across all cohorts.

/pdf/face-recognition-performance-role-of-demographic-information-fu9wgvpy8y.pdf

Face Recognition Performance: Role of Demographic Information

We describe a novel neural network architecture, which can recognize human faces with any view in a certain viewing angle range (from left 30 degrees to right 30 degrees out of plane rotation). View-specific eigenface analysis is used as the front-end of the system to extract features, and the neural network ensemble is used for recognition. Experimental results show that the recognition accuracy of our network ensemble is higher than conventional methods such as using a single neural network to recognize faces of a specific view.

/pdf/pose-invariant-face-recognition-1v297b18fq.pdf

Pose invariant face recognition

Describes a real-time face-tracking algorithm. We start with single face tracking based on statistical color modeling and a deformable template. We then expand the algorithm to track multiple faces, possibly with occlusion, by constraining the speed and size changes of the faces. We test the algorithm on sequences with different occlusion patterns and analyze the tracking performance. We also present a tracking software library based on this algorithm. This library can be applied to human-computer interfaces, lip-reading and virtual environments.

Tracking of multiple faces for human-computer interfaces and virtual environments

In this demo, we present a technique for synthesizing the mouth movement from acoustic speech information The algorithm maps the audio parameter set to the visual parameter set using the Gaussian mixture model and the hidden Markov model With this technique, we can create smooth and realistic lip movements

/pdf/real-time-lip-synch-face-animation-driven-by-human-voice-c9kx0xeex6.pdf

Real-time lip-synch face animation driven by human voice

We propose a method for integrating audio and visual information to enhance speech recognition in adverse environments. We train the audio hidden Markov model and the visual hidden Markov model separately, and then use a Viterbi algorithm to decode both channels in parallel. The decoding process is asynchronous between the two channels to capture the asynchronous nature of audio and visual speech. We test the proposed method using speech corrupted by various types of noise and speech with the Lombard effect.

Fu Jie Huang

Papers

Pose invariant face recognition

Tracking of multiple faces for human-computer interfaces and virtual environments

Real-time lip-synch face animation driven by human voice

Consideration of Lombard effect for speechreading