We present a random forest-based framework for real time head pose estimation from depth images and extend it to localize a set of facial features in 3D. Our algorithm takes a voting approach, where each patch extracted from the depth image can directly cast a vote for the head pose or each of the facial features. Our system proves capable of handling large rotations, partial occlusions, and the noisy depth data acquired using commercial sensors. Moreover, the algorithm works on each frame independently and achieves real time performance without resorting to parallel computations on a GPU. We present extensive experiments on publicly available, challenging datasets and present a new annotated head pose database recorded using a Microsoft Kinect.

/pdf/random-forests-for-real-time-3d-face-analysis-1jgwf3ss8e.pdf

Random Forests for Real Time 3D Face Analysis

Facial expressions are an important way through which humans interact socially. Building a system capable of automatically recognizing facial expressions from images and video has been an intense field of study in recent years. Interpreting such expressions remains challenging and much research is needed about the way they relate to human affect. This paper presents a general overview of automatic RGB, 3D, thermal and multimodal facial expression analysis. We define a new taxonomy for the field, encompassing all steps from face detection to facial expression recognition, and describe and classify the state of the art methods accordingly. We also present the important datasets and the bench-marking of most influential methods. We conclude with a general discussion about trends, important questions and future lines of research.

Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications

Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-related Applications

We propose a novel geometric framework for analyzing 3D faces, with the specific goals of comparing, matching, and averaging their shapes. Here we represent facial surfaces by radial curves emanating from the nose tips and use elastic shape analysis of these curves to develop a Riemannian framework for analyzing shapes of full facial surfaces. This representation, along with the elastic Riemannian metric, seems natural for measuring facial deformations and is robust to challenges such as large facial expressions (especially those with open mouths), large pose variations, missing parts, and partial occlusions due to glasses, hair, and so on. This framework is shown to be promising from both-empirical and theoretical-perspectives. In terms of the empirical evaluation, our results match or improve upon the state-of-the-art methods on three prominent databases: FRGCv2, GavabDB, and Bosphorus, each posing a different type of challenge. From a theoretical perspective, this framework allows for formal statistical inferences, such as the estimation of missing facial parts using PCA on tangent spaces and computing average shapes.

/pdf/3d-face-recognition-under-expressions-occlusions-and-pose-26qgp6f3vp.pdf

3D Face Recognition under Expressions, Occlusions, and Pose Variations

Human pose estimation has been actively studied for decades. While traditional approaches rely on 2d data like images or videos, the development of Time-of-Flight cameras and other depth sensors created new opportunities to advance the field. We give an overview of recent approaches that perform human motion analysis which includes depth-based and skeleton-based activity recognition, head pose estimation, facial feature detection, facial performance capture, hand pose estimation and hand gesture recognition. While the focus is on approaches using depth data, we also discuss traditional image based methods to provide a broad overview of recent developments in these areas.

A Survey on Human Motion Analysis from Depth Data

This paper presents a novel automatic framework to perform 3D face recognition. The proposed method uses a simulated annealing-based approach (SA) for range image registration with the surface interpenetration measure (SIM), as similarity measure, in order to match two face images. The authentication score is obtained by combining the SIM values corresponding to the matching of four different face regions: circular and elliptical areas around the nose, forehead, and the entire face region. Then, a modified SA approach is proposed taking advantage of invariant face regions to better handle facial expressions. Comprehensive experiments were performed on the FRGC v2 database, the largest available database of 3D face images composed of 4,007 images with different facial expressions. The experiments simulated both verification and identification systems and the results compared to those reported by state-of-the-art works. By using all of the images in the database, a verification rate of 96.5 percent was achieved at a false acceptance rate (FAR) of 0.1 percent. In the identification scenario, a rank-one accuracy of 98.4 percent was achieved. To the best of our knowledge, this is the highest rank-one score ever achieved for the FRGC v2 database when compared to results published in the literature.

3D Face Recognition Using Simulated Annealing and the Surface Interpenetration Measure

We present a methodology for face segmentation and facial landmark detection in range images. Our goal was to develop an automatic process to be embedded in a face recognition system using only depth information as input. To this end, our segmentation approach combines edge detection, region clustering, and shape analysis to extract the face region, and our landmark detection approach combines surface curvature information and depth relief curves to find the nose and eye landmarks. The experiments were performed using the two available versions of the Face Recognition Grand Challenge database and the BU-3DFE database, in order to validate our proposed methodology and its advantages for 3-D face recognition purposes. We present an analysis regarding the accuracy of our segmentation and landmark detection approaches. Our results were better compared to state-of-the-art works published in the literature. We also performed an evaluation regarding the influence of the segmentation process in our 3-D face recognition system and analyzed the improvements obtained when applying landmark-based techniques to deal with facial expressions.

Automatic Face Segmentation and Facial Landmark Detection in Range Images

This paper presents our methodology for face and facial features detection to improve 3D face recognition in a presence of facial expression variation. Our goal was to develop an automatic process to be embedded in a face recognition system, using only range images as input. To do that, our approach combines traditional image segmentation techniques for face segmentation and detect facial features by combining an adapted method for 2D facial features extraction with the surface curvature information. The experiments were performed in a large, well-known face image database available on the Biometric Experimentation Environment (BEE), including 4,950 images. The results confirms that our method is efficient for the proposed application.

Automatic 3D facial segmentation and landmark detection

This paper focuses a comparative evaluation of our framework for 3D face recognition and state-of-theart systems. Our method uses a Simulated Annealing-based approach (SA) for range image registration with the surface interpenetration measure (SIM) as the similarity measure, in order to match two face images. The authentication score is obtained by combining the SIM values corresponding to the matching of four different face regions. Experiments were performed on the FRGC v2 database simulating both verification and identification systems and the obtained results were compared to those reported in the literature. By using all the images in the database, a verification rate of 95.9% was achieved, at a False Acceptance Rate (FAR) of 0.1%. In the identification scenario, a rank-one accuracy of 99.5% was obtained. To our knowledge, this is the best rank-one score obtained on the FRGC v2 database, as compared to previously published results.

3D face recognition using the Surface Interpenetration Measure: A comparative evaluation on the FRGC database

The surface interpenetration measure (SIM) was recently proposed as a promising measure for 3D face matching, although using two limited, small range image databases. In this paper we present novel, more extensive experiments using the SIM in a well-known 3D face database available on the biometric experimentation environment (BEE) to confirm qualitatively that the SIM is a effective, discriminatory measure. The experiments were performed based on range image registration by using two different methods: iterative closest point (ICP) and simulated annealing (SA). By computing the SIM after the registration of two 3D face images one can identify if those images come from the same subject or not. With our SA-based approach we obtained high verification rate scores, which is indeed one of the main goals of the Face Recognition Grand Challenge 2006.

Chauã C. Queirolo

Papers

3D Face Recognition Using Simulated Annealing and the Surface Interpenetration Measure

Automatic Face Segmentation and Facial Landmark Detection in Range Images

Automatic 3D facial segmentation and landmark detection

3D face recognition using the Surface Interpenetration Measure: A comparative evaluation on the FRGC database

3D Face Image Registration for Face Matching Guided by the Surface Interpenetration Measure