Facial expression recognition with temporal modeling of shapes

TLDR: This work proposes a framework for automatic facial expression recognition from continuous video sequence by modeling temporal variations within shapes using Latent-Dynamic Conditional Random Fields, and shows that the proposed approach outperforms CRFs for recognizing facial expressions.

Abstract: Conditional Random Fields (CRFs) can be used as a discriminative approach for simultaneous sequence segmentation and frame labeling. Latent-Dynamic Conditional Random Fields (LDCRFs) incorporates hidden state variables within CRFs which model sub-structure motion patterns and dynamics between labels. Motivated by the success of LDCRFs in gesture recognition, we propose a framework for automatic facial expression recognition from continuous video sequence by modeling temporal variations within shapes using LDCRFs. We show that the proposed approach outperforms CRFs for recognizing facial expressions. Using Principal Component Analysis (PCA) we study the separability of various expression classes in lower dimension projected spaces. By comparing the performance of CRFs and LDCRFs against that of Support Vector Machines (SVMs), we demonstrate that temporal variations within shapes are crucial in classifying expressions especially for those with a small range of facial motion like anger and sadness. We also show empirically that only using changes in facial appearance over time, without using shape variations, is not sufficient to obtain high performance for facial expression recognition.

Figures

Table 4.1: Overview of the dataset.

Table 4.5: Recognition Rates with Neutral expression using Shape Features (7-class classification). [Neutral(Ne), Anger(An), Disgust(Dn), Fear(Fe), Happiness (Ha), Sadness(Sa), Surprise(Su), Average(Avg)]

Figure 3.7: Linear Chain Conditional Random Fields. Here each Xi is the observation vector for each frame and each Yi is the class label for each frame in the sequence. A linear chain structure means that a first-order Markov assumption is made for class labels.

Table 4.4: Confusion Matrix (percentage) and Precision-Recall Statistics for 6-class classification using proposed method based on LDCRFs. [Anger(An), Disgust(Dn), Fear(Fe), Happiness (Ha), Sadness(Sa), Surprise(Su)]

Table 4.3: Confusion Matrix (percentage) and Precision-Recall Statistics for 6- class classification using CRFs. [Anger(An), Disgust(Dn), Fear(Fe), Happiness (Ha), Sadness(Sa), Surprise(Su)]

Table 4.6: Confusion Matrix (percentage) and Precision-Recall Statistics for 7-class classification using CRFs. [Neutral(Ne), Anger(An), Disgust(Dn), Fear(Fe), Happiness (Ha), Sadness(Sa), Surprise(Su)]

Full text

Copyright
by
Suyog Dutt Jain
2011

The Thesis committee for Suyog Dutt Jain
Certifies that this is the approved version of the following thesis
Facial Expression Recognition with Temporal Modeling
of Shapes
APPROVED BY
SUPERVISING COMMITTEE:
J. K. Aggarwal, Supervisor
Kristen Grauman

Facial Expression Recognition with Temporal Modeling
of Shapes
by
Suyog Dutt Jain, B.E.
THESIS
Presented to the Faculty of the Graduate School of
The University of Texas at Austin
in Partial Fulfillment
of the Requirements
for the Degree of
Master Of Science in Computer Science
THE UNIVERSITY OF TEXAS AT AUSTIN
August 2011

Dedicated to my family

Acknowledgments
This work has been possible because of the following special people who
have always been a source of inspiration for me to do research and pursue my
dreams.
I offer my greatest regards to Professor J. K. Aggarwal for providing
me the opportunity to work with him. His ideas have formed the foundations
of this work and his continuous motivation, guidance and vision enabled me
to explore the problem in right perspective and find solutions.
I also deeply thank Professor Kristen Grauman for introducing me to
the world of computer vision research. Her teaching and discussions helped
me in building a strong background which is invaluable for my research.
I would like to thank Dr. Changbo Hu, Birgi Tamersoy, Jong Taek Lee
and all other Computer & Vision Research Center members who made my
stay here a memorable one. I will always cherish the numerous philosophical
discussions on evolution and existence which were a part of most lunches we
had together.
I owe my wonderful time in Austin as a graduate student to my friends
Yashesh, Nikhil, Akash, Deepak, Sameer, Pooja, Akanksha, Gouri, Lakshmi
and Anushree who were always there with their support, love and encourage-
ment.
v

Frequently asked questions

Q1. What have the authors contributed in "Facial expression recognition with temporal modeling of shapes" ?

In this paper, the authors proposed a new approach for facial expression recognition by modeling temporal dynamics of face shapes. 

Q2. What have the authors stated for future works in "Facial expression recognition with temporal modeling of shapes" ?

There are several open possibilities for enhancing their current 51 work. The authors wish to go beyond that and see if they can extend the current work to handle real world issues like pose and illumination variations, recognizing expressions from continuous video streams like web-cams etc. The authors also want to analyze 3D face shapes and see if temporal modeling of 3D data can give us better results for recognizing facial expressions. 

Q3. What is the purpose of the optimization procedure?

The optimization procedure also involves a regularization term which is decided using cross validation with values ranging from 10−3 to 103 during training. 

Q4. What is the problem of supervised sequence labeling?

The problem of supervised sequence labeling requires us to learn a clas-sifier from training data consisting of a set of labeled sequences. 

Q5. What is the way to model facial expressions?

Their proposed approach using LDCRFs is also more robust in modeling facial expressions as compared to CRFs which shows that capturing subtle facial motion is very essential in differentiating between facial expressions. 

Q6. How many regions are used to classify facial expressions?

It has been shown[40] that using a single histogram for the entire imageis not a good technique for facial expression recognition, hence the cropped face image is subdivided into 42 regions using a 6 x 7 grid (see Figure 3.5). 

Q7. What is the way to perform the localization of landmark points?

Kanaujia et al. [24] use active shape models with localized Non-negative Matrix Factorization in order to perform the localization of landmark points.