Spontaneous facial micro-expression analysis using Spatiotemporal Completed Local Quantized Patterns

Recently, there are increasing interests in inferring mirco-expression from facial image sequences. For micro-expression recognition, feature extraction is an important critical issue. In this paper, we proposes a novel framework based on a new spatiotemporal facial representation to analyze micro-expressions with subtle facial movement. Firstly, an integral projection method based on difference images is utilized for obtaining horizontal and vertical projection, which can preserve the shape attributes of facial images and increase the discrimination for micro-expressions. Furthermore, we employ the local binary pattern operators to extract the appearance and motion features on horizontal and vertical projections. Intensive experiments are conducted on three available published micro-expression databases for evaluating the performance of the method. Experimental results demonstrate that the new spatiotemporal descriptor can achieve promising performance in micro-expression recognition.

/pdf/facial-micro-expression-recognition-using-spatiotemporal-4evfugjd62.pdf

Facial Micro-Expression Recognition Using Spatiotemporal Local Binary Pattern with Integral Projection

Recently, there have been increasing interests in inferring mirco-expression from facial image sequences. Due to subtle facial movement of micro-expressions, feature extraction has become an important and critical issue for spontaneous facial micro-expression recognition. Recent works used spatiotemporal local binary pattern (STLBP) for micro-expression recognition and considered dynamic texture information to represent face images. However, they miss the shape attribute of face images. On the other hand, they extract the spatiotemporal features from the global face regions while ignore the discriminative information between two micro-expression classes. The above-mentioned problems seriously limit the application of STLBP to micro-expression recognition. In this paper, we propose a discriminative spatiotemporal local binary pattern based on an integral projection to resolve the problems of STLBP for micro-expression recognition. First, we revisit an integral projection for preserving the shape attribute of micro-expressions by using robust principal component analysis. Furthermore, a revisited integral projection is incorporated with local binary pattern across spatial and temporal domains. Specifically, we extract the novel spatiotemporal features incorporating shape attributes into spatiotemporal texture features. For increasing the discrimination of micro-expressions, we propose a new feature selection based on Laplacian method to extract the discriminative information for facial micro-expression recognition. Intensive experiments are conducted on three availably published micro-expression databases including CASME, CASME2 and SMIC databases. We compare our method with the state-of-the-art algorithms. Experimental results demonstrate that our proposed method achieves promising performance for micro-expression recognition.

/pdf/discriminative-spatiotemporal-local-binary-pattern-with-4wasw2ysnu.pdf

Discriminative Spatiotemporal Local Binary Pattern with Revisited Integral Projection for Spontaneous Facial Micro-Expression Recognition

Differential optical flow applied to automatic facial expression recognition

Most traditional facial expression-recognition systems track facial components such as eyes, eyebrows, and mouth for feature extraction. Though some of these features can provide clues for expression recognition, other finer changes of the facial muscles can also be deployed for classifying various facial expressions. This study locates facial components by active shape model to extract seven dynamic face regions (frown, nose wrinkle, two nasolabial folds, two eyebrows, and mouth). Proposed semantic facial features could then be acquired using directional gradient operators like Gabor filters and Laplacian of Gaussian. A multi-class support vector machine (SVM) was trained to classify six facial expressions (neutral, happiness, surprise, anger, disgust, and fear). The popular Cohn---Kanade database was tested and the average recognition rate reached 94.7 %. Also, 20 persons were invited for on-line test and the recognition rate was about 93 % in a real-world environment. It demonstrated that the proposed semantic facial features could effectively represent changes between facial expressions. The time complexity could be lower than the other SVM based approaches due to the less number of deployed features.

Effective semantic features for facial expressions recognition using SVM

This paper presents a vision-based multimodal human computer interface system using eye and hand motion tracking. Conventional vision-based human computer interface use eye or hand motion tracking individually. However, the proposed vision-based virtual interface is integrating the function of the motion tracking of eye blinking and hand gesture with the function of their recognition as a virtual interface. The proposed virtual multimodal interface system provides vision-based mechanism to communicate between human and computer system rather than using conventional human computer interface. For motion tracking and recognition of eye and hand gesture we exploite optical flow method and template matching. In order to minimize the error to detected and track the specific human features caused by light variation, the enhancement of each frame is performed by histogram equalization and max-min normalization. For eye and hand region detection we use HT skin color which is nonparametric model and robust to light variation. While tracking the positions of hand and eye using optical flow method, predefined hand gesture and eye blinking are recognized by template matching. In the experiment, we apply the developed interface to control the motion of 3D models developed in Open GL environments. From experiments we can show that the proposed interface can effectively subsitute the role of existing interface device such as a mouse.

Vision-Based Multimodal Human Computer Interface Based on Parallel Tracking of Eye and Hand Motion

Vision-based human computer interaction is an emerging field of science and industry to provide natural way to communicate with computer. In that sense, one of the skills is to infer the emotional state of the person based on the facial expression recognition. In this paper, we present a novel approach to recognize facial expression from a sequence of input images using HMM (Hidden Markov Model) and facial motion tracking based on optical flow. Conventionally, in the HMM which consists of seven basic emotional states, it is considered natural that transitions between emotions are imposed to pass through neutral state. However, in this work we propose an enhanced transition framework model which consists of transitions between each emotional state without passing through neutral state in addition to a traditional transition model. For the localization of facial features from video sequence we exploit template matching and optical flow. The facial feature displacements traced by the optical flow are used for input parameters to HMM for facial expression recognition. From the experiment, we can prove that the proposed framework can effectively recognize the facial expression in real time.

Spatio-temporal Facial Expression Recognition Using Optical Flow and HMM

Vision-based Human computer interaction is an emerging field of science and industry to provide natural way to communicate with human and computer. In that sense, inferring the emotional state of the person based on the facial expression recognition is an important issue. In this paper, we present a novel approach to recognize facial expression from a sequence of input images using emotional specific HMM (Hidden Markov Model) and facial motion tracking based on optical flow. Conventionally, in the HMM which consists of basic emotional states, it is considered natural that transitions between emotions are imposed to pass through neutral state. However, in this work we propose an enhanced transition framework model which consists of transitions between each emotional state without passing through neutral state in addition to a traditional transition model. For the localization of facial features from video sequence we exploit template matching and optical flow. The facial feature displacements traced by the optical flow are used for input parameters to HMM for facial expression recognition. From the experiment, we can prove that the proposed framework can effectively recognize the facial expression in real time.

http://www.koreascience.or.kr:80/article/JAKO200932056740862.pdf

Gihan Shin

Papers

Vision-Based Multimodal Human Computer Interface Based on Parallel Tracking of Eye and Hand Motion

Spatio-temporal Facial Expression Recognition Using Optical Flow and HMM

Realtime Facial Expression Recognition from Video Sequences Using Optical Flow and Expression HMM