Facial expression recognition in videos is an active area of research in computer vision. However, fake facial expressions are difficult to be recognized even by humans. On the other hand, facial micro-expressions generally represent the actual emotion of a person, as it is a spontaneous reaction expressed through human face. Despite of a few attempts made for recognizing micro-expressions, still the problem is far from being a solved problem, which is depicted by the poor rate of accuracy shown by the state-of-the-art methods. A few CNN based approaches are found in the literature to recognize micro-facial expressions from still images. Whereas, a spontaneous microexpression video contains multiple frames that have to be processed together to encode both spatial and temporal information. This paper proposes two 3D-CNN methods: MicroExpSTCNN and MicroExpFuseNet, for spontaneous facial micro-expression recognition by exploiting the spatiotemporal information in CNN framework. The MicroExpSTCNN considers the full spatial information, whereas the MicroExpFuseNet is based on the 3D-CNN feature fusion of the eyes and mouth regions. The experiments are performed over CAS(ME)2 and SMIC microb expression databases. The proposed MicroExpSTCNN model outperforms the state-of-the-art methods.

Spontaneous Facial Micro-Expression Recognition using 3D Spatiotemporal Convolutional Neural Networks

Facial expression recognition in the wild, by fusion of deep learnt and hand-crafted features

Facial expression recognition in videos is an active area of research in computer vision. However, fake facial expressions are difficult to be recognized even by humans. On the other hand, facial micro-expressions generally represent the actual emotion of a person, as it is a spontaneous reaction expressed through human face. Despite of a few attempts made for recognizing micro-expressions, still the problem is far from being a solved problem, which is depicted by the poor rate of accuracy shown by the state-of-the-art methods. A few CNN based approaches are found in the literature to recognize micro-facial expressions from still images. Whereas, a spontaneous micro-expression video contains multiple frames that have to be processed together to encode both spatial and temporal information. This paper proposes two 3D-CNN methods: MicroExpSTCNN and MicroExpFuseNet, for spontaneous facial micro-expression recognition by exploiting the spatiotemporal information in CNN framework. The MicroExpSTCNN considers the full spatial information, whereas the MicroExpFuseNet is based on the 3D-CNN feature fusion of the eyes and mouth regions. The experiments are performed over CAS(ME)^2 and SMIC micro-expression databases. The proposed MicroExpSTCNN model outperforms the state-of-the-art methods.

Spontaneous Facial Micro-Expression Recognition using 3D Spatiotemporal Convolutional Neural Networks.

In recent years, many researchers are taking an interest in the research area of Face recognition due to its diverse applications such as security systems, medical systems, entertainment. Nowadays, many kinds of biometric information processing systems are used for various purpose face-recognition systems. The facial expression that can define the human mental state and behavior and it is used for security purpose. FER is used in domains such as healthcare, marketing, environment, safety and social media. This paper presents the survey of the facial expression recognition system that includes the four main stages i.e. face detection, pre-processing, extraction of features, and classification. This research provides a broad overview of the FER process includes all stages of FER system as well as the various methods used to evaluate the efficiency of the various methods of facial expression recognition. This survey paper also helps to understand the approaches, different techniques that address and analyse the problems and challenges comes in the real-time environment. Finally, this paper concludes the state-of-the-art and explore the challenges faced in implementation of FER process along with the scope of future development.

A Survey on Facial Expression Recognition using Machine Learning Techniques

Expressions play an imperative part in human interactions as it let humans express their intentions and feelings without words. Mental state of a subject can be mined from the expression extracted. In recent times, deep neural network has outperformed traditional handcrafted descriptors including spatiotemporal local binary pattern (LBP), LBP-TOP, HOG, etc., when it comes to feature extraction for facial expression recognition (FER). In this paper, an amalgam of convolution neural network (CNN) and long short-term memory (LSTM) [recurrent neural network (RNN)] is employed to extract essential features for recognizing the expression from the target frame. To increase the performance, transfer learning concept is engaged to get learned parameters (weight/bias). To accomplish transfer learning, leading layers of ResNet-50 (trained on thousands of image frames) are used. Further, a LSTM layer (time distributed) is affixed to the existing model. The model is further trained (CK+ database) with different activation functions, and a relative analysis is performed. Maximum accuracy of 94% is attained with the hybrid model (CNN-LSTM with SELU and ELU).

CNN-LSTM-Based Facial Expression Recognition

This paper introduces two novel motion based features for recognizing human facial expressions. The proposed motion features are applied for recognizing facial expressions from a video sequence. The proposed bag-of-words based scheme represents each frame of a video sequence as a vector depicting local motion patterns during a facial expression. The local motion patterns are captured by an efficient derivation from optical flow. Motion features are clustered and stored as words in a dictionary. We further generate a reduced dictionary by ranking the words based on some ambiguity measure. We prune out the ambiguous words and continue with key words in the reduced dictionary. The ambiguity measure is given by applying a graph-based technique, where each word is represented as a node in the graph. Ambiguity measures are obtained by modelling the frequency of occurrence of the word during the expression. We form expression descriptors for each expression from the reduced dictionary, by applying an efficient kernel. The training of the expression descriptors are made following an adaptive learning technique. We tested the proposed approach with standard dataset. The proposed approach shows better accuracy compared to the state-of-the-art.

K. V. Sai Vineeth Kumar Reddy

Papers

Recognizing facial expressions using novel motion based features