Facial expression recognition with Convolutional Neural Networks

COVID-19 identification in chest X-ray images on flat and hierarchical classification scenarios.

Facial expression recognition (FER) is increasingly gaining importance in various emerging affective computing applications. In practice, achieving accurate FER is challenging due to the large amount of inter-personal variations such as expression intensity variations. In this paper, we propose a new spatio-temporal feature representation learning for FER that is robust to expression intensity variations. The proposed method utilizes representative expression-states (e.g., onset, apex and offset of expressions) which can be specified in facial sequences regardless of the expression intensity. The characteristics of facial expressions are encoded in two parts in this paper. As the first part, spatial image characteristics of the representative expression-state frames are learned via a convolutional neural network. Five objective terms are proposed to improve the expression class separability of the spatial feature representation. In the second part, temporal characteristics of the spatial feature representation in the first part are learned with a long short-term memory of the facial expression. Comprehensive experiments have been conducted on a deliberate expression dataset (MMI) and a spontaneous micro-expression dataset (CASME II). Experimental results showed that the proposed method achieved higher recognition rates in both datasets compared to the state-of-the-art methods.

Multi-Objective Based Spatio-Temporal Feature Representation Learning Robust to Expression Intensity Variations for Facial Expression Recognition

Local line directional pattern for palmprint recognition

Facial expression recognition (FER) is a significant task for the machines to understand the emotional changes in human beings. However, accurate hand-crafted features that are highly related to changes in expression are difficult to extract because of the influences of individual difference and variations in emotional intensity. Therefore, features that can accurately describe the changes in facial expressions are urgently required. Method: A weighted mixture deep neural network (WMDNN) is proposed to automatically extract the features that are effective for FER tasks. Several pre-processing approaches, such as face detection, rotation rectification, and data augmentation, are implemented to restrict the regions for FER. Two channels of facial images, including facial grayscale images and their corresponding local binary pattern (LBP) facial images, are processed by WMDNN. Expression-related features of facial grayscale images are extracted by fine-tuning a partial VGG16 network, the parameters of which are initialized using VGG16 model trained on ImageNet database. Features of LBP facial images are extracted by a shallow convolutional neural network (CNN) built based on DeepID. The outputs of both channels are fused in a weighted manner. The result of final recognition is calculated using softmax classification. Results: Experimental results indicate that the proposed algorithm can recognize six basic facial expressions (happiness, sadness, anger, disgust, fear, and surprise) with high accuracy. The average recognition accuracies for benchmarking data sets “CK+,” “JAFFE,” and “Oulu-CASIA” are 0.970, 0.922, and 0.923, respectively. Conclusions: The proposed FER method outperforms the state-of-the-art FER methods based on the hand-crafted features or deep networks using one channel. Compared with the deep networks that use multiple channels, our proposed network can achieve comparable performance with easier procedures. Fine-tuning is effective to FER tasks with a well pre-trained model if sufficient samples cannot be collected.

Facial Expression Recognition Using Weighted Mixture Deep Neural Network Based on Double-Channel Facial Images

This paper proposes a novel local feature descriptor, local directional number pattern (LDN), for face analysis, i.e., face and expression recognition. LDN encodes the directional information of the face's textures (i.e., the texture's structure) in a compact way, producing a more discriminative code than current methods. We compute the structure of each micro-pattern with the aid of a compass mask that extracts directional information, and we encode such information using the prominent direction indices (directional numbers) and sign-which allows us to distinguish among similar structural patterns that have different intensity transitions. We divide the face into several regions, and extract the distribution of the LDN features from them. Then, we concatenate these features into a feature vector, and we use it as a face descriptor. We perform several experiments in which our descriptor performs consistently under illumination, noise, expression, and time lapse variations. Moreover, we test our descriptor with different masks to analyze its performance in different face analysis tasks.

Local Directional Number Pattern for Face Analysis: Face and Expression Recognition

Local Directional Texture Pattern image descriptor

Deriving an effective facial feature from original face images is a vital step for a successful automatic facial expression recognition In this paper, we proposed a new feature descriptor, Local Principal Texture Pattern (LPTP), for expression recognition We compute the LPTP feature, at each pixel, by extracting the principal directions of the local neighborhood, and we code the intensity differences on these directions The mixture of direction and contrast information makes our descriptor robust against rotation and illumination changes Consequently, we represent each expression as a distribution of LPTP codes Our experiments demonstrate the superiority of the proposed feature, on two facial expression databases, over the existing methods

Recognition of face expressions using Local Principal Texture Pattern

In this paper, we propose a novel local feature descriptor, Local Sign Directional Pattern (LSDP), for face expression recognition. LSDP encodes the directional information of the face's textures-i.e., the texture's structure-in a compact way, producing a more discriminating code than other state-of-the-art methods. The structure of each micro-pattern is encoded by using its prominent directions and sign-which allows it to distinguish among similar structural patterns that have different intensity transitions. We divide the face into several regions, from which we extract the distributions of the LSDP features. These features are concatenated into a feature vector, and used as a face descriptor, and the expression recognition is obtained with the aid of Support Vector Machine classifiers.

Jorge Rojas Castillo

Papers

Local Directional Number Pattern for Face Analysis: Face and Expression Recognition

Local Directional Texture Pattern image descriptor

Recognition of face expressions using Local Principal Texture Pattern

Facial expression recognition based on Local Sign Directional Pattern