With the transition of facial expression recognition (FER) from laboratory-controlled to challenging in-the-wild conditions and the recent success of deep learning techniques in various fields, deep neural networks have increasingly been leveraged to learn discriminative representations for automatic FER. Recent deep FER systems generally focus on two important issues: overfitting caused by a lack of sufficient training data and expression-unrelated variations, such as illumination, head pose and identity bias. In this paper, we provide a comprehensive survey on deep FER, including datasets and algorithms that provide insights into these intrinsic problems. First, we describe the standard pipeline of a deep FER system with the related background knowledge and suggestions of applicable implementations for each stage. We then introduce the available datasets that are widely used in the literature and provide accepted data selection and evaluation principles for these datasets. For the state of the art in deep FER, we review existing novel deep neural networks and related training strategies that are designed for FER based on both static images and dynamic image sequences, and discuss their advantages and limitations. Competitive performances on widely used benchmarks are also summarized in this section. We then extend our survey to additional related issues and application scenarios. Finally, we review the remaining challenges and corresponding opportunities in this field as well as future directions for the design of robust deep FER systems.

Deep Facial Expression Recognition: A Survey

Automatic machine-based Facial Expression Analysis (FEA) has made substantial progress in the past few decades driven by its importance for applications in psychology, security, health, entertainment, and human–computer interaction. The vast majority of completed FEA studies are based on nonoccluded faces collected in a controlled laboratory environment. Automatic expression recognition tolerant to partial occlusion remains less understood, particularly in real-world scenarios. In recent years, efforts investigating techniques to handle partial occlusion for FEA have seen an increase. The context is right for a comprehensive perspective of these developments and the state of the art from this perspective. This survey provides such a comprehensive review of recent advances in dataset creation, algorithm development, and investigations of the effects of occlusion critical for robust performance in FEA systems. It outlines existing challenges in overcoming partial occlusion and discusses possible opportunities in advancing the technology. To the best of our knowledge, it is the first FEA survey dedicated to occlusion and aimed at promoting better-informed and benchmarked future work.

Facial Expression Analysis under Partial Occlusion: A Survey

Automatic machine-based Facial Expression Analysis (FEA) has made substantial progress in the past few decades driven by its importance for applications in psychology, security, health, entertainment and human computer interaction. The vast majority of completed FEA studies are based on non-occluded faces collected in a controlled laboratory environment. Automatic expression recognition tolerant to partial occlusion remains less understood, particularly in real-world scenarios. In recent years, efforts investigating techniques to handle partial occlusion for FEA have seen an increase. The context is right for a comprehensive perspective of these developments and the state of the art from this perspective. This survey provides such a comprehensive review of recent advances in dataset creation, algorithm development, and investigations of the effects of occlusion critical for robust performance in FEA systems. It outlines existing challenges in overcoming partial occlusion and discusses possible opportunities in advancing the technology. To the best of our knowledge, it is the first FEA survey dedicated to occlusion and aimed at promoting better informed and benchmarked future work.

With the transition of facial expression recognition (FER) from laboratory-controlled to challenging in-the-wild conditions and the recent success of deep learning techniques in various fields, deep neural networks have increasingly been leveraged to learn discriminative representations for automatic FER. Recent deep FER systems generally focus on two important issues: overfitting caused by a lack of sufficient training data and expression-unrelated variations, such as illumination, head pose, and identity bias. In this survey, we provide a comprehensive review of deep FER, including datasets and algorithms that provide insights into these intrinsic problems. First, we introduce the available datasets that are widely used in the literature and provide accepted data selection and evaluation principles for these datasets. We then describe the standard pipeline of a deep FER system with the related background knowledge and suggestions for applicable implementations for each stage. For the state-of-the-art in deep FER, we introduce existing novel deep neural networks and related training strategies that are designed for FER based on both static images and dynamic image sequences and discuss their advantages and limitations. Competitive performances and experimental comparisons on widely used benchmarks are also summarized. We then extend our survey to additional related issues and application scenarios. Finally, we review the remaining challenges and corresponding opportunities in this field as well as future directions for the design of robust deep FER systems. 

/pdf/deep-facial-expression-recognition-a-survey-2dbgyaku.pdf

Convolutional neural network (CNN) is a very effective method to recognize facial emotions. However, the preprocessing and selection of parameters of these methods heavily depend on the human experience and require a large amount of trial-and-errors. This paper presents an ensemble of convolutional neural networks method with probability-based fusion for facial expression recognition, where the architecture of CNN was adapted by using the convolutional rectified linear layer as the first layer and multiple hidden maxout layers. It was constructed by randomly varying parameters and architecture around the optimal values for CNN, where each CNN as the base classifier was trained to output a probability for each class. These probabilities were then fused through the probability-based fusion method. The conducted experiments on benchmark data sets validated our method, which had better accuracy than the compared methods. The proposed method was novel and efficient for facial expression recognition.

Ensemble of Deep Neural Networks with Probability-Based Fusion for Facial Expression Recognition

According to the complex manifestation of human facial expression in realistic environment, occlusion problem has become a new challenge and a hot spot in the field of expression recognition. To make facial expression recognition applied in broader way, the main work is to increase the accuracy under different partial occlusion with feasible robust, which is limited by the information missing and insufficient training with fewer samples. Therefore, an algorithm with a deep structure has been proposed in this paper dealing with four types of frequently occurred occlusion. As a classic method, the Gabor filter is used for feature extraction at first. Then, multi-layers network is used to pre-train the training data samples, with re-describing the input Gabor features in complex way and fine-tuning the weights to refine the learning model. The experimental results on JAFFE database show that the proposed method is valid to achieve better recognition rate especially for partial occlusion on eyes and mouth.

A Deep Structure for Facial Expression Recognition under Partial Occlusion

A major challenge in pattern recognition is labeling of large numbers of sam- ples. This problem has been solved by extending supervised learning to semi-supervised learning. Thus semi-supervised learning has become one of the most important methods on the research of facial expression recognition. Frontal and un-occluded face images have been well recognized using traditional facial expression recognition based on semi- supervised learning. However, pose-variants caused by body movement, may decrease facial expression recognition rate. A novel facial expression recognition algorithm based on semi-supervised learning is proposed to improve the robustness in multi-pose facial expression recognition. In the proposed method, transfer learning has been brought into semi-supervised learning to solve the problem of multi-pose facial expression recognition. Experiments show that our method is competent for semi-supervised facial expression recognition on the condition of multi-pose. The recognition rates are 82.68% and 87.71% on the RaFD database and BHU database, respectively.

Semi-supervised Facial Expression Recognition Algorithm on The Condition of Multi-pose

Robust facial expression recognition under facial occlusion condition is the main research orientation, which has important research significance Many problems are caused by facial occlusion, not only missing facial expression information, but also bringing outliers or lots of noise Aiming at the point, firstly, the face to be recognized is reconstructed using robust principal component analysis (RPCA); secondly, Eigenfaces and Fisherfaces are used to extract facial expression features respectively; finally, nearest neighbor method and support vector machine are used as classifiers Facial expression recognition experiments are implemented in different occlusion conditions on Japanese female facial expression database (JAFFE) On the condition of big occlusion and small sample, RPCA algorithms gained better recognition results than many other methods, showing that this method based on RPCA is robust to kinds of facial occlusions

https://link.springer.com/content/pdf/10.1007/978-3-642-23620-4_13.pdf

Research of robust facial expression recognition under facial occlusion condition

Under the condition of multi-databases, a novel algorithm of facial expression recognition was proposed to improve the robustness of traditional semi-supervised methods dealing with individual differences in facial expression recognition. First, the regions of interest of facial expression images were determined by face detection and facial expression features were extracted using Linear Discriminant Analysis. Then Transfer Learning Adaptive Boosting (TrAdaBoost) algorithm was improved as semi-supervised learning method for multi-classification. The results show that the proposed method has stronger robustness than the traditional methods, and improves the facial expression recognition rate from multiple databases.

Research on the Algorithm of Semi-supervised Robust Facial Expression Recognition

Facial Expression Recognition has become a hot research direction in human computer interaction, machine learning and image processing. However, the inaccuracy of facial expression similarity measure is always a key problem in facial expression classification. In order to solve this problem, a novel algorithm which is based on Discriminative Component Analysis algorithm, is proposed in this paper. Compared with Discriminative Component Analysis, our algorithm is based on maximizing within-class distance and minimizing between-class distance to choose chunklets, which guarantee the stability and accuracy of algorithm. The mean recognition rates are 80.42% and 95.71% under the condition of using hold-out method and leave-one-out method respectively. Experimental results show the effectiveness of our algorithm.

Bin Jiang

Papers

A Deep Structure for Facial Expression Recognition under Partial Occlusion

Semi-supervised Facial Expression Recognition Algorithm on The Condition of Multi-pose

Research of robust facial expression recognition under facial occlusion condition

Research on the Algorithm of Semi-supervised Robust Facial Expression Recognition

A Novel Algorithm of Facial Expression Recognition Based on Discriminative Component Analysis