Emotion recognition plays a significant role in cognitive psychology research. However, measuring emotions is a challenging task. Thus, several approaches have been designed for facial expression recognition (FER). Although, the challenges increases further as the data transits from laboratory-controlled environment to <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">in-the-wild</i> circumstances. Nowadays, applications are overwhelmed by a profusion of deep learning (DL) techniques in the real-world problems. DL networks have steadily led to a better understanding of low-dimensional discriminative features from high-dimensional complex face patterns for automatic FER. The modern FER systems based on deep neural networks mainly suffer from two problems: overfitting due to the inadequate availability of training data and complications unassociated with the expressions, such as occlusion, posture, illumination, and identity bias. This study aims to provide a comprehensive survey of the significant DL-based methods that have made a notable contribution to the field of FER. Different components of the methods, such as pre-processing, feature extraction, and classification of facial expressions, are described systematically. Moreover, the discussed approaches are analyzed to compare their performance along with their advantages and limitations. Further, different databases relevant to FER are also explored in this study. Essentially, the main aim of this survey is twofold. The former is to discuss the current scenario of FER approaches and the latter is to present some thoughts on the future directions of facial emotion recognition by machines: what are the obstacles and prospects for FER researchers?. 

Understanding Deep Learning Techniques for Recognition of Human Emotions Using Facial Expressions: A Comprehensive Survey

Emotion recognition plays a significant role in cognitive psychology research. However, measuring emotions is a challenging task. Thus, several approaches have been designed for facial expression recognition (FER). Although, the challenges increase further as the data transit from the laboratory-controlled environment to in-the-wild circumstances, nowadays, applications are overwhelmed by a profusion of deep learning (DL) techniques in real-world problems. DL networks have steadily led to a better understanding of low-dimensional discriminative features from high-dimensional complex face patterns for automatic FER. The modern FER systems based on deep neural networks mainly suffer from two problems: overfitting due to the inadequate availability of training data and complications unassociated with the expressions, such as occlusion, posture, illumination, and identity bias. This study aims to provide a comprehensive survey of the significant DL-based methods that have made a notable contribution to the field of FER. Different components of the methods, such as preprocessing, feature extraction, and classification of facial expressions, are described systematically. Moreover, the discussed approaches are analyzed to compare their performance along with their advantages and limitations. Furthermore, different databases relevant to FER are also explored in this study. Essentially, the main aim of this survey is twofold. The former is to discuss the current scenario of FER approaches and the latter is to present some thoughts on the future directions of facial emotion recognition by machines: what are the obstacles and prospects for FER researchers?

With the development of deep learning technology, the performance of facial expression recognition (FER) has been significantly improved. The current main challenge comes from the confusion of facial expressions caused by the highly nonlinear changes of facial expressions. However, the existing FER methods based on Convolutional Neural Networks (CNN) often ignore the underlying relationship between expressions which is crucial to meliorate the performance of recognition for confusable expressions. And the methods based on Graph Convolutional Networks (GCN) can capture the relationship between vertices, but the aggregation degree of subgraphs generated by these methods is low. They are easy to include unconfident neighbors, which increases the learning difficulty of the network. To solve the above problems, this paper proposes a method to recognize facial expressions on the high aggregation subgraphs (HASs) by combing the advantages of CNN extracting features and GCN modeling complex graph patterns. Specifically, we formulate FER as a vertex prediction problem. Considering the importance of high-order neighbors and higher efficiency, we utilize vertex confidence to find high-order neighbors. Then we construct the HASs based on the top embedding features of these high-order neighbors. And we utilize the GCN to perform reasoning and infer the class of vertices for HASs without a large number of overlapping subgraphs. Our method captures the underlying relationship between expressions on the HASs and improves the accuracy and efficiency of FER. Experimental results on both the in-the-lab datasets and the in-the-wild datasets show that our method achieves higher recognition accuracy than several state-of-the-art methods. This highlights the benefit of the underlying relationship between expressions for FER. 

Facial Expression Recognition on the High Aggregation Subgraphs

A novel modular deep fully convolutional network for efficient low resolution facial expression recognition

Recent deep learning-based approaches on facial expression recognition face challenges on achieving optimal solution due to ineffective extraction of semantic expression features and difficulty in perceiving the positional relationship among facial expression features. In this paper we propose a novel framework, named Ventral-Dorsal Attention Capsule Network (VDACaps) to address the challenges. VDACaps adopts ResNet18 with a ventral channel attention and dilated spatial attention mechanism to strengthen the feature extraction. A novel attention layer containing multiple groups of ventral and dorsal attention is also designed to focus on the key features of facial expressions at the channel and spatial levels, exploring optimal sizes of receptive fields. Experiments on three well-known public expression datasets (CK+, JAFFE, and SFEW) have demonstrated that VDACaps achieves better or competitive performance on facial expression recognition against the state-of-the-art, with the accuracy of 98.98%, 98.44% and 54.07% respectively. 

Ventral-Dorsal Attention Capsule Network for facial expression recognition

Recognizing emotion through facial expression has now been widely applied in our daily lives. Therefore, facial expression recognition (FER) is attracting increasing research interests in the field of artificial intelligence and multimedia. With the development of convolutional neural networks (CNN), end-to-end deep learning frameworks for FER have achieved great success on large-scale datasets. However, these works still face the problems of redundant information and data bias, which obviously decrease the performance of FER. In this article, we propose a novel multiscale graph convolutional network (GCN) based on landmark graphs extracted from facial images. The proposed method is evaluated on different popular datasets. The results show that the proposed method outperforms the traditional deep learning frameworks and achieves more stable performance on different datasets.

Tianrong Rao

Papers

Facial Expression Recognition With Multiscale Graph Convolutional Networks