Facial emotion recognition (FER) is an important topic in the fields of computer vision and artificial intelligence owing to its significant academic and commercial potential. Although FER can be conducted using multiple sensors, this review focuses on studies that exclusively use facial images, because visual expressions are one of the main information channels in interpersonal communication. This paper provides a brief review of researches in the field of FER conducted over the past decades. First, conventional FER approaches are described along with a summary of the representative categories of FER systems and their main algorithms. Deep-learning-based FER approaches using deep networks enabling “end-to-end” learning are then presented. This review also focuses on an up-to-date hybrid deep-learning approach combining a convolutional neural network (CNN) for the spatial features of an individual frame and long short-term memory (LSTM) for temporal features of consecutive frames. In the later part of this paper, a brief review of publicly available evaluation metrics is given, and a comparison with benchmark results, which are a standard for a quantitative comparison of FER researches, is described. This review can serve as a brief guidebook to newcomers in the field of FER, providing basic knowledge and a general understanding of the latest state-of-the-art studies, as well as to experienced researchers looking for productive directions for future work.

A Brief Review of Facial Emotion Recognition Based on Visual Information.

Automatic machine-based Facial Expression Analysis (FEA) has made substantial progress in the past few decades driven by its importance for applications in psychology, security, health, entertainment, and human–computer interaction. The vast majority of completed FEA studies are based on nonoccluded faces collected in a controlled laboratory environment. Automatic expression recognition tolerant to partial occlusion remains less understood, particularly in real-world scenarios. In recent years, efforts investigating techniques to handle partial occlusion for FEA have seen an increase. The context is right for a comprehensive perspective of these developments and the state of the art from this perspective. This survey provides such a comprehensive review of recent advances in dataset creation, algorithm development, and investigations of the effects of occlusion critical for robust performance in FEA systems. It outlines existing challenges in overcoming partial occlusion and discusses possible opportunities in advancing the technology. To the best of our knowledge, it is the first FEA survey dedicated to occlusion and aimed at promoting better-informed and benchmarked future work.

Facial Expression Analysis under Partial Occlusion: A Survey

Occlusion and pose variations, which can change facial appearance significantly, are two major obstacles for automatic Facial Expression Recognition (FER). Though automatic FER has made substantial progresses in the past few decades, occlusion-robust and pose-invariant issues of FER have received relatively less attention, especially in real-world scenarios. This paper addresses the real-world pose and occlusion robust FER problem in the following aspects. First, to stimulate the research of FER under real-world occlusions and variant poses, we annotate several in-the-wild FER datasets with pose and occlusion attributes for the community. Second, we propose a novel Region Attention Network (RAN), to adaptively capture the importance of facial regions for occlusion and pose variant FER. The RAN aggregates and embeds varied number of region features produced by a backbone convolutional neural network into a compact fixed-length representation. Last, inspired by the fact that facial expressions are mainly defined by facial action units, we propose a region biased loss to encourage high attention weights for the most important regions. We validate our RAN and region biased loss on both our built test datasets and four popular datasets: FERPlus, AffectNet, RAF-DB, and SFEW. Extensive experiments show that our RAN and region biased loss largely improve the performance of FER with occlusion and variant pose. Our method also achieves state-of-the-art results on FERPlus, AffectNet, RAF-DB, and SFEW. Code and the collected test data will be publicly available.

Region Attention Networks for Pose and Occlusion Robust Facial Expression Recognition

Speech emotion recognition is challenging because of the affective gap between the subjective emotions and low-level features. Integrating multilevel feature learning and model training, deep convolutional neural networks (DCNN) has exhibited remarkable success in bridging the semantic gap in visual tasks like image classification, object detection. This paper explores how to utilize a DCNN to bridge the affective gap in speech signals. To this end, we first extract three channels of log Mel-spectrograms (static, delta, and delta delta) similar to the red, green, blue (RGB) image representation as the DCNN input. Then, the AlexNet DCNN model pretrained on the large ImageNet dataset is employed to learn high-level feature representations on each segment divided from an utterance. The learned segment-level features are aggregated by a discriminant temporal pyramid matching (DTPM) strategy. DTPM combines temporal pyramid matching and optimal Lp-norm pooling to form a global utterance-level feature representation, followed by the linear support vector machines for emotion classification. Experimental results on four public datasets, that is, EMO-DB, RML, eNTERFACE05, and BAUM-1s, show the promising performance of our DCNN model and the DTPM strategy. Another interesting finding is that the DCNN model pretrained for image applications performs reasonably good in affective speech feature extraction. Further fine tuning on the target emotional speech datasets substantially promotes recognition performance.

Speech Emotion Recognition Using Deep Convolutional Neural Network and Discriminant Temporal Pyramid Matching

Written for students of linguistics, applied linguistics and speech therapy, this dictionary covers over 2,000 terms in phonetics and phonology. In addition to providing a comprehensive, yet concise, guide to an enormous number of individual terms, it also includes an explanation of the most important theoretical approaches to phonology. Its usefulness as a reference tool is further enhanced by the inclusion of pronunciations, notational devices and symbols, earliest sources of terms, suggestions for further reading, and advice with regard to usage. The wide range of topics explained include: * Classical phonology, including American Structuralism and the Prague School * Contemporary approaches, including Autosegmental Phonology, Metrical Phonology, Dependency Phonology, Government Phonology and Lexical Phonology * Prosodic ideas in phonology, both traditional and contemporary ^ * * historical phonology * Intonation and tonology This dictionary devotes space to the various theoretical approaches in proportion to their importance, but it concentrates most heavily on non-theory-bound descriptive terminology. It will remain a definitive reference for years to come.

语音学和音系学词典 = A dictionary of phonetics and phonology

Recently, compressive sensing (CS) has attracted increasing attention in the areas of signal processing, computer vision and pattern recognition. In this paper, a new method based on the CS theory is presented for robust facial expression recognition. The CS theory is used to construct a sparse representation classifier (SRC). The effectiveness and robustness of the SRC method is investigated on clean and occluded facial expression images. Three typical facial features, i.e., the raw pixels, Gabor wavelets representation and local binary patterns (LBP), are extracted to evaluate the performance of the SRC method. Compared with the nearest neighbor (NN), linear support vector machines (SVM) and the nearest subspace (NS), experimental results on the popular Cohn-Kanade facial expression database demonstrate that the SRC method obtains better performance and stronger robustness to corruption and occlusion on robust facial expression recognition tasks.

/pdf/robust-facial-expression-recognition-via-compressive-sensing-snmf9kacb2.pdf

Robust facial expression recognition via compressive sensing.

Automatic facial expression recognition is an interesting and challenging subject in signal processing, pattern recognition, artificial intelligence, etc. In this paper, a new method of facial expression recognition based on local binary patterns (LBP) and local Fisher discriminant analysis (LFDA) is presented. The LBP features are firstly extracted from the original facial expression images. Then LFDA is used to produce the low dimensional discriminative embedded data representations from the extracted high dimensional LBP features with striking performance improvement on facial expression recognition tasks. Finally, support vector machines (SVM) classifier is used for facial expression classification. The experimental results on the popular JAFFE facial expression database demonstrate that the presented facial expression recognition method based on LBP and LFDA obtains the best recognition accuracy of 90.7% with 11 reduced features, outperforming the other used methods such as principal component analysis (PCA), linear discriminant analysis (LDA), locality preserving projection (LPP).

/pdf/facial-expression-recognition-based-on-local-binary-patterns-h01ef85zo8.pdf

Facial expression recognition based on local binary patterns and local fisher discriminant analysis

Emotion recognition in speech signals is currently a very active research topic and has attracted much attention within the engineering application area. This paper presents a new approach of robust emotion recognition in speech signals in noisy environment. By using a weighted sparse representation model based on the maximum likelihood estimation, an enhanced sparse representation classifier is proposed for robust emotion recognition in noisy speech. The effectiveness and robustness of the proposed method is investigated on clean and noisy emotional speech. The proposed method is compared with six typical classifiers, including linear discriminant classifier, K-nearest neighbor, C4.5 decision tree, radial basis function neural networks, support vector machines as well as sparse representation classifier. Experimental results on two publicly available emotional speech databases, that is, the Berlin database and the Polish database, demonstrate the promising performance of the proposed method on the task of robust emotion recognition in noisy speech, outperforming the other used methods.

Robust emotion recognition in noisy speech via sparse representation

Speech emotion recognition is currently an active research subject and has attracted extensive interest in the science community due to its vital application to human-robot interaction. Most speech...

Speech Emotion Recognition Using an Enhanced Kernel Isomap for Human-Robot Interaction

A bstract: - Facial expression recognition is an interesting and challenging subject in signal processing and artificial intelligence. In this paper, a new method of facial expression recognition based on the sparse representation classifier (SRC) is presented. Two typical appearance facial features, i.e., local binary patterns (LBP) and Gabor wavelets representations are extracted to evaluate the performance of the SRC method on facial expression recognition tasks. Three representative classification methods, including artificial neural network (ANN), K-nearest neighbor (KNN), support vector machines (SVM), are used to compare with the SRC method. Experimental results on two popular facial expression databases, i.e., the JAFFE database and the Cohn-Kanade database, demonstrate the promising performance of the presented SRC method on facial expression recognition tasks, outperforming the other used methods.

https://www.wseas.org/multimedia/journals/systems/2012/56-519.pdf

Xiaoming Zhao

Papers

Robust facial expression recognition via compressive sensing.

Facial expression recognition based on local binary patterns and local fisher discriminant analysis

Robust emotion recognition in noisy speech via sparse representation

Speech Emotion Recognition Using an Enhanced Kernel Isomap for Human-Robot Interaction

Facial Expression Recognition Using Sparse Representation