Facial expressions are an important way through which humans interact socially. Building a system capable of automatically recognizing facial expressions from images and video has been an intense field of study in recent years. Interpreting such expressions remains challenging and much research is needed about the way they relate to human affect. This paper presents a general overview of automatic RGB, 3D, thermal and multimodal facial expression analysis. We define a new taxonomy for the field, encompassing all steps from face detection to facial expression recognition, and describe and classify the state of the art methods accordingly. We also present the important datasets and the bench-marking of most influential methods. We conclude with a general discussion about trends, important questions and future lines of research.

Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications

Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-related Applications

This article proposes an Enhanced Memetic Differential Evolution (EMDE) for designing digital filters which aim at detecting defects of the paper produced during an industrial process. Defect detection is handled by means of two Gabor filters and their design is performed by the EMDE. The EMDE is a novel adaptive evolutionary algorithm which combines the powerful explorative features of Differential Evolution with the exploitative features of three local search algorithms employing different pivot rules and neighborhood generating functions. These local search algorithms are the Hooke Jeeves Algorithm, a Stochastic Local Search, and Simulated Annealing. The local search algorithms are adaptively coordinated by means of a control parameter that measures fitness distribution among individuals of the population and a novel probabilistic scheme. Numerical results confirm that Differential Evolution is an efficient evolutionary framework for the image processing problem under investigation and show that the EMDE performs well. As a matter of fact, the application of the EMDE leads to a design of an efficiently tailored filter. A comparison with various popular metaheuristics proves the effectiveness of the EMDE in terms of convergence speed, stagnation prevention, and capability in detecting solutions having high performance.

An enhanced memetic differential evolution in filter design for defect detection in paper production

This work describes how evolutionary computation can be used to synthesize low-level image operators that detect interesting points on digital images. Interest point detection is an essential part of many modern computer vision systems that solve tasks such as object recognition, stereo correspondence, and image indexing, to name but a few. The design of the specialized operators is posed as an optimization/search problem that is solved with genetic programming (GP), a strategy still mostly unexplored by the computer vision community. The proposed approach automatically synthesizes operators that are competitive with state-of-the-art designs, taking into account an operator's geometric stability and the global separability of detected points during fitness evaluation. The GP search space is defined using simple primitive operations that are commonly found in point detectors proposed by the vision community. The experiments described in this paper extend previous results (Trujillo and Olague, 2006a,b) by presenting 15 new operators that were synthesized through the GP-based search. Some of the synthesized operators can be regarded as improved manmade designs because they employ well-known image processing techniques and achieve highly competitive performance. On the other hand, since the GP search also generates what can be considered as unconventional operators for point detection, these results provide a new perspective to feature extraction research.

Automated design of image operators that detect interest points

Facial Expression Recognition (FER) can be widely applied to various research areas, such as mental diseases diagnosis and human social/physiological interaction detection. With the emerging advanced technologies in hardware and sensors, FER systems have been developed to support real-world application scenes, instead of laboratory environments. Although the laboratory-controlled FER systems achieve very high accuracy, around 97%, the technical transferring from the laboratory to real-world applications faces a great barrier of very low accuracy, approximately 50%. In this survey, we comprehensively discuss three significant challenges in the unconstrained real-world environments, such as illumination variation, head pose, and subject-dependence, which may not be resolved by only analysing images/videos in the FER system. We focus on those sensors that may provide extra information and help the FER systems to detect emotion in both static images and video sequences. We introduce three categories of sensors that may help improve the accuracy and reliability of an expression recognition system by tackling the challenges mentioned above in pure image/video processing. The first group is detailed-face sensors, which detect a small dynamic change of a face component, such as eye-trackers, which may help differentiate the background noise and the feature of faces. The second is non-visual sensors, such as audio, depth, and EEG sensors, which provide extra information in addition to visual dimension and improve the recognition reliability for example in illumination variation and position shift situation. The last is target-focused sensors, such as infrared thermal sensors, which can facilitate the FER systems to filter useless visual contents and may help resist illumination variation. Also, we discuss the methods of fusing different inputs obtained from multimodal sensors in an emotion system. We comparatively review the most prominent multimodal emotional expression recognition approaches and point out their advantages and limitations. We briefly introduce the benchmark data sets related to FER systems for each category of sensors and extend our survey to the open challenges and issues. Meanwhile, we design a framework of an expression recognition system, which uses multimodal sensor data (provided by the three categories of sensors) to provide complete information about emotions to assist the pure face image/video analysis. We theoretically analyse the feasibility and achievability of our new expression recognition system, especially for the use in the wild environment, and point out the future directions to design an efficient, emotional expression recognition system.

https://www.mdpi.com/1424-8220/19/8/1863/pdf

A Review on Automatic Facial Expression Recognition Systems Assisted by Multimodal Sensor Data

We propose an unsupervised Local and Global feature extraction paradigm to approach the problem of facial expression recognition in thermal images. Starting from local, low-level features computed at interest point locations, our approach combines the localization of facial features with the holistic approach. The detailed steps are as follows: First, face localization using bi-modal thresholding is accomplished in order to localize facial features by way of a novel interest point detection and clustering approach. Second, we compute representative Eigenfeatures for feature extraction. Third, facial expression classification is made with a Support Vector Machine Committiee. Finally, the experiments over the IRIS data-set show that automation was achieved with good feature localization and classification performance.

Automatic Feature Localization in Thermal Images for Facial Expression Recognition

Visual learning of texture descriptors for facial expression recognition in thermal imagery

A new accurate and flexible model based multi-corner detector for measurement and recognition

The need for highly accurate classification systems capable of working on real-time applications has increased in recent years. Nowadays, several computer vision tasks apply a classification step as part of bigger systems, hence requiring classification models that work at a fast pace. This rendered interesting the concept of real-time object classification to several research communities. In this paper, we propose to accelerate a bio-inspired model for object classification, which has given very good results when compared with other state-of-the-art proposals using the compute unified device architecture (CUDA) and exploiting computational capabilities of graphic processing units. The classification model that is used is called the artificial visual cortex, a novel bio-inspired approach for image classification. In this work, we show that through an implementation of this model in the CUDA framework it is possible to achieve real-time functionality. As a result, the proposed system is able to process images in average of up to 90 times faster than the original system.

CUDA-based parallelization of a bio-inspired model for fast object classification

Corner feature extraction is studied in this paper as a global optimization problem. We propose a new parametric corner modeling based on a Unit Step Edge Function (USEF) that defines a straight line edge. This USEF function is a distribution function, which models the optical and physical characteristics present in digital photogrammetric systems. We search model parameters characterizing completely single gray-value structures by means of least squares fit of the model to the observed image intensities. As the identification results relies on the initial parameter values and as usual with non-linear cost functions in general we cannot guarantee to find the global minimum. Hence, we introduce an evolutionary algorithm using an affine transformation in order to estimate the model parameters. This transformation encapsulates within a single algebraic form the two main operations, mutation and crossover, of an evolutionary algorithm. Experimental results show the superiority of our L-corner model applying several levels of noise with respect to simplex and simulated annealing.

Benjamín Hernández

Papers

Automatic Feature Localization in Thermal Images for Facial Expression Recognition

Visual learning of texture descriptors for facial expression recognition in thermal imagery

A new accurate and flexible model based multi-corner detector for measurement and recognition

CUDA-based parallelization of a bio-inspired model for fast object classification

Accurate L-corner measurement using USEF functions and evolutionary algorithms