Emotion plays an important role in communication. For human-computer interaction, facial expression recognition has become an indispensable part. Recently, deep neural networks (DNNs) are widely used in this field and they overcome the limitations of conventional approaches. However, application of DNNs is very limited due to excessive hardware specifications requirement. Considering low hardware specifications used in real-life conditions, to gain better results without DNNs, in this paper, we propose an algorithm with the combination of the oriented FAST and rotated BRIEF (ORB) features and Local Binary Patterns (LBP) features extracted from facial expression. First of all, every image is passed through face detection algorithm to extract more effective features. Second, in order to increase computational speed, the ORB and LBP features are extracted from the face region; specifically, region division is innovatively employed in the traditional ORB to avoid the concentration of the features. The features are invariant to scale and grayscale as well as rotation changes. Finally, the combined features are classified by Support Vector Machine (SVM). The proposed method is evaluated on several challenging databases such as Cohn-Kanade database (CK+), Japanese Female Facial Expressions database (JAFFE), and MMI database; experimental results of seven emotion state (neutral, joy, sadness, surprise, anger, fear, and disgust) show that the proposed framework is effective and accurate.

/pdf/facial-expression-recognition-with-lbp-and-orb-features-e9vb4xu08w.pdf

Facial Expression Recognition with LBP and ORB Features

Affective computing has been an active area of research for the past two decades. One of the major component of affective computing is automatic emotion recognition. This chapter gives a detailed overview of different emotion recognition techniques and the predominantly used signal modalities. The discussion starts with the different emotion representations and their limitations. Given that affective computing is a data-driven research area, a thorough comparison of standard emotion labelled databases is presented. Based on the source of the data, feature extraction and analysis techniques are presented for emotion recognition. Further, applications of automatic emotion recognition are discussed along with current and important issues such as privacy and fairness.

A survey on automatic multimodal emotion recognition in the wild

In this letter, we propose a novel lightweight network HiNet: hybrid inherited feature learning network and its variants: HiNet -ReLU, HiNet -Concat, Large Scale HiNet, 2 stack HiNet and 4 stack HiNet for facial expression recognition. In HiNet, we introduce the hybrid feature (HyFeat) block that follows a two - level hybrid structure to capture the local contextual information of expressive regions. In level -1, HyFeat block uses two different scaled conv filters to preserve the domain knowledge features for expressive regions. Similarly, level -2 captures the enriched features through refine edge variation of the receptive fields by employing another two filters. Thus, HyFeat block allows network to perpetuate prominent features of facial expressions and enhances the discriminability of the HiNet. Moreover, performance of the proposed HiNet is evaluated by conducting person dependent (PD) and person independent (PI) experiments on four datasets: CK+, MUG, AFEW and OULU respectively. Experimental results on CK+ dataset prove the adequacy of the proposed HiNet over its variants in terms of accuracy and computational complexity. Furthermore, the proposed HiNet uses approximately 5 /16, 1/138, 1/144 and 1/31 times less parameters as compared to the MobileNet, VGG -16, VGG - 19 and ResNet, respectively. The experimental and computational complexity analysis demonstrate the significant performance improvement of the proposed method over the state-of-the-art techniques in terms of recognition rate.

HiNet: Hybrid Inherited Feature Learning Network for Facial Expression Recognition

Cognitive science and neuroscience use human facial expressions of emotion. Every single facial expression can be seen at different passions in a face space. Nowadays, facial expression recognition and analysis is vital due to the demand of introducing advanced biometric applications in every domain space. The imperative task in facial expressions of emotion classification is precise feature extraction, which helps to get detailed description of facial marks. Existing feature descriptors are suffering from various problems such as intensity variations, discrimination, vulnerability etc. In this paper, propose a new feature descriptor method called LDMEP (Local Directional Maximum Edge Patterns) for facial expression analysis to overcome the hindrance. We calculated the gradients in four directions of reference pixel to elicit the more feature for better recognition instead of calculating the local differences among neighboring pixels. We also access the orientations of the pixels then thresholded based on the dynamic threshold to avoid the featureless area calculation. Furthermore, we considered only dominant magnitude and orientation directions instead of all eight directions to generate feature. Thus, imperative and efficient features are covered in dominant positions to detect the strong edges. The paper confers that how the subsequent model can be used for the recognition of facial expression of emotion.

Local Directional Maximum Edge Patterns for facial expression recognition

Facial expression recognition (FER) is crucial for social communication. However, current studies present limitations when addressing facial expression difference due to demographic variation, such as race, gender, and age, etc. In this article, we first propose a deeply-supervised attention network (DSAN) to recognize human emotions based on facial images automatically. Based on DSAN, a two-stage training scheme is designed, taking full advantage of the race/gender/age-related information. In our DSAN framework, multi-scale features are leveraged to capture more discriminative information from the deep layers to the shallow layers. Furthermore, we adopt the attention block to highlight the essential local facial characteristics; it performs well when it is incorporated into the deeply-supervised framework. Finally, we combine the complementary characteristics of multiple convolutional layers in deeply-supervised manner and ensemble the intermediate predicted scores. Our experimental results have shown that our proposed framework can (i) effectively integrate demographic information in improving the performance of a variety of FER tasks, (ii) learn informative feature representations with a visual explanation by capturing the regions of interests (ROI), (iii) achieve superior performance for both the posed and the spontaneous FER databases, each containing pictures of human facial expressions varied in gender, age or race. 

Facial Expression Recognition With Deeply-Supervised Attention Network

Automated facial expression recognition plays a significant role in the study of human behaviour analysis. In this study, the authors propose a robust feature descriptor named regional adaptive affinitive patterns (RADAP) for facial expression recognition. The RADAP computes positional adaptive thresholds in the local neighbourhood and encodes multi-distance magnitude features which are robust to intra-class variations and irregular illumination variation in an image. Furthermore, they established cross-distance co-occurrence relations in RADAP by using logical operators. They proposed XRADAP, ARADAP, and DRADAP using xor, adder and decoder, respectively. The XRADAP engrains the quality of robustness to intra-class variations in RADAP features using pairwise co-occurrence. Similarly, ARADAP and DRADAP extract more stable and illumination invariant features and capture the minute expression features which are usually missed by regular descriptors. The performance of the proposed methods is evaluated by conducting experiments on nine benchmark datasets Cohn-Kanade+ (CK+), Japanese female facial expression (JAFFE), Multimedia Understanding Group (MUG), MMI, OULU-CASIA, Indian spontaneous expression database, DISFA, AFEW and Combined (CK+, JAFFE, MUG, MMI & GEMEP-FERA) database in both person dependent and person independent setup. The experimental results demonstrate the effectiveness of the proposed method over state-of-the-art approaches.

https://ietresearch.onlinelibrary.wiley.com/doi/epdf/10.1049/iet-ipr.2018.5683

Regional adaptive affinitive patterns (RADAP) with logical operators for facial expression recognition

A novel color feature descriptor, Multichannel Distributed Local Pattern (MDLP) is proposed in this manuscript. The MDLP combines the salient features of both local binary and local mesh patterns in the neighborhood. The multi-distance information computed by the MDLP aids in robust extraction of the texture arrangement. Further, MDLP features are extracted for each color channel of an image. The retrieval performance of the MDLP is evaluated on the three benchmark datasets for CBIR, namely Corel-5000, Corel-10000 and MIT-Color Vistex respectively. The proposed technique attains substantial improvement as compared to other state-of-the-art feature descriptors in terms of various evaluation parameters such as ARP and ARR on the respective databases.

Multichannel Distributed Local Pattern for Content Based Indexing and Retrieval

A novel color feature descriptor, Multichannel Distributed Local Pattern (MDLP) is proposed in this manuscript. The MDLP combines the salient features of both local binary and local mesh patterns in the neighborhood. The multi-distance information computed by the MDLP aids in robust extraction of the texture arrangement. Further, MDLP features are extracted for each color channel of an image. The retrieval performance of the MDLP is evaluated on the three benchmark datasets for CBIR, namely Corel-5000, Corel-10000 and MIT-Color Vistex respectively. The proposed technique attains substantial improvement as compared to other state-of- the-art feature descriptors in terms of various evaluation parameters such as ARP and ARR on the respective databases.

Sonakshi Mathur

Papers

Regional adaptive affinitive patterns (RADAP) with logical operators for facial expression recognition

Multichannel Distributed Local Pattern for Content Based Indexing and Retrieval

Multichannel Distributed Local Pattern for Content Based Indexing and Retrieval