ME-PLAN: A deep prototypical learning with local attention network for dynamic micro-expression recognition

Micro-expression (ME) is a significant non-verbal communication clue that reveals one person's genuine emotional state. The development of micro-expression analysis (MEA) has just gained attention in the last decade. However, the small sample size problem constrains the use of deep learning on MEA. Besides, ME samples distribute in six different databases, leading to database bias. Moreover, the ME database development is complicated. In this article, we introduce a large-scale spontaneous ME database: CAS(ME) 3. The contribution of this article is summarized as follows: (1) CAS(ME) 3 offers around 80 hours of videos with over 8,000,000 frames, including manually labeled 1,109 MEs and 3,490 macro-expressions. Such a large sample size allows effective MEA method validation while avoiding database bias. (2) Inspired by psychological experiments, CAS(ME) 3 provides the depth information as an additional modality unprecedentedly, contributing to multi-modal MEA. (3) For the first time, CAS(ME) 3 elicits ME with high ecological validity using the mock crime paradigm, along with physiological and voice signals, contributing to practical MEA. (4) Besides, CAS(ME) 3 provides 1,508 unlabeled videos with more than 4,000,000 frames, i.e., a data platform for unsupervised MEA methods. (5) Finally, we demonstrate the effectiveness of depth information by the proposed depth flow algorithm and RGB-D information. 

CAS(ME)&lt;sup&gt;3&lt;/sup&gt;: A Third Generation Facial Spontaneous Micro-Expression Database with Depth Information and High Ecological Validity

A micro-expression is a subtle, local and brief facial movement. It can reveal the genuine emotions that a person tries to conceal and is considered an important clue for lie detection. The micro-expression research has attracted much attention due to its promising applications in various fields. However, due to the short duration and low intensity of micro-expression movements, micro-expression recognition faces great challenges, and the accuracy still demands improvement. To improve the efficiency of micro-expression feature extraction, inspired by the psychological study of attentional resource allocation for micro-expression cognition, we propose a deep local-holistic network method for micro-expression recognition. Our proposed algorithm consists of two sub-networks. The first is a Hierarchical Convolutional Recurrent Neural Network (HCRNN), which extracts the local and abundant spatio-temporal micro-expression features. The second is a Robust principal-component-analysis-based recurrent neural network (RPRNN), which extracts global and sparse features with micro-expression-specific representations. The extracted effective features are employed for micro-expression recognition through the fusion of sub-networks. We evaluate the proposed method on combined databases consisting of the four most commonly used databases, i.e., CASME, CASME II, CAS(ME)2, and SAMM. The experimental results show that our method achieves a reasonably good performance.

/pdf/facial-micro-expression-recognition-based-on-deep-local-29gs9pcq.pdf

Facial Micro-Expression Recognition Based on Deep Local-Holistic Network

Facial micro-expression (ME) refers to a brief spontaneous facial movement that can reveal the genuine emotion of a person. The absence of data is a major problem for ME. Thankfully, generative deep neural network models can aid in producing desired samples. In this work, we proposed a deep learning based adaptive dual motion model (ADMM) for generating facial ME samples. A dual motion extraction (DME) module extracts robust motions from two modalities: original color images and edge-based grayscale images, with dual streams. Using edge-based grayscale images can help the method focus on learning subtle movements by eliminating the influences of noises and illumination variants. The motions extracted by the dual streams are fed into an adaptive motion fusion (AMF) module for combing the motions adaptively to generate the dense motion. Our method was trained on the CASME II, SMIC, and SAMM datasets. The evaluation and analysis of the results demonstrated the effectiveness of our method.

Adaptive Dual Motion Model for Facial Micro-Expression Generation

Temporal augmented contrastive learning for micro-expression recognition

Feature refinement: An expression-specific feature learning and fusion method for micro-expression recognition

Micro-Expression Recognition has become challenging, as it is extremely difficult to extract the subtle facial changes of micro-expressions. Recently, several approaches proposed several expression-shared features algorithms for micro-expression recognition. However, they do not reveal the specific discriminative characteristics, which lead to sub-optimal performance. This paper proposes a novel Feature Refinement ({FR}) with expression-specific feature learning and fusion for micro-expression recognition. It aims to obtain salient and discriminative features for specific expressions and also predict expression by fusing the expression-specific features. FR consists of an expression proposal module with attention mechanism and a classification branch. First, an inception module is designed based on optical flow to obtain expression-shared features. Second, in order to extract salient and discriminative features for specific expression, expression-shared features are fed into an expression proposal module with attention factors and proposal loss. Last, in the classification branch, labels of categories are predicted by a fusion of the expression-specific features. Experiments on three publicly available databases validate the effectiveness of FR under different protocol. Results on public benchmarks demonstrate that our FR provides salient and discriminative information for micro-expression recognition. The results also show our FR achieves better or competitive performance with the existing state-of-the-art methods on micro-expression recognition.

Feifei Zhang

Papers

Feature refinement: An expression-specific feature learning and fusion method for micro-expression recognition

Feature refinement: An expression-specific feature learning and fusion method for micro-expression recognition