scispace - formally typeset
Search or ask a question
Proceedings Article•DOI•

Recognizing facial expressions using novel motion based features

TL;DR: This paper introduces two novel motion based features for recognizing human facial expressions, which represents each frame of a video sequence as a vector depicting local motion patterns during a facial expression and forms expression descriptors for each expression from the reduced dictionary.
Abstract: This paper introduces two novel motion based features for recognizing human facial expressions. The proposed motion features are applied for recognizing facial expressions from a video sequence. The proposed bag-of-words based scheme represents each frame of a video sequence as a vector depicting local motion patterns during a facial expression. The local motion patterns are captured by an efficient derivation from optical flow. Motion features are clustered and stored as words in a dictionary. We further generate a reduced dictionary by ranking the words based on some ambiguity measure. We prune out the ambiguous words and continue with key words in the reduced dictionary. The ambiguity measure is given by applying a graph-based technique, where each word is represented as a node in the graph. Ambiguity measures are obtained by modelling the frequency of occurrence of the word during the expression. We form expression descriptors for each expression from the reduced dictionary, by applying an efficient kernel. The training of the expression descriptors are made following an adaptive learning technique. We tested the proposed approach with standard dataset. The proposed approach shows better accuracy compared to the state-of-the-art.
Citations
More filters
Proceedings Article•DOI•
14 Jul 2019
TL;DR: Two 3D-CNN methods are proposed: MicroExpSTCNN and MicroExpFuseNet, for spontaneous facial micro-expression recognition by exploiting the spatiotemporal information in CNN framework, which outperforms the state-of-the-art methods.
Abstract: Facial expression recognition in videos is an active area of research in computer vision. However, fake facial expressions are difficult to be recognized even by humans. On the other hand, facial micro-expressions generally represent the actual emotion of a person, as it is a spontaneous reaction expressed through human face. Despite of a few attempts made for recognizing micro-expressions, still the problem is far from being a solved problem, which is depicted by the poor rate of accuracy shown by the state-of-the-art methods. A few CNN based approaches are found in the literature to recognize micro-facial expressions from still images. Whereas, a spontaneous microexpression video contains multiple frames that have to be processed together to encode both spatial and temporal information. This paper proposes two 3D-CNN methods: MicroExpSTCNN and MicroExpFuseNet, for spontaneous facial micro-expression recognition by exploiting the spatiotemporal information in CNN framework. The MicroExpSTCNN considers the full spatial information, whereas the MicroExpFuseNet is based on the 3D-CNN feature fusion of the eyes and mouth regions. The experiments are performed over CAS(ME)2 and SMIC microb expression databases. The proposed MicroExpSTCNN model outperforms the state-of-the-art methods.

50 citations


Cites methods from "Recognizing facial expressions usin..."

  • ...micro-expression descriptors extracted from the images/videos [3], [4]....

    [...]

Journal Article•DOI•
TL;DR: The proposed fusion of the hand-crafted and XceptionNet features outperforms the state-of-the-art methods for facial expression recognition in the wild.

24 citations

Posted Content•
TL;DR: Zhang et al. as discussed by the authors proposed two 3D-CNN methods, MicroExpSTCNN and MicroExpFuseNet, for spontaneous facial micro-expression recognition by exploiting the spatio-temporal information in CNN framework.
Abstract: Facial expression recognition in videos is an active area of research in computer vision. However, fake facial expressions are difficult to be recognized even by humans. On the other hand, facial micro-expressions generally represent the actual emotion of a person, as it is a spontaneous reaction expressed through human face. Despite of a few attempts made for recognizing micro-expressions, still the problem is far from being a solved problem, which is depicted by the poor rate of accuracy shown by the state-of-the-art methods. A few CNN based approaches are found in the literature to recognize micro-facial expressions from still images. Whereas, a spontaneous micro-expression video contains multiple frames that have to be processed together to encode both spatial and temporal information. This paper proposes two 3D-CNN methods: MicroExpSTCNN and MicroExpFuseNet, for spontaneous facial micro-expression recognition by exploiting the spatiotemporal information in CNN framework. The MicroExpSTCNN considers the full spatial information, whereas the MicroExpFuseNet is based on the 3D-CNN feature fusion of the eyes and mouth regions. The experiments are performed over CAS(ME)^2 and SMIC micro-expression databases. The proposed MicroExpSTCNN model outperforms the state-of-the-art methods.

22 citations

Proceedings Article•DOI•
01 Feb 2020
TL;DR: This research provides a broad overview of the FER process includes all stages of FER system as well as the various methods used to evaluate the efficiency of theVarious methods of facial expression recognition.
Abstract: In recent years, many researchers are taking an interest in the research area of Face recognition due to its diverse applications such as security systems, medical systems, entertainment. Nowadays, many kinds of biometric information processing systems are used for various purpose face-recognition systems. The facial expression that can define the human mental state and behavior and it is used for security purpose. FER is used in domains such as healthcare, marketing, environment, safety and social media. This paper presents the survey of the facial expression recognition system that includes the four main stages i.e. face detection, pre-processing, extraction of features, and classification. This research provides a broad overview of the FER process includes all stages of FER system as well as the various methods used to evaluate the efficiency of the various methods of facial expression recognition. This survey paper also helps to understand the approaches, different techniques that address and analyse the problems and challenges comes in the real-time environment. Finally, this paper concludes the state-of-the-art and explore the challenges faced in implementation of FER process along with the scope of future development.

3 citations


Cites background from "Recognizing facial expressions usin..."

  • ...[11], suggest two motion descriptors and applied to recognize facial expressions....

    [...]

  • ...The extended version (CK+) comprises 593 posed sequences of speech from 122 sequences of 66 subjects of the spontaneous smile.[11, 29]...

    [...]

Book Chapter•DOI•
01 Jan 2021
TL;DR: In this paper, an amalgam of convolution neural network (CNN) and long short-term memory (LSTM) was employed to extract essential features for recognizing the expression from the target frame.
Abstract: Expressions play an imperative part in human interactions as it let humans express their intentions and feelings without words. Mental state of a subject can be mined from the expression extracted. In recent times, deep neural network has outperformed traditional handcrafted descriptors including spatiotemporal local binary pattern (LBP), LBP-TOP, HOG, etc., when it comes to feature extraction for facial expression recognition (FER). In this paper, an amalgam of convolution neural network (CNN) and long short-term memory (LSTM) [recurrent neural network (RNN)] is employed to extract essential features for recognizing the expression from the target frame. To increase the performance, transfer learning concept is engaged to get learned parameters (weight/bias). To accomplish transfer learning, leading layers of ResNet-50 (trained on thousands of image frames) are used. Further, a LSTM layer (time distributed) is affixed to the existing model. The model is further trained (CK+ database) with different activation functions, and a relative analysis is performed. Maximum accuracy of 94% is attained with the hybrid model (CNN-LSTM with SELU and ELU).
References
More filters
Book•
01 Jan 1990
TL;DR: The updated new edition of the classic Introduction to Algorithms is intended primarily for use in undergraduate or graduate courses in algorithms or data structures and presents a rich variety of algorithms and covers them in considerable depth while making their design and analysis accessible to all levels of readers.
Abstract: From the Publisher: The updated new edition of the classic Introduction to Algorithms is intended primarily for use in undergraduate or graduate courses in algorithms or data structures. Like the first edition,this text can also be used for self-study by technical professionals since it discusses engineering issues in algorithm design as well as the mathematical aspects. In its new edition,Introduction to Algorithms continues to provide a comprehensive introduction to the modern study of algorithms. The revision has been updated to reflect changes in the years since the book's original publication. New chapters on the role of algorithms in computing and on probabilistic analysis and randomized algorithms have been included. Sections throughout the book have been rewritten for increased clarity,and material has been added wherever a fuller explanation has seemed useful or new information warrants expanded coverage. As in the classic first edition,this new edition of Introduction to Algorithms presents a rich variety of algorithms and covers them in considerable depth while making their design and analysis accessible to all levels of readers. Further,the algorithms are presented in pseudocode to make the book easily accessible to students from all programming language backgrounds. Each chapter presents an algorithm,a design technique,an application area,or a related topic. The chapters are not dependent on one another,so the instructor can organize his or her use of the book in the way that best suits the course's needs. Additionally,the new edition offers a 25% increase over the first edition in the number of problems,giving the book 155 problems and over 900 exercises thatreinforcethe concepts the students are learning.

21,651 citations

01 Jan 2005

19,250 citations

Book Chapter•DOI•
01 Jan 2014
TL;DR: This chapter provides an overview of the fundamentals of algorithms and their links to self-organization, exploration, and exploitation.
Abstract: Algorithms are important tools for solving problems computationally. All computation involves algorithms, and the efficiency of an algorithm largely determines its usefulness. This chapter provides an overview of the fundamentals of algorithms and their links to self-organization, exploration, and exploitation. A brief history of recent nature-inspired algorithms for optimization is outlined in this chapter.

8,285 citations

Proceedings Article•DOI•
13 Jun 2010
TL;DR: The Cohn-Kanade (CK+) database is presented, with baseline results using Active Appearance Models (AAMs) and a linear support vector machine (SVM) classifier using a leave-one-out subject cross-validation for both AU and emotion detection for the posed data.
Abstract: In 2000, the Cohn-Kanade (CK) database was released for the purpose of promoting research into automatically detecting individual facial expressions. Since then, the CK database has become one of the most widely used test-beds for algorithm development and evaluation. During this period, three limitations have become apparent: 1) While AU codes are well validated, emotion labels are not, as they refer to what was requested rather than what was actually performed, 2) The lack of a common performance metric against which to evaluate new algorithms, and 3) Standard protocols for common databases have not emerged. As a consequence, the CK database has been used for both AU and emotion detection (even though labels for the latter have not been validated), comparison with benchmark algorithms is missing, and use of random subsets of the original database makes meta-analyses difficult. To address these and other concerns, we present the Extended Cohn-Kanade (CK+) database. The number of sequences is increased by 22% and the number of subjects by 27%. The target expression for each sequence is fully FACS coded and emotion labels have been revised and validated. In addition to this, non-posed sequences for several types of smiles and their associated metadata have been added. We present baseline results using Active Appearance Models (AAMs) and a linear support vector machine (SVM) classifier using a leave-one-out subject cross-validation for both AU and emotion detection for the posed data. The emotion and AU labels, along with the extended image data and tracked landmarks will be made available July 2010.

3,439 citations

Proceedings Article•DOI•
26 Mar 2000
TL;DR: The problem space for facial expression analysis is described, which includes level of description, transitions among expressions, eliciting conditions, reliability and validity of training and test data, individual differences in subjects, head orientation and scene complexity image characteristics, and relation to non-verbal behavior.
Abstract: Within the past decade, significant effort has occurred in developing methods of facial expression analysis. Because most investigators have used relatively limited data sets, the generalizability of these various methods remains unknown. We describe the problem space for facial expression analysis, which includes level of description, transitions among expressions, eliciting conditions, reliability and validity of training and test data, individual differences in subjects, head orientation and scene complexity image characteristics, and relation to non-verbal behavior. We then present the CMU-Pittsburgh AU-Coded Face Expression Image Database, which currently includes 2105 digitized image sequences from 182 adult subjects of varying ethnicity, performing multiple tokens of most primary FACS action units. This database is the most comprehensive testbed to date for comparative studies of facial expression analysis.

2,705 citations