Showing papers on "Facial Action Coding System published in 2011"
TL;DR: An automated FACS based on advanced computer science technology was developed and the quantitative measures of flatness and inappropriateness showed clear differences between patients and the controls, highlighting their potential in automatic and objective quantification of symptom severity.
242 citations
TL;DR: Testing the assumption that emotional mimicry and contagion are moderated by group membership shows that ingroup anger and fear displays were mimicked to a greater extent than outgroup displays of these emotions, and mimicry increased liking for ingroup models but not for outgroup models.
Abstract: In the present research, we test the assumption that emotional mimicry and contagion are moderated by group membership. We report two studies using facial electromyography (EMG; Study 1), Facial Action Coding System (FACS; Study 2), and self-reported emotions (Study 2) as dependent measures. As predicted, both studies show that ingroup anger and fear displays were mimicked to a greater extent than outgroup displays of these emotions. The self-report data in Study 2 further showed specific divergent reactions to outgroup anger and fear displays. Outgroup anger evoked fear, and outgroup fear evoked aversion. Interestingly, mimicry increased liking for ingroup models but not for outgroup models. The findings are discussed in terms of the social functions of emotions in group contexts.
217 citations
TL;DR: In this paper, the authors developed FACSGen, a tool that allows the creation of realistic synthetic 3D facial stimuli, both static and dynamic, based on the Facial Action Coding System.
Abstract: To investigate the perception of emotional facial expressions, researchers rely on shared sets of photos or videos, most often generated by actor portrayals. The drawback of such standardized material is a lack of flexibility and controllability, as it does not allow the systematic parametric manipulation of specific features of facial expressions on the one hand, and of more general properties of the facial identity (age, ethnicity, gender) on the other. To remedy this problem, we developed FACSGen: a novel tool that allows the creation of realistic synthetic 3D facial stimuli, both static and dynamic, based on the Facial Action Coding System. FACSGen provides researchers with total control over facial action units, and corresponding informational cues in 3D synthetic faces. We present four studies validating both the software and the general methodology of systematically generating controlled facial expression patterns for stimulus presentation.
105 citations
TL;DR: Facial emotional reactions in patients with borderline personality disorder when playing Cyberball, a virtual ball-tossing game that reliably induces social exclusion, revealed that BPD patients reacted with fewer positive expressions and with significantly more mixed emotional expressions when excluded.
Abstract: BackgroundDisturbances in social interaction are a defining feature of patients with borderline personality disorder (BPD). In this study, facial emotional expressions, which are crucial for adaptive interactions in social contexts, were assessed in patients with BPD in response to social exclusion.MethodWe examined facial emotional reactions of 35 patients with BPD and 33 healthy controls when playing Cyberball, a virtual ball-tossing game that reliably induces social exclusion. Besides self-reported emotional responses, facial emotional expressions were analyzed by applying the Emotional Facial Action Coding System (EMFACS).ResultsPatients with BPD showed a biased perception of participation. They more readily reported feeling excluded compared to controls even when they were included. In BPD, social exclusion led to an increase in self-reported other-focused negative emotions. Overall, EMFACS analyses revealed that BPD patients reacted with fewer positive expressions and with significantly more mixed emotional expressions (two emotional facial expressions at the same time) compared to the healthy control group when excluded.ConclusionsBesides a negative bias for perceived social participation, ambiguous facial emotional expressions may play an important role in the disturbed relatedness in patients with BPD.
98 citations
TL;DR: It is shown that thermal fluctuations are specific to the activated AUs and are sensitive to the kinetics and intensities of AU production, which open new avenues for studying patterns of facial muscle activity related to emotion or other cognitively induced activities, in a noninvasive manner, avoiding potential lighting issues.
Abstract: Facial expressions can be systematically coded using the Facial Action Coding System (FACS) that describes the specific action unit (AU) or combination of AUs elicited during different kinds of expressions. This study investigated the thermal patterns concomitant to specific action units performance. As thermal imaging can track dynamic patterns in facial temperature at any distance (>; 0.4 m), with high temporal (<; 20 m) and thermal (<; 20 mK@300 K) resolutions, this noninvasive technique was tested as a method to assess fluctuations of facial heat patterns induced by facial muscles contractions. Four FACS-trained coders produced nine different AUs or combination of AUs at various speeds and intensities. Using a spatial pattern approach based on PCA decomposition of the thermal signal, we showed that thermal fluctuations are specific to the activated AUs and are sensitive to the kinetics and intensities of AU production. These results open new avenues for studying patterns of facial muscle activity related to emotion or other cognitively induced activities, in a noninvasive manner, avoiding potential lighting issues.
76 citations
TL;DR: The authors investigated prototypical expressions of four positive emotions (interest, pride, pleasure, and joy) from the Geneva Multimodal Emotion Portrayal corpus and found that the frequency and duration of several action units differed between emotions, indicating that actors did not use the same pattern of expression to encode them.
Abstract: Positive emotions are crucial to social relationships and social interaction. Although smiling is a frequently studied facial action, investigations of positive emotional expressions are underrepresented in the literature. This may be partly because of the assumption that all positive emotions share the smile as a common signal but lack specific facial configurations. The present study investigated prototypical expressions of four positive emotions—interest, pride, pleasure, and joy. The Facial Action Coding System was used to microcode facial expression of representative samples of these emotions taken from the Geneva Multimodal Emotion Portrayal corpus. The data showed that the frequency and duration of several action units differed between emotions, indicating that actors did not use the same pattern of expression to encode them. The authors argue that an appraisal perspective is suitable to describe how subtly differentiated positive emotional states differ in their prototypical facial expressions.
70 citations
TL;DR: The results suggest that each emotion can be portrayed by several different expressions that share multiple facial actions, cast doubt on the existence of fixed patterns of facial responses for each emotion, resulting in unique facial prototypes.
Abstract: Affect bursts consist of spontaneous and short emotional expressions in which facial, vocal, and gestural components are highly synchronized. Although the vocal characteristics have been examined in several recent studies, the facial modality remains largely unexplored. This study investigated the facial correlates of affect bursts that expressed five different emotions: anger, fear, sadness, joy, and relief. Detailed analysis of 59 facial actions with the Facial Action Coding System revealed a reasonable degree of emotion differentiation for individual action units (AUs). However, less convergence was shown for specific AU combinations for a limited number of prototypes. Moreover, expression of facial actions peaked in a cumulative-sequential fashion with significant differences in their sequential appearance between emotions. When testing for the classification of facial expressions within a dimensional approach, facial actions differed significantly as a function of the valence and arousal level of the five emotions, thereby allowing further distinction between joy and relief. The findings cast doubt on the existence of fixed patterns of facial responses for each emotion, resulting in unique facial prototypes. Rather, the results suggest that each emotion can be portrayed by several different expressions that share multiple facial actions.
68 citations
21 Mar 2011
TL;DR: This project applies the computer expression recognition toolbox (CERT) to videos of behavioral data from children between the ages of 3 and 9, particularly focusing on the spontaneous expressions of uncertainty, and demonstrates differences in expression dynamics between older and younger children during problem solving.
Abstract: There has been growing recognition of the importance of adaptive tutoring systems that respond to the student's emotional and cognitive state. However little is known about children's facial expressions during a problem solving task. What are the actual signals of boredom, interest, or confusion in real, spontaneous behavior of students? The field also is in need of spontaneous datasets to drive automated recognition of these states. This project aims to collect, measure, and describe spontaneous facial expressions of children during problem solving. We apply the computer expression recognition toolbox (CERT) to videos of behavioral data from children between the ages of 3 and 9, particularly focusing on the spontaneous expressions of uncertainty. From the Facial Action outputs, we analyze changes in facial expression during problem solving, and differences in expression between correct and incorrect trials. Moreover, we demonstrate differences in expression dynamics between older and younger children during problem solving. Future work examines differences between facial action and other modalities such as voice pitch.
66 citations
Metz1
TL;DR: Experimental results demonstrate that the proposed approach is an effective method to recognize emotions through facial expression with an emotion recognition rate more than90% in real time.
Abstract: This paper describes emotion recognition system based on facial expression. A fully automatic facial expression recognition system is based on three steps: face detection, facial characteristic extraction and facial expression classification. We have developed an anthropometric model to detect facial feature points combined to Shi&, Thomasi method. The variations of 21 distances which describe the facial features deformations from the neutral face, were used to coding the facial expression. Classification step is based on SVM method (Support Vector Machine). Experimental results demonstrate that the proposed approach is an effective method to recognize emotions through facial expression with an emotion recognition rate more than90% in real time. This approach is used to control music player based on the variation of the emotional state of the computer user.
61 citations
22 May 2011
TL;DR: A robust method to map detected facial Action Units (AUs) to six basic emotions using a learned statistical relationship and a suitable matching technique to reduce false predictions and improve performance with rule based techniques is presented.
Abstract: We present a robust method to map detected facial Action Units (AUs) to six basic emotions. Automatic AU recognition is prone to errors due to illumination, tracking failures and occlusions. Hence, traditional rule based methods to map AUs to emotions are very sensitive to false positives and misses among the AUs. In our method, a set of chosen AUs are mapped to the six basic emotions using a learned statistical relationship and a suitable matching technique. Relationships between the AUs and emotions are captured as template strings comprising the most discriminative AUs for each emotion. The template strings are computed using a concept called discriminative power. The Longest Common Subsequence (LCS) distance, an approach for approximate string matching, is applied to calculate the closeness of a test string of AUs with the template strings, and hence infer the underlying emotions. LCS is found to be efficient in handling practical issues like erroneous AU detection and helps to reduce false predictions. The proposed method is tested with various databases like CK+, ISL, FACS, JAFFE, MindReading and many real-world video frames. We compare our performance with rule based techniques, and show clear improvement on both benchmark databases and real-world datasets.
56 citations
TL;DR: Scales that provided specific descriptions using the empirically displayed facial actions associated with pain yielded greater sensitivity, interjudge reliability, and validity as indices of pain.
Abstract: Objectives: Assessing pain in elderly persons, who have diminished capacity to communicate verbally, requires use of observational scales that focus upon nonverbal behavior. Facial expression has been recognized as providing the most specific and sensitive nonverbal cues for pain. This study examined the validity of facial expression components of 6 widely used pain assessment scales developed for elders with dementia. Descriptions of the facial expression of pain vary widely on these scales. Methods: The detailed, anatomically based, objectively coded, and validated Facial Action Coding System was used as a criterion index to provide a definitive description of the facial expression of pain. Thirty elderly inpatients with clinically significant pain in the back or hip, the majority of whom had cognitive impairments, provided videotaped reactions to physical activities. Participants’ facial expressions were videotaped during 4 randomly ordered physical activities and coded by a qualified Facial Action Coding System coder. Three 6-second clips indicative of mild, moderate, and severe pain intensities were selected for study for each participant. The 90 clips were coded by 5 raters using the facial expression components of the following observational scales: Doloplus-2, Mahoney, Abbey, pain assessment checklist for seniors with limited ability to communicate, noncommunicative patient’s Pain Assessment Instrument, and Pain Assessment in Advanced Dementia. Results: Overall, scales that provided specific descriptions using the empirically displayed facial actions associated with pain yielded greater sensitivity, interjudge reliability, and validity as indices of pain. Discussion: Facial expression items on observational scales for assessing pain in the elderly benefit from adherence to empirically derived descriptions. Those using the scales should receive specific direction concerning cues to be assessed. Observational scales that provide descriptors that correspond to how people actually display facial expressions of pain perform better at differentiating intensities of pain.
TL;DR: The robust AFER system can be applied in many areas such as emotion science, clinical psychology and pain assessment it includes facial feature extraction and pattern recognition phases that discriminates among different facial expressions.
Abstract: This paper presents a computer vision system for automatic facial expression recognition (AFER). The robust AFER system can be applied in many areas such as emotion science, clinical psychology and pain assessment it includes facial feature extraction and pattern recognition phases that discriminates among different facial expressions. In feature extraction phase a combination between holistic and analytic approaches is presented to extract 83 facial expression features. Expression recognition is performed by using radial basis function based artificial neural network to recognize the six basic emotions (anger, fear, disgust, joy, surprise, sadness). The experimental results show that 96% recognition rate can be achieved when applying the proposed system on person-dependent database and 93.5% when applying on person-independent one.
01 Dec 2011
TL;DR: This paper proposes a framework capable of handling the potential imprecision of automatic landmarking techniques, thanks to a region approach, and was able to provide comparable results to existing 3D FER methods over the same protocol, while being fully automatic.
Abstract: In this paper, we address the problem of automatic 3D facial expression recognition. Automatic 3D Facial Expression Recognition techniques are generally limited in that they require manual, precise landmark points. Here, we propose a framework capable of handling the potential imprecision of automatic landmarking techniques, thanks to a region approach. After an automatic feature point localization step, we cluster the face into several regions, chosen for their importance into the facial expression process, according to the Facial Action Coding System (FACS) and anatomic considerations. Then, we match those regions to reference models representing the six prototypical expressions using Iterative Closest Points (ICP). ICP tends to compensate the imprecisions in the face clustering relative to landmarks localization. Resulting matching scores are concatenated into a descriptor for the probe model. Finally, we use a standard classification tool; in our experiments, we used Support Vector Machines (SVM), and were able to provide comparable results to existing 3D FER methods over the same protocol, while being fully automatic.
TL;DR: This article analyzed the facial behavior of 100 volunteers who video-recorded their own expressions while experiencing an episode of sexual excitement that concluded in an orgasm, and then posted their video clip on an Internet site.
Abstract: We analyzed the facial behavior of 100 volunteers who video-recorded their own expressions while experiencing an episode of sexual excitement that concluded in an orgasm, and then posted their video clip on an Internet site. Four distinct observational periods from the video clips were analyzed and coded by FACS (Facial Action Coding System, Ekman and Friesen 1978). We found nine combinations of muscular movements produced by at least 5% of the senders. These combinations were consistent with facial expressions of sexual excitement described by Masters and Johnson (Human sexual response, 1966), and they included the four muscular movements of the core expression of pain (Prkachin, Pain, 51, 297–306, 1992).
TL;DR: This study proposed two methods for recognizing 8 different facial expressions such as natural, happiness in three conditions, ange r, rage, gesturing 'a' like in apple word and pulling up the eyebrows based on Three-channels in Bi-polar configuration by SEMG.
Abstract: Problem statement: Facial expression recognition has been improved rec ently and it has become a significant issue in diagnostic and medica l fields, particularly in the areas of assistive technology and rehabilitation. Apart from their use fulness, there are some problems in their applications like peripheral conditions, lightening , contrast and quality of video and images. Approach: Facial Action Coding System (FACS) and some other methods based on images or videos were applied. This study proposed two methods for r ecognizing 8 different facial expressions such as natural (rest), happiness in three conditions, ange r, rage, gesturing 'a' like in apple word and gestu ring no by pulling up the eyebrows based on Three-channels in Bi-polar configuration by SEMG. Raw signals were processed in three main steps (filtrat ion, feature extraction and active features selecti on) sequentially. Processed data was fed into Support V ector Machine and Fuzzy C-Means classifiers for being classified into 8 facial expression groups. Results: 91.8 and 80.4% recognition ratio had been achieved for FCM and SVM respectively. Conclusion: The confirmed enough accuracy and power in this field of study and FCM showed its better abili ty and performance in comparison with SVM. It's expected that in near future, new approaches in the frequency bandwidth of each facial gesture will provide better results.
TL;DR: The findings suggest that the influence of facial expression is due to disruptive effects of angry expressions rather than facilitative effects of happy expressions, providing additional evidence that facial identity and facial expression are not processed completely independently.
Abstract: Research has shown that neutral faces are better recognized when they had been presented with happy rather than angry expressions at study, suggesting that emotional signals conveyed by facial expressions influenced the encoding of novel facial identities in memory. An alternative explanation, however, would be that the influence of facial expression resulted from differences in the visual features of the expressions employed. In this study, this possibility was tested by manipulating facial expression at study versus test. In line with earlier studies, we found that neutral faces were better recognized when they had been previously encountered with happy rather than angry expressions. On the other hand, when neutral faces were presented at study and participants were later asked to recognize happy or angry faces of the same individuals, no influence of facial expression was detected. As the two experimental conditions involved exactly the same amount of changes in the visual features of the stimuli between study and test, the results cannot be simply explained by differences in the visual properties of different facial expressions and may instead reside in their specific emotional meaning. The findings further suggest that the influence of facial expression is due to disruptive effects of angry expressions rather than facilitative effects of happy expressions. This study thus provides additional evidence that facial identity and facial expression are not processed completely independently.
09 Oct 2011
TL;DR: This paper proposes Fast-FACS, a computer vision aided system that improves speed and reliability of FACS coding, and is the first paper to predict onsets and offsets from peaks.
Abstract: FACS (Facial Action Coding System) coding is the state of the art in manual measurement of facial actions. FACS coding, however, is labor intensive and difficult to standardize. A goal of automated FACS coding is to eliminate the need for manual coding and realize automatic recognition and analysis of facial actions. Success of this effort depends in part on access to reliably coded corpora; however, manual FACS coding remains expensive and slow. This paper proposes Fast-FACS, a computer vision aided system that improves speed and reliability of FACS coding. Three are the main novelties of the system: (1) to the best of our knowledge, this is the first paper to predict onsets and offsets from peaks, (2) use Active Appearance Models for computer assisted FACS coding, (3) learn an optimal metric to predict onsets and offsets from peaks. The system was tested in the RU-FACS database, which consists of natural facial behavior during a twoperson interview. Fast-FACS reduced manual coding time by nearly 50% and demonstrated strong concurrent validity with manual FACS coding.
05 Oct 2011
TL;DR: The LIFEisGAME project is presented, a serious game that will help children with ASDs to recognize and express emotions through facial expressions and describes the technology behind the game, which focus on a character animation pipeline and sketching algorithm.
Abstract: This article presents the LIFEisGAME project, a serious game that will help children with ASDs to recognize and express emotions through facial expressions. The game design tackles one of the main experiential learning cycle of emotion recognition: recognize and mimic (game mode: build a face). We describe the technology behind the game, which focus on a character animation pipeline and a sketching algorithm. We detailed the facial expression analyzer that is used to calculate the score in the game. We also present a study that analyzes what type of characters children prefer when playing a game. Last, we present a pilot study we have performed with kids with ASD.
TL;DR: The potential value of videophone contact for providing access to visual nonverbal emotional communication through facial expressions of emotion during a videophone interaction between nurse and family caregiver is demonstrated.
Abstract: Purpose: The purpose of this study was to demonstrate the range of emotional expressions that can be displayed by nurse and family caregiver during a telehospice videophone consultation. We hypothesized that a nurse providing telehospice care via videophone would gain access to rich nonverbal emotional signals from the caregiver and communicate her own social presence to the caregiver, to potentially enhance the building of empathy between nurse and caregiver. Methodology: Videorecording of a case exemplar of videophone contact was obtained using the Beamer, a commercially available product that allows display of both caller and receiver on an available television through standard telephone lines. Nonverbal communication through facial expressions of emotion was quantified using detailed coding of facial movement and expression (facial action coding system). Results: In this study, we demonstrated the presence of visual nonverbal information in the form of facial expressions of emotion during a videophone interaction between nurse and family caregiver. Over the course of a typical after-hours telehospice call, a variety of facial expressions of emotion were displayed by both nurse and family caregiver. Expression of positive and negative emotions, as well as mixed emotions, was apparent. Through detailed analysis of this case of videophone interaction, we have demonstrated the potential value of videophone contact for providing access to visual nonverbal emotional communication.
TL;DR: In this article, the authors focus on the perception of emotions from naturally occurring, dynamic facial displays produced in social interactions and find that participants agree on five emotional factors: enjoyment, hostility, embarrassment, surprise, and sadness.
Abstract: Despite the fact that the facial expressions of emotions are naturally dynamic social signals, their communicative value has typically been studied using static photographs. In this paper, we focus on the perception of emotions from naturally occurring, dynamic facial displays produced in social interactions. In describing their impressions of 200 video records of spontaneous emotional expressions produced during a face-to-face emotional sharing task, observers were found to agree on five emotional factors: enjoyment, hostility, embarrassment, surprise, and sadness. FACS coding and sequential analysis using a pattern detection algorithm showed that recordings rated high on one emotional factor were characterized by unique sequences of facial actions coordinated with eye and/or gaze actions. Our results suggest that the dynamic unfolding of facial displays and their combination with additional nonverbal signals may play an important and still under-investigated role in emotion perception in face-to-face interactions.
01 Sep 2011
TL;DR: This study investigated the linkage between facial expression and mental workload in the performance of arithmetic tasks and showed that facial expression is a viable index for measuring mental workload.
Abstract: The measurement of mental workload is a commonly used and widely accepted means of assessing cognitively driven human performance tasks. The aim of this study was to investigate the linkage between facial expression and mental workload in the performance of arithmetic tasks. Eighteen participants were recruited and asked to perform various levels of arithmetic tasks. Classical subjective and physiological measures were used to track mental workload levels; these measures included NASA Task Load Index (NASA TLX), Subjective Workload Assessment Technique (SWAT), and electroencephalogram (EEG). In addition we utilize and propose a new measurement based on facial expressions, coded with the Facial Action Coding System. The results showed that facial expression is a viable index for measuring mental workload.
01 Jan 2011
TL;DR: This chapter discusses the meaning of facial behaviors in nonhuman primates, addressing specifically individual attributes of Social Attraction, facial expressivity, and the connection of facial behavior to emotion, and creates descriptive systems to characterize chimpanzees facial behavior.
Abstract: Independently, we created descriptive systems to characterize chimpanzee facial behavior, responding to a common need to have an objective, standardized coding system to ask questions about primate facial behaviors. Even with slightly different systems, we arrive at similar outcomes, with convergent conclusions about chimpanzee facial mobility. This convergence is a validation of the importance of the approach, and provides support for the future use of a facial action coding system for chimpanzees, ChimpFACS. Chimpanzees share many facial behaviors with those of humans. Therefore, processes and mechanisms that explain individual differences in facial activity can be compared with the use of a standardized systems such as ChimpFACS and FACS. In this chapter we describe our independent methodological approaches, comparing how we arrived at our facial coding categories. We present some Action Descriptors (ADs) from Gaspar’s initial studies, especially focusing on an ethogram of chimpanzee and bonobo facial behavior, based on studies conducted between 1997 and 2004 at three chimpanzee colonies (The Detroit Zoo; Cleveland Metroparks Zoo; and Burger’s Zoo) and two bonobo colonies (The Columbus Zoo and Aquarium; The Milwaukee County Zoo). We discuss the potential significance of arising issues, the minor qualitative species differences that were found, and the larger quantitative differences in particular facial behaviors observed between species, e.g., bonobos expressed more movements containing particular action units (Brow Lowerer, Lip Raiser, Lip Corner Puller) compared with chimpanzees. The substantial interindividual variation in facial behavior within each species was most striking. Considering individual differences and the impact of development, we highlight the flexibility in facial activity of chimpanzees. We discuss the meaning of facial behaviors in nonhuman primates, addressing specifically individual attributes of Social Attraction, facial expressivity, and the connection of facial behavior to emotion. We do not rule out the communicative function of facial behavior, in which case an individual’s properties of facial behavior are seen as influencing his or her social life, but provide strong arguments in support of the role of facial behavior in the expression of internal states.
29 Aug 2011
TL;DR: This work proposes SVM-based regression on AU feature space, and investigates person-independent estimation of 25 AUs that appear singly or in various combinations, and finds that fusion of 2D and 3D can boost the estimation performance, especially when modalities compensate for each other's shortcomings.
Abstract: The paradigm of Facial Action Coding System (FACS) offers a comprehensive solution for facial expression measurements. FACS defines atomic expression components called Action Units (AUs) and describes their strength on a five-point scale. Despite considerable progress in AU detection, the AU intensity estimation has not been much investigated. We propose SVM-based regression on AU feature space, and investigate person-independent estimation of 25 AUs that appear singly or in various combinations. Our method is novel in that we use regression for estimating intensities and comparatively evaluate the performances of 2D and 3D modalities. The proposed technique shows improvements over the state-of-the-art person-independent estimation, and that especially the 3D modality offers significant advantages for intensity coding. We have also found that fusion of 2D and 3D can boost the estimation performance, especially when modalities compensate for each other's shortcomings.
09 Jul 2011
TL;DR: This work investigated a new dynamic approach for detecting emotions in facial expressions in an artificial setting and in a driving context by analyzing the changes of an area defined by a number of dots that were arranged on participants' faces.
Abstract: Emotion detection provides a promising basis for designing future-oriented human centered design of Human-Machine Interfaces. Affective Computing can facilitate human-machine communication. Such adaptive advanced driver assistance systems (ADAS) which are dependent on the emotional state of the driver can be applied in cars. In contrast to the majority of former studies that only used static recognition methods, we investigated a new dynamic approach for detecting emotions in facial expressions in an artificial setting and in a driving context. By analyzing the changes of an area defined by a number of dots that were arranged on participants' faces, variables were extracted to classify the participants' emotions according to the Facial Action Coding System. The results of our novel way to categorize emotions lead to a discussion on additional applications and limitations that frames an attempted approach of emotion detection in cars. Implications for further research and applications are outlined.
TL;DR: The data suggest that upright facial gender and expression are encoded via distinct processes and that inversion does not just result in impaired sensitivity.
Abstract: Judgments of upright faces tend to be more rapid than judgments of inverted faces. This is consistent with encoding at different rates via discrepant mechanisms, or via a common mechanism that is more sensitive to upright input. However, to the best of our knowledge no previous study of facial coding speed has tried to equate sensitivity across the characteristics under investigation (eg emotional expression, facial gender, or facial orientation). Consequently we cannot tell whether different decision speeds result from mechanisms that accrue information at different rates, or because facial images can differ in the amount of information they make available. To address this, we examined temporal integration times, the times across which information is accrued toward a perceptual decision. We examined facial gender and emotional expressions. We first identified image pairs that could be differentiated on 80% of trials with protracted presentations (1 s). We then presented these images at a range of brief durations to determine how rapidly performance plateaued, which is indicative of integration time. For upright faces gender was associated with a protracted integration relative to expression judgments. This difference was eliminated by inversion, with both gender and expression judgments associated with a common, rapid, integration time. Overall, our data suggest that upright facial gender and expression are encoded via distinct processes and that inversion does not just result in impaired sensitivity. Rather, inversion caused gender judgments, which had been associated with a protracted integration, to become associated with a more rapid process.
21 Mar 2011
TL;DR: A unified stochastic framework based on the Dynamic Bayesian network (DBN) is proposed to explicitly represent the facial evolvements in different levels, their interactions and their observations and can improve the tracking (or recognition) performance in all three levels.
Abstract: The tracking of facial activities from video is an important and challenging problem. Nowadays, many computer vision techniques have been proposed to characterize the facial activities in the following three levels (from local to global): First, in the bottom level, the facial feature tracking focuses on detecting and tracking the prominent local landmarks surrounding facial components (e.g. mouth, eyebrow, etc); Second, the facial action units (AUs) characterize the specific behaviors of these local facial components (e.g. mouth open, eyebrow raiser, etc); Finally, facial expression, which is a representation of the subjects' emotion (e.g. Surprise, Happy, Anger, etc.), controls the global muscular movement of the whole face. Most of the existing methods focus on one or two levels of facial activities, and track (or recognize) them separately. In this paper, we propose to exploit the relationships among multi-level facial activities and track the facial activities in the three levels simultaneously. Specifically, we propose a unified stochastic framework based on the Dynamic Bayesian network (DBN) to explicitly represent the facial evolvements in different levels, their interactions and their observations. By modeling the relationships among the three level facial activities, the proposed method can improve the tracking (or recognition) performance in all three levels.
TL;DR: A real-time facial animation system in which speech drives mouth movements and facial expressions synchronously and the synthesized facial sequences reach a comparative convincing quality.
Abstract: In this paper, we present a real-time facial animation system in which speech drives mouth movements and facial expressions synchronously. Considering five basic emotions, a hierarchical structure with an upper layer of emotion classification is established. Based on the recognized emotion label, the under-layer classification at sub-phonemic level has been modelled on the relationship between acoustic features of frames and audio labels in phonemes. Using certain constraint, the predicted emotion labels of speech are adjusted to gain the facial expression labels which are combined with sub-phonemic labels. The combinations are mapped into facial action units (FAUs), and audio-visual synchronized animation with mouth movements and facial expressions is generated by morphing between FAUs. The experimental results demonstrate that the two-layer structure succeeds in both emotion and sub-phonemic classifications, and the synthesized facial sequences reach a comparative convincing quality.
01 Jan 2011
TL;DR: Within the context face expression classification using the facial action coding system (FACS), the problem of detecting facial action units (AUs) is addressed and it is shown that both these sources of error can be reduced by enhancing ECOC through the application of bootstrapping and class-separability weighting.
Abstract: Within the context face expression classification using the facial action coding system (FACS), we address the problem of detecting facial action units (AUs). The method adopted is to train a single Error-Correcting Output Code (ECOC) multiclass classifier to estimate the probabilities that each one of several commonly occurring AU groups is present in the probe image. Platt scaling is used to calibrate the ECOC outputs to probabilities and appropriate sums of these probabilities are taken to obtain a separate probability for each AU individually. Feature extraction is performed by generating a large number of local binary pattern (LBP) features and then selecting from these using fast correlation-based filtering (FCBF). The bias and variance properties of the classifier are measured and we show that both these sources of error can be reduced by enhancing ECOC through the application of bootstrapping and class-separability weighting.
15 May 2011
TL;DR: A novel algorithm for building real time 3D virtual facial animation of avatar by drawing facial expression features of real characters by sequentially estimates the emotional state of user's face expressions by Hidden Markov Model.
Abstract: In this paper, we provide a novel algorithm for building real time 3D virtual facial animation of avatar by drawing facial expression features of real characters. This algorithm first (automatically) labels the feature points of user's facial imagines drew from camera. Then sequentially estimates the emotional state of user's face expressions by Hidden Markov Model. At last construct the avatar's facial animation according to these emotional states. We explore the complex expression space by outputting synthetic facial animations blended by 6 standard facial expressions.
TL;DR: In this article, the authors derived culture-specific models of facial expressions using state-of-the-art 4D imaging (dynamics of 3D face shape and texture) combined with reverse correlation techniques.
Abstract: Six ‘universal’ facial expressions – ‘Happy,’ ‘Surprise,’ ‘Fear,’ ‘Disgust,’ ‘Anger,’ and ‘Sadness’ – are defined by specific, static patterns of facial muscle activation (Facial Action Coding System codes, FACS). However, systematic differences in facial expression recognition between Western Caucasians (WC) and East Asians (EA) question the notion of universality, raising a new question: How do different cultures represent facial expressions? Here, we derived culture-specific models of facial expressions using state-of-the-art 4D imaging (dynamics of 3D face shape and texture) combined with reverse correlation techniques. Specifically, we modelled 41 core Action Units (AUs, groups of facial muscles) from certified FACS coders and parameterized each using 6 temporal parameters (peak amplitude; peak latency; onset latency; offset latency; acceleration; deceleration). The 41 AUs and their parameters formed the basis of a pseudo-random generative model of expressive signals. On each trial, we pseudo-randomly selected parametric values for each AU, producing an expressive facial animation (see Figure S1 in Supplementary Material). Ten WC and 10 EA na & iuml;ve observers each categorized 9,600 such animations according to the 6 emotion categories listed above and rated the perceived intensity of the emotion (see Figure S1 in Supplementary Material). We then reverse correlated the dynamic properties of the AUs with the emotion categories they elicited, producing “dynamic classification models” (i.e., expected 4D face information) per emotion and observer. Analyses of the models reveal clear cultural contrasts in (a) the presence/absence of specific AUs predicting the reported EA miscategorizations and (b) radically different temporal dynamics of emotional expression whereby EA observers expect “smoother” emotional displays with lower acceleration and amplitude (see link in Supplementary Material for example videos). For the first time, we reveal cultural diversity in the dynamic signals representing each basic emotion, demonstrating that the complexities of emotion cannot adequately be reduced to a single set of static ‘universal’ signals.