Showing papers on "Facial Action Coding System published in 2006"

A 3D facial expression database for facial behavior research

Abstract: Traditionally, human facial expressions have been studied using either 2D static images or 2D video sequences. The 2D-based analysis is incapable of handing large pose variations. Although 3D modeling techniques have been extensively used for 3D face recognition and 3D face animation, barely any research on 3D facial expression recognition using 3D range data has been reported. A primary factor for preventing such research is the lack of a publicly available 3D facial expression database. In this paper, we present a newly developed 3D facial expression database, which includes both prototypical 3D facial expression shapes and 2D facial textures of 2,500 models from 100 subjects. This is the first attempt at making a 3D facial expression database available for the research community, with the ultimate goal of fostering the research on affective computing and increasing the general understanding of facial behavior and the fine 3D structure inherent in human facial expressions. The new database can be a valuable resource for algorithm assessment, comparison and evaluation.

Dynamics of facial expression: recognition of facial actions and their temporal segments from face profile image sequences

Abstract: Automatic analysis of human facial expression is a challenging problem with many applications. Most of the existing automated systems for facial expression analysis attempt to recognize a few prototypic emotional expressions, such as anger and happiness. Instead of representing another approach to machine analysis of prototypic facial expressions of emotion, the method presented in this paper attempts to handle a large range of human facial behavior by recognizing facial muscle actions that produce expressions. Virtually all of the existing vision systems for facial muscle action detection deal only with frontal-view face images and cannot handle temporal dynamics of facial actions. In this paper, we present a system for automatic recognition of facial action units (AUs) and their temporal models from long, profile-view face image sequences. We exploit particle filtering to track 15 facial points in an input face-profile sequence, and we introduce facial-action-dynamics recognition from continuous video input using temporal rules. The algorithm performs both automatic segmentation of an input video into facial expressions pictured and recognition of temporal segments (i.e., onset, apex, offset) of 27 AUs occurring alone or in a combination in the input face-profile video. A recognition rate of 87% is achieved.

Automatic Recognition of Facial Actions in Spontaneous Expressions

Abstract: Spontaneous facial expressions differ from posed expressions in both which muscles are moved, and in the dynamics of the movement. Advances in the field of automatic facial expression measurement will require development and assessment on spontaneous behavior. Here we present preliminary results on a task of facial action detection in spontaneous facial expressions. We employ a user independent fully automatic system for real time recognition of facial actions from the Facial Action Coding System (FACS). The system automatically detects frontal faces in the video stream and coded each frame with respect to 20 Action units. The approach applies machine learning methods such as support vector machines and AdaBoost, to texture-based image representations. The output margin for the learned classifiers predicts action unit intensity. Frame-by-frame intensity measurements will enable investigations into facial expression dynamics which were previously intractable by human coding.

A real-time automated system for the recognition of human facial expressions

Abstract: A fully automated, multistage system for real-time recognition of facial expression is presented. The system uses facial motion to characterize monochrome frontal views of facial expressions and is able to operate effectively in cluttered and dynamic scenes, recognizing the six emotions universally associated with unique facial expressions, namely happiness, sadness, disgust, surprise, fear, and anger. Faces are located using a spatial ratio template tracker algorithm. Optical flow of the face is subsequently determined using a real-time implementation of a robust gradient model. The expression recognition system then averages facial velocity information over identified regions of the face and cancels out rigid head motion by taking ratios of this averaged motion. The motion signatures produced are then classified using Support Vector Machines as either nonexpressive or as one of the six basic emotions. The completed system is demonstrated in two simple affective computing applications that respond in real-time to the facial expressions of the user, thereby providing the potential for improvements in the interaction between a computer user and technology.

Fully Automatic Facial Action Recognition in Spontaneous Behavior.

Abstract: We present results on a user independent fully automatic system for real time recognition of facial actions from the Facial Action Coding System (FACS). The system automatically detects frontal faces in the video stream and codes each frame with respect to 20 Action units. We present preliminary results on a task of facial action detection in spontaneous expressions during discourse. Support vector machines and AdaBoost classifiers are compared. For both classifiers, the output margin predicts action unit intensity.

Fully Automatic Facial Action Recognition in Spontaneous Behavior

Abstract: We present results on a user independent fully automatic system for real time recognition of facial actions from the Facial Action Coding System (FACS). The system automatically detects frontal faces in the video stream and codes each frame with respect to 20 Action units. We present preliminary results on a task of facial action detection in spontaneous expressions during discourse. Support vector machines and AdaBoost classifiers are compared. For both classifiers, the output margin predicts action unit intensity.

Genuine, suppressed and faked facial expressions of pain in children

Abstract: Children’s efforts to hide or exaggerate facial expressions of pain were compared to their genuine expressions using the cold pressor task. Fifty healthy 8- to 12-year-olds (25 boys, 25 girls) submerged their hands in cold and warm water and were instructed about what to show on their faces. Cold 10 °C water was used for the genuine and suppressed conditions and warm 30 °C water was used for the faked condition. Facial activity was videotaped and coded using the Facial Action Coding System to provide objective, detailed accounts of facial expressions in each condition, as well as during a baseline condition. Parents were subsequently asked to correctly identify each of the four conditions by viewing each video clip twice. Faked expressions of pain in children were found to show more frequent and more intense facial actions compared to their genuine pain expression, indicating that children had some understanding but were not fully successful in faking expressions of pain. Children’s suppressed expressions, however, showed no differences from baseline facial actions, indicating that they were able to successfully suppress their expressions of pain. Parents correctly identified the four conditions significantly more frequently than would be expected by chance. They were generally quite successful at detecting faked pain, but experienced difficulty differentiating among the other conditions. The results indicate that children are capable of controlling their facial expressions of pain when instructed to do so, but are better able to hide their pain than to fake it.

Design and Evaluation of Emotion-Display EDDIE

Abstract: This paper focuses on the development of EDDIE, a flexible low-cost emotion-display with 23 degrees of freedom. Actuators are assigned to particular action units of the facial action coding system (FACS). Emotion states represented by the circumplex model of affect are mapped to individual action units. Thereby, continuous, dynamic, and realistic emotion state transitions are achieved. EDDIE is largely developed and manufactured in a rapid-prototyping process. Miniature off-the-shelf mechatronics components are used providing high functionality at low-cost. Evaluations conducted in a user-study show that emotions can be recognized very well. Further experiments show that additional features adapted from animals have significant but small influence on the display of the human emotion `disgust'.

A model based method for automatic facial expression recognition

Abstract: Automatic facial expression recognition is a research topic with interesting applications in the field of human-computer interaction, psychology and product marketing. The classification accuracy for an automatic system which uses static images as input is however largely limited by the image quality, lighting conditions and the orientation of the depicted face. These problems can be partially overcome by using a holistic model based approach called the Active Appearance Model. A system will be described that can classify expressions from one of the emotional categories joy, anger, sadness, surprise, fear and disgust with remarkable accuracy. It is also able to detect smaller, local facial features based on minimal muscular movements described by the Facial Action Coding System (FACS). Finally, we show how the system can be used for expression analysis and synthesis.

Semantic 3D motion retargeting for facial animation

Abstract: We present a system for realistic facial animation that decomposes facial motion capture data into semantically meaningful motion channels based on the Facial Action Coding System. A captured performance is retargeted onto a morphable 3D face model based on a semantic correspondence between motion capture and 3D scan data. The resulting facial animation reveals a high level of realism by combining the high spatial resolution of a 3D scanner with the high temporal accuracy of motion capture data that accounts for subtle facial movements with sparse measurements.Such an animation system allows us to systematically investigate human perception of moving faces. It offers control over many aspects of the appearance of a dynamic face, while utilizing as much measured data as possible to avoid artistic biases. Using our animation system, we report results of an experiment that investigates the perceived naturalness of facial motion in a preference task. For expressions with small amounts of head motion, we find a benefit for our part-based generative animation system over an example-based approach that deforms the whole face at once.

Method and report assessing consumer reaction to a stimulus by matching eye position with facial coding

Abstract: A method of reporting consumer reaction to a stimulus and resultant report generated by (i) recording facial expressions and eye positions of a human subject while exposed to a stimulus throughout a time period, (ii) coding recorded facial expressions to emotions, and (iii) reporting recorded eye positions and coded emotions, along with an identification of the stimulus.

Intramuscular Electrical Stimulation of Facial Muscles in Humans and Chimpanzees: Duchenne Revisited and Extended

Abstract: The pioneering work of Duchenne (1862/1990) was replicated in humans using intramuscular electrical stimulation and extended to another species (Pan troglodytes: chimpanzees) to facilitate comparative facial expression research. Intramuscular electrical stimulation, in contrast to the original surface stimulation, offers the opportunity to activate individual muscles as opposed to groups of muscles. In humans, stimulation resulted in appearance changes in line with Facial Action Coding System (FACS) action units (AUs), and chimpanzee facial musculature displayed functional similarity to human facial musculature. The present results provide objective identification of the muscle substrate of human and chimpanzee facial expressions- data that will be useful in providing a common language to compare the units of human and chimpanzee facial expression.

Audio-visual emotion recognition in adult attachment interview

Abstract: Automatic multimodal recognition of spontaneous affective expressions is a largely unexplored and challenging problem. In this paper, we explore audio-visual emotion recognition in a realistic human conversation setting - Adult Attachment Interview (AAI). Based on the assumption that facial expression and vocal expression be at the same coarse affective states, positive and negative emotion sequences are labeled according to Facial Action Coding System Emotion Codes. Facial texture in visual channel and prosody in audio channel are integrated in the framework of Adaboost multi-stream hidden Markov model (AMHMM) in which Adaboost learning scheme is used to build component HMM fusion. Our approach is evaluated in the preliminary AAI spontaneous emotion recognition experiments.

Biologically vs. Logic Inspired Encoding of Facial Actions and Emotions in Video

Abstract: Automatic facial expression analysis is an important aspect of human machine interaction as the face is an important communicative medium. We use our face to signal interest, disagreement, intentions or mood through subtle facial motions and expressions. Work on automatic facial expression analysis can roughly be divided into the recognition of prototypic facial expressions such as the six basic emotional states and the recognition of atomic facial muscle actions (action units, AUs). Detection of AUs rather than emotions makes facial expression detection independent of culture-dependent interpretation, reduces the dimensionality of the problem and reduces the amount of training data required. Classic psychological studies suggest that humans consciously map AUs onto the basic emotion categories using a finite number of rules. On the other hand, recent studies suggest that humans recognize emotions unconsciously with a process that is perhaps best modeled by artificial neural networks (ANNs). This paper investigates these two claims. A comparison is made between detection of emotions directly from features vs. a two-step approach where we first detect AUs and use the AUs as input to either a rulebase or an ANN to recognize emotions. The results suggest that the two-step approach is possible with a small loss of accuracy and that biologically inspired classification techniques outperform those that approach the classification problem from a logical perspective, suggesting that biologically inspired classifiers are more suitable for computer-based analysis of facial behavior than logic inspired methods

Beihang University Facial Expression Database and Multiple Facial Expression Recognition

Abstract: Over the last decade, significant effort has been made in developing methods of facial expression analysis. Because of the limited data sets of facial expressions, comprehensive emotional facial expression recognition that is essential in the areas such as affective computing and human-machine interaction has not been well studied. This paper provides the Beihang University Facial Expression Database, which includes 25 facial expressions (18 pure facial expressions, 3 mixed facial expressions and 4 complex facial expressions) and provides data of many unique emotional facial expressions not appeared in other face databases. The questionnaire investigation of videos of 18 pure facial expressions shows the effectiveness of the data. Facial expression recognition experiment on 9 pure facial expressions demonstrates the feasibility of multiple facial expression recognition.

To Capture a Face: A Novel Technique for the Analysis and Quantification of Facial Expressions in American Sign Language

Abstract: OVER THE PAST TWO DECADES research on American Sign Language (ASL) has shown that, although the hands and arms articulate most of the content words (nouns, verbs, and adjectives), a large part of the grammar of ASL is expressed nonmanually. The hands and arms do play central grammatical roles, but, in addition, movements of the head, torso, and face are used to express certain aspects of ASL syntax such as functional categories, syntactic agreement, syntactic features, complementizers, and discourse markers. Since the pioneering work on nonmanuals (i.e., parts of ASL not expressed through the arms and hands) by Liddell (1986), BakerShenk (1983, 1986), and Baker-Shenk and Padden (1978), research has increasingly focused upon facial expressions in ASL and their syntactic significance (Neidle et al. 2000; Aarons 1994; Aarons et al. 1992; Baker-Shenk 1985). It is now well established that ASL requires the use of the face not only to express emotions but also to mark several different kinds of questions: wh-questions (questions using who, what, where, when, or why), yes/no (y/n) questions (Neidle et al. 1997; Petronio and LilloMartin 1997; Baker-Shenk 1983, 1986), and rhetorical questions (Hoza et al. 1997), as well as many other syntactic and adverbial constructs (Anderson and Reilly 1998; Shepard-Kegl, Neidle, and Kegl 1995; Reilly, Mclntire, and Bellugi 1990, Wilbur and Schick 1987; Coulter 1978, 1983; Liddell 1978; Baker-Shenk and Padden 1978; Friedman 1974). In addition to these grammatical facial expressions and the full range of emotional facial expressions, which Ekman and Friesen (1975, 1978) contend are universal, both spoken and signed languages use facial expressions such as quizzical, doubtful, and scornful, which can be categorized as nonemotional and nongrammatical (NENG). These NENG facial expressions are commonly used during social interaction, without carrying emotional or grammatical meaning. We include them here in order to study a class of facial expressions that exhibits neither the automatic qualities of emotional expressions (Whalen et al. 1998) nor the structured and grammar-specific characteristics of ASL syntax described earlier. ASL is a language of dynamic visuo-spatial changes that are often difficult to describe but nonetheless essential for our understanding of the language (Emmorey 1995). Grossman (2001) and Grossman and Kegl (submitted) emphasize the need to use dynamic facial expressions (video clips), as opposed to the commonly used static images (photographs), in order to obtain a more realistic assessment of the way in which hearing and deaf people recognize and categorize facial expressions. However, only a few detailed analyses of the production of dynamic emotional and grammatical facial expressions are available in ASL. Baker-Shenk (1983) and Bahan (1996) have dealt extensively with the development of dynamic ASL facial expressions and their link to the manual components of ASL sentences. They have detailed their development and noted their onset, apex (maximal expression), duration of apex, and offset. Baker-Shenk and Bahan observed these dynamic changes in numerous ASL sentences and looked for common denominators among samples of the same type of expression (e.g., wh-question, y/n question) to determine how specific expression types differ from each other. Baker-Shenk used Ekman and Friesen's Facial Action Coding System (FACS, Ekman and Friesen 1975, 1978) to analyze ASL question faces. In this system, each muscle group of the face is assigned an action unit (AU) number, and the specific combination of AUs defines a given facial expression. Using this technique, Baker-Shenk produced detailed descriptions of several different types of ASL question faces. This approach, however, encounters some difficulties in describing dynamic features or gestures such as head tilts. For example, when looking at y/n questions, Baker-Shenk found that six samples out of sixteen had a downward head tilt, nine a forward tilt, and three had both. …

A system for converting robot 'emotion' into facial expressions

Abstract: This paper presents a method that enable a domestic robot to show emotions with its facial expressions. The previous methods using built-in facial expressions were able to show only scanty face. To express faces showing various emotion, (e.g. mixed emotions and different strengths of emotions) more facial expressions are needed. We have therefore developed a system that converts emotions into robot's facial expressions automatically. They are created from emotion parameters, which represent its emotions. We show that the system can generate facial expressions reasonably

EDDIE - An Emotion Display with Dynamic Intuitive Expressions

Abstract: This paper focuses on the development of EDDIE, a flexible low-cost emotion-display with 23 degrees of freedom. Actuators are assigned to particular action units of the facial action coding system (FACS). Emotion states represented by the circumplex model of affect are mapped to individual action units. Thereby, continuous, dynamic, and realistic emotion state transitions are achieved. EDDIE is largely developed and manufactured in a rapid-prototyping process. Miniature off-the-shelf mechatronics are used providing high functionality while extremely low-cost. Evaluations are conducted based on a user-study.

Development of a Facial Expression Imitation System

Abstract: In the last decade, face analysis, e.g. face recognition, face detection, face tracking and facial expression recognition, is a very lively and expanding research field. As computer animated agents and robots bring a social dimension to human computer interaction, interest in this research field is increasing rapidly. In this paper, we introduce an artificial facial expression mimic system which can recognize facial expressions of human and also imitate the recognized facial expressions. We propose a classifier that is based on weak classifiers obtained by using modified rectangular features to recognize human facial expression in real-time. Next, we introduce our robot that is manipulated by a distributed control algorithm and that can make artificial facial expressions. Finally, experimental results of facial expression recognition and facial expression generation are shown for the validity of our artificial facial expression imitator.

Facial emotion recognition by adaptive processing of tree structures

Abstract: We present an emotion recognition system based on a probabilistic approach to adaptive processing of Facial Emotion Tree Structures (FETS). FETS are made up of localized Gabor features related to the facial components according to the Facial Action Coding System. The proposed model is an extension of the probabilistic based recursive neural network model applying in face recognition by Cho and Wong [1]. The robustness of the model in an emotion recognition system is evaluated by testing with known and unknown subjects with different emotions. The experiment results shows that the proposed model significantly improved the recognition rate in terms of generalization.

Local versus global segmentation for facial expression recognition

Abstract: We examined the open issue of whether FACS action units (AUs) can be recognized more accurately by classifying local regions around the eyes, brows, and mouth compared to analyzing the face as a whole. Our empirical results showed that, contrary to our intuition, local expression analysis showed no consistent improvement in recognition accuracy. Moreover, global analysis outperformed local analysis on certain AUs of the eye and brow regions. We attributed this unexpected result partly to high correlations between different AUs in the Cohn-Kanade expression database. This underlines the importance of establishing a large, publicly available AU database with singly-occurring AUs to facilitate future research.

Facial Tracking for an Emotion-Diagnosis Robot to Support E-Learning

Abstract: There have been a lot of researches on the detection/estimation of human emotions from facial expressions. However, most of them have extracted facial features for some specific emotions from the still pictures of artificial actions or performances. This paper describes facial tracking for e-learning support robot which can estimate a emotion of e-learning user from his/her facial expression in real-time; (1) the criteria of the facial expression to classify the eight emotions was obtained by the time sequential subjective evaluation on the emotions as well as the time sequential analysis of a facial expression by image processing. (2) The coincidence ratio between the discriminated emotions based upon the criteria of emotion diagnosis and the time sequential subjective evaluation on emotions for ten e-learning subjects was 69%. (3) Then, the possibility of the real time emotion diagnosis robot to support e-learning was confirmed by the facial image processing at the 15 frame/sec. rate as well as the simple emotion diagnosis algorithm based upon the Mahalonobis distance

Control of avatar’s facial expression using fundamental frequency in multi-user voice chat system

Abstract: An automatic facial expression control algorithm of CG avatar based on the fundamental frequency of the user’s utterance is proposed, in order to facilitate the multi-party casual chat in a multi-user virtual-space voice chat system The proposed method utilizes the common tendency of the voice fundamental frequency that reflects the emotional activity, especially the strength of the delight This study simplified the facial expression control problem by limiting the expression in the strength of the delight, because it appears the expression of the delight is the most important to facilitate the casual chat The problem of using the fundamental frequency is that fundamental frequency varies with intonation as well as emotion; hence the use of the raw fundamental frequency changes the expression of the avatar passionately Therefore, Emotional Point by emotional Activity (EPa) was defined as the moving-average of the normalized fundamental frequency, to suppress the influence of the intonation The strength of the delight of the avatar facial expression was linearly controlled using EPa, based on the Facial Action Coding System (FACS) The duration of the moving average was chosen as five seconds experimentally However, the moving average delays the avatar behavior, and the delay is more serious especially in the response utterance Therefore, to compensate the delay of the response, the Emotional Point by Response (EPr), was defined using the initial voice volume of the response utterance EPr was calculated for only the response utterance, which means the utterance just after another user’s utterance The ratio of EPr to EPa was decided experimentally as one to one The proposed automatic avatar facial expression control algorithm was implemented on the previously developed virtual-space multi-user voice chat system The subjective evaluation was performed in ten subjects The each subject in separate room was required to chat with an experimental partner using the system for four minutes and to answer four questions using Likert scale Throughout the experiments, the subjects reported better impression of the automatic control of facial expression according to the utterances The facial control using both EPa and EPr demonstrated better performance in terms of naturalness, favorability, familiarity and interactivity, compared to the fixed facial expression, the automatic control using EPa alone and the EPr alone conditions

Visualization of music impression in facial expression to represent emotion

Abstract: In this paper, we propose a visualization method of music impression in facial expression to represent emotion. We apply facial expression to represent the complicated and mixed emotions. This method can generate facial expression corresponding to impressions of music data by measurement of relationship between each basic emotion for facial expression and impressions extracted from music data. The feature of this method is a realization of an integration between music data and the facial expression that convey various emotions effectively. One of the important issues is a realization of communication media corresponding to human Kansei with less difficulty for a user. Facial expression can express complicated emotions with which various emotions are mixed. Assuming that an integration between existing mediadata and facial expression is possible, visualization corresponding to human Kansei with less difficulty realized for a user.

Facial expression and piano performance

Abstract: Previous studies (Caterina et al. 2004) carried out on the body and facial expressions of pianists during thei r performances, have shown that there is a specific rela tion between non verbal expressions and music structure. We tried to compare the value of single facial acti on unit movements‐ such as eyebrow raising and frowning ‐ f ound in our observations, with the concurring musical st ructure. Videorecording of a group of professional pianists performing different repertoires were observed. Two in dependent expert judges analysed the facial expressio ns using Ekman and Friesen FACS. Facial action unit movement s were examined together with main musical features, in order to better understand the meaning of the most frequent facial signals.

Recognizing Facial Expressions in Videos Using a Facial Action Analysis-Synthesis Scheme

Abstract: In this paper, we propose a novel approach for facial expression analysis and recognition. The proposed approach relies on tracked facial actions provided by an appearance-based 3D face tracker. For each universal expression, a dynamical model for facial actions given by an auto-regressive process is learned from training data. We classify a given image in an unseen video into one of the universal facial expression categories using an analysis-synthesis scheme. This scheme uses all models and select the one that provides the most consistent synthesized spatio-temporal facial actions. The dynamical models can be utilized in the tasks of synthesis and prediction. Experiments using unseen videos demonstrated the effectiveness of the developed method.

Facial Expression of Emotion in Human Frontal and Temporal Lobe Epileptic Seizures

Abstract: While there is evidence of an impaired perception of emotion in verbal and facial expression in epileptic patients with unilateral focal resection of frontal, temporal, or parieto-occipital cortex,1 there is up to now a lack of research on the encoding aspects of facial expressions during seizure. In this study the video recordings of 146 seizures of 20 patients with temporal lobe epilepsy (12 females and 8 males) and 9 patients with frontal lobe epilepsy (2 females and 7 males) were analyzed using the Facial Action Coding System (FACS). Seizures were recorded in a standard hospital setting. Each video was paired with an EEG recording in order to ascertain the relationship between the clinical manifestations and the ictal discharge. The hypothesis was that, during the seizure, in addition to well-established facial expressions such as the “blank stare” during a “petit mal” absence with impaired consciousness, and the grimaces (unilateral or bilateral jerking and tonic contractions of the facial musculature), the facial displays can show a coherent pattern that is comparable to the facial expressions of emotions as they appear in normal subjects. Coherent patterns of facial expressions of emotions during the ictal event (see FIG. 1) can emerge as a result of the activa-

Facial expressions recognition in a single static as well as dynamic facial images using tracking and probabilistic neural networks

Abstract: An efficient, global and local image-processing based extraction and tracking of intransient facial features and automatic recognition of facial expressions from both static and dynamic 2D image/video sequences is presented. Expression classification is based on Facial Action Coding System (FACS) a lower and upper face action units (AUs), and discrimination is performed using Probabilistic Neural Networks (PNN) and a Rule-Based system. For the upper face detection and tracking, we use systems based on a novel two-step active contour tracking system while for the upper face, cross-correlation based tracking system is used to detect and track of Facial Feature Points (FFPs). Extracted FFPs are used to extract some geometric features to form a feature vector which is used to classify input image or image sequences into AUs and basic emotions. Experimental results show robust detection and tracking and reasonable classification where an average recognition rate is 96.11% for six basic emotions in facial image sequences and 94% for five basic emotions in static face images.