Showing papers on "Facial Action Coding System published in 2009"

Development of a FACS-verified set of basic and self-conscious emotion expressions.

Abstract: In 2 studies, the authors developed and validated of a new set of standardized emotion expressions, which they referred to as the University of California, Davis, Set of Emotion Expressions (UCDSEE) The precise components of each expression were verified using the Facial Action Coding System (FACS) The UCDSEE is the first FACS-verified set to include the three "self-conscious" emotions known to have recognizable expressions (embarrassment, pride, and shame), as well as the 6 previously established "basic" emotions (anger, disgust, fear, happiness, sadness, and surprise), all posed by the same 4 expressers (African and White males and females) This new set has numerous potential applications in future research on emotion and related topics

A framework for automated measurement of the intensity of non-posed Facial Action Units

Abstract: This paper presents a framework to automatically measure the intensity of naturally occurring facial actions. Naturalistic expressions are non-posed spontaneous actions. The facial action coding system (FACS) is the gold standard technique for describing facial expressions, which are parsed as comprehensive, nonoverlapping action units (Aus). AUs have intensities ranging from absent to maximal on a six-point metric (i.e., 0 to 5). Despite the efforts in recognizing the presence of non-posed action units, measuring their intensity has not been studied comprehensively. In this paper, we develop a framework to measure the intensity of AU12 (lip corner puller) and AU6 (cheek raising) in videos captured from infant-mother live face-to-face communications. The AU12 and AU6 are the most challenging case of infant's expressions (e.g., low facial texture in infant's face). One of the problems in facial image analysis is the large dimensionality of the visual data. Our approach for solving this problem is to utilize the spectral regression technique to project high dimensionality facial images into a low dimensionality space. Represented facial images in the low dimensional space are utilized to train support vector machine classifiers to predict the intensity of action units. Analysis of 18 minutes of captured video of non-posed facial expressions of several infants and mothers shows significant agreement between a human FACS coder and our approach, which makes it an efficient approach for automated measurement of the intensity of non-posed facial action units.

Automated Facial Expression Recognition System

Abstract: Heightened concerns about the treatment of individuals during interviews and interrogations have stimulated efforts to develop “non-intrusive” technologies for rapidly assessing the credibility of statements by individuals in a variety of sensitive environments. Methods or processes that have the potential to precisely focus investigative resources will advance operational excellence and improve investigative capabilities. Facial expressions have the ability to communicate emotion and regulate interpersonal behavior. Over the past 30 years, scientists have developed human-observer based methods that can be used to classify and correlate facial expressions with human emotion. However, these methods have proven to be labor intensive, qualitative, and difficult to standardize. The Facial Action Coding System (FACS) developed by Paul Ekman and Wallace V. Friesen is the most widely used and validated method for measuring and describing facial behaviors. The Automated Facial Expression Recognition System (AFERS) automates the manual practice of FACS, leveraging the research and technology behind the CMU/PITT Automated Facial Image Analysis System (AFA) system developed by Dr. Jeffery Cohn and his colleagues at the Robotics Institute of Carnegie Mellon University. This portable, near real-time system will detect the seven universal expressions of emotion (figure 1), providing investigators with indicators of the presence of deception during the interview process. In addition, the system will include features such as full video support, snapshot generation, and case management utilities, enabling users to re-evaluate interviews in detail at a later date.

The face is not an empty canvas: how facial expressions interact with facial appearance

Abstract: Faces are not simply blank canvases upon which facial expressions write their emotional messages. In fact, facial appearance and facial movement are both important social signalling systems in their own right. We here provide multiple lines of evidence for the notion that the social signals derived from facial appearance on the one hand and facial movement on the other interact in a complex manner, sometimes reinforcing and sometimes contradicting one another. Faces provide information on who a person is. Sex, age, ethnicity, personality and other characteristics that can define a person and the social group the person belongs to can all be derived from the face alone. The present article argues that faces interact with the perception of emotion expressions because this information informs a decoder's expectations regarding an expresser's probable emotional reactions. Facial appearance also interacts more directly with the interpretation of facial movement because some of the features that are used to derive personality or sex information are also features that closely resemble certain emotional expressions, thereby enhancing or diluting the perceived strength of particular expressions.

Automated Facial Action Coding System

Abstract: An automatic facial action coding system and method can include processing an image to identify a face in the image, to detect and align one or more facial features shown in the image, and to define one or more windows on the image. One or more distributions of pixels and color intensities can be quantified in each of the one or more windows to derive one or more two-dimensional intensity distributions of one or more colors within the window. The one or more two-dimensional intensity distributions can be processed to select image features appearing in the one or more windows and to classify one or more predefined facial actions on the face in the image. A facial action code score that includes a value indicating a relative amount of the predefined facial action occurring in the face in the image can be determined for the face in the image for each of the one or more predefined facial actions.

An approach for automatically measuring facial activity in depressed subjects

Abstract: This paper is motivated by Ellgring's work in non-verbal communication in depression to measure and compare the levels of facial activity, before and after treatment, of endogenous and neurotic depressives. Similar to that work, we loosely associate the measurements with Action Units (AU) groups from the Facial Action Coding System (FACS). However, we use the neologism Region Units (RU) to describe regions of the face that encapsulate AUs. In contrast to Ell-gring's approach, we automatically generate the measurements and provide both prototypical expression recognition and RU-specific activity measurements. Latency between expressions is also measured and the system is conducive to comparison across groups and individual subjects. By using Active Appearance Models (AAM) to locate the fiduciary facial points, and MultiBoost to classify prototypical expressions and the RUs, we can provide a simple, objective, flexible and cost-effective means of automatically measuring facial activity.

Symmetry, probability, and recognition in face space

Abstract: The essential midline symmetry of human faces is shown to play a key role in facial coding and recognition. This also has deep and important connections with recent explorations of the organization of primate cortex, as well as human psychophysical experiments. Evidence is presented that the dimension of face recognition space for human faces is dramatically lower than previous estimates. One result of the present development is the construction of a probability distribution in face space that produces an interesting and realistic range of (synthetic) faces. Another is a recognition algorithm that by reasonable criteria is nearly 100% accurate.

Mixed signals: combining linguistic and affective functions of eyebrows in questions in sign language of the Netherlands.

Abstract: The eyebrows are used as conversational signals in face-to-face spoken interaction (Ekman, 1979) In Sign Language of the Netherlands (NGT), the eyebrows are typically furrowed in content questions, and raised in polar questions (Coerts, 1992) On the other hand, these eyebrow positions are also associated with anger and surprise, respectively, in general human communication (Ekman, 1993) This overlap in the functional load of the eyebrow positions results in a potential conflict for NGT signers when combining these functions simultaneously In order to investigate the effect of the simultaneous realization of both functions on the eyebrow position we elicited instances of both question types with neutral affect and with various affective states The data were coded using the Facial Action Coding System (FACS: Ekman, Friesen, & Hager, 2002) for type of brow movement as well as for intensity FACS allows for the coding of muscle groups, which are termed Action Units (AUs) and which produce facial appearance changes The results show that linguistic and affective functions of eyebrows may influence each other in NGT That is, in surprised polar questions and angry content question a phonetic enhancement takes place of raising and furrowing, respectively In the items with contrasting eyebrow movements, the grammatical and affective AUs are either blended (occur simultaneously) or they are realized sequentially Interestingly, the absence of eyebrow raising (marked by AU 1+2) in angry polar questions, and the presence of eyebrow furrowing (realized by AU 4) in surprised content questions suggests that in general AU 4 may be phonetically stronger than AU 1 and AU 2, independent of its linguistic or affective function

Allometry of facial mobility in anthropoid primates: implications for the evolution of facial expression.

Abstract: Body size may be an important factor influencing the evolution of facial expression in anthropoid primates due to allometric constraints on the perception of facial movements. Given this hypothesis, I tested the prediction that observed facial mobility is positively correlated with body size in a comparative sample of nonhuman anthropoids. Facial mobility, or the variety of facial movements a species can produce, was estimated using a novel application of the Facial Action Coding System (FACS). I used FACS to estimate facial mobility in 12 nonhuman anthropoid species, based on video recordings of facial activity in zoo animals. Body mass data were taken from the literature. I used phylogenetic generalized least squares (PGLS) to perform a multiple regression analysis with facial mobility as the dependent variable and two independent variables: log body mass and dummy-coded infraorder. Together, body mass and infraorder explain 92% of the variance in facial mobility. However, the partial effect of body mass is much stronger than for infraorder. The results of my study suggest that allometry is an important constraint on the evolution of facial mobility, which may limit the complexity of facial expression in smaller species. More work is needed to clarify the perceptual bases of this allometric pattern.

Machine Learning Systems for Detecting Driver Drowsiness

Abstract: Drowsy driver detection is one of the potential applications of intelligent vehicle systems. Previous approaches to drowsiness detection primarily make pre-assumptions about the relevant behavior, focusing on blink rate, eye closure, and yawning. Here we employ machine learning to datamine actual human behavior during drowsiness episodes. Automatic classifiers for 30 facial actions from the facial action coding system were developed using machine learning on a separate database of spontaneous expressions. These facial actions include blinking and yawn motions, as well as a number of other facial movements. These measures were passed to learning-based classifiers such as Adaboost and multinomial ridge regression. Head motion information was collected through automatic eye tracking and an accelerometer. The system was able to predict sleep and crash episodes on a simulator with 98% accuracy across subjects. It is the highest prediction rate reported to date for detecting drowsiness. Moreover, the analysis revealed new information about human facial behavior for drowsy drivers.

Pleasure-arousal-dominance driven facial expression simulation

Abstract: Expressing and recognizing affective states with respect to facial expressions is an important aspect in perceiving virtual humans as more natural and believable. Based on the results of an empirical study a system for simulating emotional facial expressions for a virtual human has been evolved. This system consists of two parts: (1) a control architecture for simulating emotional facial expressions with respect to Pleasure, Arousal, and Dominance (PAD) values, (2) an expressive output component for animating the virtual human's facial muscle actions called Action Units (AUs), modeled following the Facial Action Coding System (FACS). A large face repertoire of about 6000 faces arranged in PAD-space with respect to two dominance values (dominant vs. submissive) is obtained as a result of the empirical study. Using the face repertoire an approach towards realizing facial mimicry for a virtual human based on backward mapping AUs displaying an emotional facial expression on PAD-values is outlined. A preliminary evaluation of this first approach is realized with AUs corresponding to the basic emotions Happy and Angry.

Perceptually Valid Facial Expressions for Character-Based Applications

Abstract: This paper addresses the problem of creating facial expression of mixed emotions in a perceptually valid way. The research has been done in the context of a “game-like” health and education applications aimed at studying social competency and facial expression awareness in autistic children as well as native language learning, but the results can be applied to many other applications such as games with need for dynamic facial expressions or tools for automating the creation of facial animations. Most existing methods for creating facial expressions of mixed emotions use operations like averaging to create the combined effect of two universal emotions. Such methods may be mathematically justifiable but are not necessarily valid from a perceptual point of view. The research reported here starts by user experiments aiming at understanding how people combine facial actions to express mixed emotions, and how the viewers perceive a set of facial actions in terms of underlying emotions. Using the results of these experiments and a three-dimensional emotion model, we associate facial actions to dimensions and regions in the emotion space, and create a facial expression based on the location of the mixed emotion in the three-dimensional space. We call these regionalized facial actions “facial expression units.”

Appraising emotional events during a real-time interactive game

Abstract: Interactive virtual agents now commonly display facial expressions of emotions. Most of these expressions are triggered using a finite set of labeled emotions, e.g. the six basic emotions (a single expression being assigned to each emotion). However, theories of emotions suggest that emotion is a componential evaluation process, during which sequential facial expressions reflect various information about the ongoing evaluation. This paper presents an implementation of an event based facial animation system based on Scherer's Component Process Theory of emotions. Our application generates appraisal events during a real-time game interaction with a user. The MARC virtual character is used to displays sequential facial expressions reflecting the evaluation process of these game events in real-time.

Video based detection of driver fatigue

Abstract: This thesis addresses the problem of drowsy driver detection using computer vision techniques applied to the human face. Specifically we explore the possibility of discriminating drowsy from alert video segments using facial expressions automatically extracted from video. Several approaches were previously proposed for the detection and prediction of drowsiness. There has recently been increasing interest in computer vision approaches as it is a potentially promising approach due to its non-invasive nature for detecting drowsiness. Previous studies with vision based approaches detect driver drowsiness primarily by making pre-assumptions about the relevant behavior, focusing on blink rate, eye closure, and yawning. Here we employ machine learning to explore, understand and exploit actual human behavior during drowsiness episodes. We have collected two datasets including facial and head movement measures. Head motion is collected through an accelerometer for the first dataset (UYAN-1) and an automatic video based head pose detector for the second dataset (UYAN-2). We use outputs of the automatic classifiers of the facial action coding system (FACS) for detecting drowsiness. These facial actions include blinking and yawn motions, as well as a number of other facial movements. These measures are passed to a learning-based classifier based on multinomial logistic regression. In UYAN-1 the system is able to predict sleep and crash episodes during a driving computer game with 0.98 performance area under the receiver operator characteristic curve for across subjects tests. This is the highest prediction rate reported to date for detecting real drowsiness. Moreover, the analysis reveals new information about human facial behavior during drowsy driving. In UYAN-2 fine discrimination of drowsy states are also explored on a separate dataset. The degree to which individual facial action units can predict the difference between moderately drowsy to acutely drowsy is studied. Signal processing techniques and machine learning methods are employed to build a person independent acute drowsiness detection system. Temporal dynamics are captured using a bank of temporal filters. Individual action unit predictive power is explored with an MLR based classifier. Best performing five action units have been determined for a person independent system. The system is able to obtain 0.96 performance of area under the receiver operator characteristic curve for a more challenging dataset with the combined features of the best performing 5 action units. Moreover the analysis reveals new markers for different levels of drowsiness.

Automatic detection of facial actions from 3D data

Abstract: We address the person-independent recognition problem of facial expressions using static 3D face data. The novel approach to the facial expression recognition uses non-rigid registration of surface curvature features. 3D face data is cast onto 2D feature images, which are then subjected to elastic deformations in their parametric space. Each Action Unit (AU) detector is trained over its respective influence domain on the face. The registration task is incorporated in the multiresolution elastic deformation scheme, which yields adequate registration accuracy for mild pose variations. The algorithm is fully automatic and is free of the burden of first localizing anatomical facial points. The algorithm was tested on 22 facial action units of Facial Action Coding System. Promising results obtained indicate that we have an operative device for facial action unit detection, and an intermediate step to infer emotional or mental states. Moreover, experiments conducted with low intensity AU12 - Lip Corner Puller points to the potential of 3D data and the proposed method in subtle expression detection.

Facial identity and facial expression matching in 5–12‐year‐old children and adults

Abstract: Facial identity and facial expression matching tasks were completed by 5–12-year-old children and adults using stimuli extracted from the same set of normalized faces. Configural and feature processing were examined using speed and accuracy of responding and facial feature selection, respectively. Facial identity matching was slower than face expression matching for all age groups. Large age effects were found on both speed and accuracy of responding and feature use in both identity and expression matching tasks. Eye region preference was found on the facial identity task and mouth region preference on the facial expression task. Use of mouth region information for facial expression matching increased with age, whereas use of eye region information for facial identity matching peaked early. The feature use information suggests that the specific use of primary facial features to arrive at identity and emotion matching judgments matures across middle childhood. Copyright © 2009 John Wiley & Sons, Ltd.

Get the FACS fast: Automated FACS face analysis benefits from the addition of velocity

Abstract: Much progress has been made in automated facial image analysis, yet current approaches still lag behind what is possible using manual labeling of facial actions. While many factors may contribute, a key one may be the limited attention to dynamics of facial action. Most approaches classify frames in terms of either displacement from a neutral, mean face or, less frequently, displacement between successive frames (i.e. velocity). In the current paper, we evaluated the hypothesis that attention to dynamics can boost recognition rates. Using the well-known Cohn-Kanade database and support vector machines, adding velocity and acceleration decreased the number of incorrectly classified results by 14.2% and 11.2%, respectively. Average classification accuracy for the displacement and velocity classifier system across all classifiers was 90.2%. Findings were replicated using linear discriminant analysis, and found a mean decrease of 16.4% in incorrect classifications across classifiers. These findings suggest that information about the dynamics of a movement, that is, the velocity and to a lesser extent the acceleration of a change, can helpfully inform classification of facial expressions.

Weighted Decoding ECOC for Facial Action Unit Classification

Abstract: There are two approaches to automating the task of facial expression recognition, the first concentrating on what meaning is conveyed by facial expression and the second on categorising deformation and motion into visual classes. The latter approach has the advantage that the interpretation of facial expression is decoupled from individual actions as in FACS (Facial Action Coding System). In this chapter, upper face action units (aus) are classified using an ensemble of MLP base classifiers with feature ranking based on PCA components. When posed as a multi-class problem using Error-Correcting-Output-Coding (ECOC), experimental results on Cohn-Kanade database demonstrate that error rates comparable to two-class problems (one-versus-rest) may be obtained. The ECOC coding and decoding strategies are discussed in detail, and a novel weighted decoding approach is shown to outperform conventional ECOC decoding. Furthermore, base classifiers are tuned using the ensemble Out-of-Bootstrap estimate, for which purpose, ECOC decoding is modified. The error rates obtained for six upper face aus around the eyes are believed to be among the best for this database.

Comparing a novel model based on the transferable belief model with humans during the recognition of partially occluded facial expressions

Abstract: Humans recognize basic facial expressions effortlessly. Yet, despite a considerable amount of research, this task remains elusive for computer vision systems. Here, we compared the behavior of one of the best computer models of facial expression recognition (Z. Hammal, L. Couvreur, A. Caplier, & M. Rombaut, 2007) with the behavior of human observers during the M. Smith, G. Cottrell, F. Gosselin, and P. G. Schyns (2005) facial expression recognition task performed on stimuli randomly sampled using Gaussian apertures. The model--which we had to significantly modify in order to give the ability to deal with partially occluded stimuli--classifies the six basic facial expressions (Happiness, Fear, Sadness, Surprise, Anger, and Disgust) plus Neutral from static images based on the permanent facial feature deformations and the Transferable Belief Model (TBM). Three simulations demonstrated the suitability of the TBM-based model to deal with partially occluded facial parts and revealed the differences between the facial information used by humans and by the model. This opens promising perspectives for the future development of the model.

Emotional State Inference Using Face Related Features

Abstract: Obtaining reliable and complete systems able to extract human emotional status from streaming videos is of paramount importance to Human Machine Interaction (HMI) applications. Side views, unnatural postures and context are challenges. This paper presents a semi-supervised fuzzy emotional classification system based on Russell’s circumplex model. This emotional inference system relies only on face related features codified with the Facial Action Coding System (FACS). These features are provided by a morphable 3D tracking system robust to posture, occlusion and illumination changes.

Detection of asymmetric eye action units in spontaneous videos

Abstract: With recent advances in machine vision, automatic detection of human expressions in video is becoming important especially because human labeling of videos is both tedious and error prone. In this paper, we present an approach for detecting facial expressions based on the Facial Action Coding System (FACS) in spontaneous videos. We present an automated system for detecting asymmetric eye open (AU41) and eye closed (AU43) actions. We use Gabor Jets to select distinctive features from the image and compare between three different classifiers—Bayesian networks, Dynamic Bayesian networks and Support Vector Machines—for classification. Experimental evaluation on a large corpus of spontaneous videos yielded an average accuracy of 98% for eye closed (AU43), and 92.75% for eye open (AU41).

Personalized Facial Expression Recognition in Color Image

Abstract: Facial expression recognition is one of the most popular topics in emotion analysis. Most facial expression recognition systems are implemented by general expression models. Since facial expressions may be expressed differently by different people, inaccurate results are unavoidable. The proposed facial expression recognition system recognizes facial expressions using facial features of the individual user. Experimental results show that the proposed system can accurately identify the emotions of facial expressions.

The representation of facial expression in Facial Muscle Animatronics which uses Moving-Units

Abstract: Animatronics, which is a synthetic spatial production method of art and engineering in the real space. This paper regards the study on the facial expression of animatronics, the process of making and operating the entertainment robots that look like real people or animals, used in films or theme parks. To realize the animatronics with rich facial expressions like human beings by efficiently controlling the smaller number of moving units than former robots, this paper proposes new MU (Moving-Unit) and its description methods, and presents the validity of the MU through the real human face as well as the physical and psychological measurements. The MU system was proposed as a new classification of action unit for the facial expression of robot. The MU is the basic unit to express the face and movement of animatronics based on the actuator's operation in consideration of mechanical muscle structure. The modem animatronics relates the control of actuator with the human facial expression. In this paper, the moving units and their description methods for the production/control of existing entertainment robots are analyzed and their characteristics are compared with each other. As a result of analysis on the moving units of former facial expression robot, the muscle name, the moving part and the description of motions related with the application of Action Units (AU) have been used in mixture. This classification of robot moving unit mainly refers to the FACS (Facial Action Coding System, 1978) which defines the minimum units of human facial expression with the AU by Paul Ekman and others. Through this research, it was found out that, for the physical expression of face technology of humanoid animatronics in the entertainment, its control method with the anticipation of human facial expression to some degree could be effective through the real robot production. In the future, the relation between the human facial expression and the MU technology should be studied more. This study is expected to help the production of robots not only for entertainment but for education, medicine, sports, and other purposes.

Development of advanced Active Appearance Model for facial emotion recognition

Abstract: We addresses the issue of expressive face modeling using an advanced active appearance model for facial emotion recognition. We consider the six universal emotional categories that are defined by Ekman. In human face, emotions are most widely represented with eyes and mouth expression. If we want to recognize the human's emotion from this facial image, we need to extract feature points such as Action Unit(AU) of Ekman. Active Appearance Model (AAM) is one of the commonly used methods for facial feature extraction and it can be applied to construct AU. Regarding the traditional AAM depends on the setting of the initial parameters of the model and this paper introduces a facial emotion recognizing method based on which is combined Advanced AAM with Bayesian Network. Firstly, we obtain the reconstructive parameters of the new gray-scale image by sample-based learning and use them to reconstruct the shape and texture of the new image and calculate the initial parameters of the AAM by the reconstructed facial model. Then reduce the distance error between the model and the target contour by adjusting the parameters of the model. Finally get the model which is matched with the facial feature outline after several iterations and use them to recognize the facial emotion by using Bayesian Network.

An Approach for Creating and Blending Synthetic Facial Expressions of Emotion

Abstract: We introduce a physically-based facial animation system for natural-looking emotional expressions. Our system enables creation of simple facial expressions using Facial Action Coding System (FACS) and complex facial expressions by blending existing expressions. We propose a method for blending facial actions, based on a parameterization that combines elements of FACS and MPEG-4 standard. We explore the complex expression space by examining the blends of basic emotions produced with our model.

Facial Expression Analysis

Abstract: Facial expression recognition is a process performed by humans or computers, which consists of: 1. Locating faces in the scene (e.g., in an image; this step is also referred to as face detection), 2. Extracting facial features from the detected face region (e.g., detecting the shape of facial components or describing the texture of the skin in a facial area; this step is referred to as facial feature extraction), 3. Analyzing the motion of facial features and/or the changes in the appearance of facial features and classifying this information into some facialexpression-interpretative categories such as facial muscle activations like smile or frown, emotion (affect) categories like happiness or anger, attitude categories like (dis)liking or ambivalence, etc. (this step is also referred to as facial expression interpretation).

Upper Facial Action Unit Recognition

Abstract: This paper concentrates on the comparisons of systems that are used for the recognition of expressions generated by six upper face action units (AU s) by using Facial Action Coding System (FACS ). Haar wavelet, Haar-Like and Gabor wavelet coefficients are compared, using Adaboost for feature selection. The binary classification results by using Support Vector Machines (SVM ) for the upper face AU s have been observed to be better than the current results in the literature, for example 96.5% for AU2 and 97.6% for AU5 . In multi-class classification case, the Error Correcting Output Coding (ECOC ) has been applied. Although for a large number of classes, the results are not as accurate as the binary case, ECOC has the advantage of solving all problems simultaneously; and for large numbers of training samples and small number of classes, error rates are improved.

FACS-coding of facial expressions

Abstract: Facial expressions can be coded manually using FACS, which is based on local visual movements of the face, called Action Units, caused by contraction/dilatation of facial muscles. Automatic coding tools based on AU's are still under development. In this paper, we present a tool to generate facial expressions. The GUI of that tool has moving sliders corresponding to the activation of AU's. All generated facial expressions can be coded by the position of the sliders and as a consequence the percentage of activation of AU's. We coded 21 well-known emotional facial expressions. The contribution of this paper consists of investigating whether the Euclidean distance measure in the AU-coded space can be used as a semantic distance measure.