Showing papers on "Facial Action Coding System published in 1998"

Automated facial expression recognition based on FACS action units

Abstract: Automated recognition of facial expression is an important addition to computer vision research because of its relevance to the study of psychological phenomena and the development of human-computer interaction (HCI). We developed a computer vision system that automatically recognizes individual action units or action unit combinations in the upper face using hidden Markov models (HMMs). Our approach to facial expression recognition is based an the Facial Action Coding System (FACS), which separates expressions into upper and lower face action. We use three approaches to extract facial expression information: (1) facial feature point tracking; (2) dense flow tracking with principal component analysis (PCA); and (3) high gradient component detection (i.e. furrow detection). The recognition results of the upper face expressions using feature point tracking, dense flow tracking, and high gradient component detection are 85%, 93% and 85%, respectively.

Feature-point tracking by optical flow discriminates subtle differences in facial expression

Abstract: Current approaches to automated analysis have focused an a small set of prototypic expressions (e.g. joy or anger). Prototypic expressions occur infrequently in everyday life, however, and emotion expression is far more varied. To capture the full range of emotion expression, automated discrimination of fine grained changes in facial expression is needed. We developed and implemented an optical flow based approach (feature point tracking) that is sensitive to subtle changes in facial expression. In image sequences from 100 young adults, action units and action unit combinations in the brow and mouth regions were selected for analysis if they occurred a minimum of 25 times in the image database. Selected facial features were automatically tracked using a hierarchical algorithm for estimating optical flow. Image sequences were randomly divided into training and test sets. Feature point tracking demonstrated high concurrent validity with human coding using the Facial Action Coding System (FACS).

Automatic recognition of facial expressions using hidden markov models and estimation of expression intensity

Abstract: Facial expressions provide sensitive cues about emotional responses and play a major role in the study of psychological phenomena and the development of nonverbal communication. Facial expressions regulate social behavior, signal communicative intent, and are related to speech production. Most facial expression recognition systems focus on only six basic expressions. In everyday life, however, these six basic expressions occur relatively infrequently, and emotion or intent is more often communicated by subtle changes in one or two discrete features, such as tightening of the lips which may communicate anger. Humans are capable of producing thousands of expressions that vary in complexity, intensity, and meaning. The objective of this dissertation is to develop a computer vision system, including both facial feature extraction and recognition, that automatically discriminates among, subtly different facial expressions based on Facial Action Coding System (FACS) action units (AUs) using Hidden Markov Models (HMMs). Three methods are developed to extract facial expression information for automatic recognition. The first method is facial feature point tracking using the coarse-to-fine pyramid method, which can be sensitive to subtle feature motion and is capable to handle large displacements with sub-pixel accuracy. The second is dense flow tracking together with principal component analysis, where the entire facial motion information per frame is compressed to a low-dimensional weight vector for discrimination. And the third is high gradient component (i.e., furrow) analysis in the spatio-temporal domain, which exploits the transient variance associated with the facial expression. Upon extraction of the facial information, non-rigid facial expressions are separated from the rigid head motion components, and the face images are automatically aligned and normalized using an affine transformation. The resulting motion vector sequence is vector quantized to provide input to an HMM-based classifier, which addresses the time warping problem. A method is developed for determining the HMM topology optimal for our recognition system. The system also provides expression intensity estimation, which has significant effect on the actual meaning of the expression. We have studied more than 400 image sequences obtained from 90 subjects. The experimental results of our trained system showed an overall recognition accuracy of 87%, and also 87% in distinguishing among sets of three and six subtly different facial expressions for upper and lower facial regions, respectively.

Subtly different facial expression recognition and expression intensity estimation

Abstract: We have developed a computer vision system, including both facial feature extraction and recognition, that automatically discriminates among subtly different facial expressions. Expression classification is based on Facial Action Coding System (FACS) action units (AUs), and discrimination is performed using Hidden Markov Models (HMMs). Three methods are developed to extract facial expression information for automatic recognition. The first method is facial feature point tracking using a coarse-to-fine pyramid method. This method is sensitive to subtle feature motion and is capable of handling large displacements with sub-pixel accuracy. The second method is dense flow tracking together with principal component analysis (PCA) where the entire facial motion information per frame is compressed to a low-dimensional weight vector. The third method is high gradient component (i.e., furrow) analysis in the spatio-temporal domain, which exploits the transient variation associated with the facial expression. Upon extraction of the facial information, non-rigid facial expression is separated from the rigid head motion component, and the face images are automatically aligned and normalized using an affine transformation. This system also provides expression intensity estimation, which has significant effect on the actual meaning of the expression.

Face image analysis by unsupervised learning and redundancy reduction

Abstract: In a task such as face recognition, much of the important information may be contained in the high-order relationships among the image pixels. Representations such as "Eigenfaces" (197) and "Holons" (48) are based on Principal component analysis (PCA), which encodes the correlational structure of the input, but does not address high-order statistical dependencies such as relationships among three or more pixels. Independent component analysis (ICA) is a generalization of PCA which encodes the high-order dependencies in the input in addition to the correlations. Representations for face recognition were developed from the independent components of face images. The ICA representations were superior to PCA for recognizing faces across sessions and changes in expression. ICA was compared to more than eight other image analysis methods on a task of recognizing facial expressions in a project to automate the Facial Action Coding System (62). These methods included estimation of optical flow; representations based on the second-order statistics of the full face images such Eigenfaces (47, 197) local feature analysis (156), and linear discriminant analysis (23); and representations based on the outputs of local filters, such as a Gabor wavelet representations (50, 113) and local PCA (153). The ICA and Gabor wavelet representations achieved the best performance of 96% for classifying 12 facial actions. Relationships between the independent component representation and the Gabor representation are discussed. Temporal redundancy contains information for learning invariances. Different views of a face tend to appear in close temporal proximity as the person changes expression, pose, or moves through the environment. The final chapter modeled the development of viewpoint invariant responses to faces from visual experience in a biological system by encoding spatio-temporal dependencies. The simulations combined temporal smoothing of activity signals with Hebbian learning (72) in a network with both feed-forward connections and a recurrent layer that was a generalization of a Hopfield attractor network. Following training on sequences of graylevel images of faces as they changed pose, multiple views of a given face fell into the same basin of attraction, and the system acquired representations of faces that were approximately viewpoint invariant.

Facial expression and the affective component of cynical hostility in male coronary heart disease patients.

Abstract: This study describes the affective component of hostility as measured by the Cook-Medley Hostility Scale (Ho; W. Cook & D. Medley, 1954) by examining the relationship between facial expressions of emotion and Ho scores in 116 male coronary heart disease patients. Patients underwent the videotaped Type A Structured Interview, from which facial expressions were later coded using the Facial Action Coding System. They also completed the Cook-Medley Ho scale. Facial expression of the emotion of contempt was significantly related to Ho scores; anger expression was not. Also, there was a significant interaction between hostility and defensiveness, wherein low-defensive, highly hostile people showed substantially more contempt expression than others. The implications of these findings for the construct validity of Ho and for identifying clinically important subtypes of hostility are discussed.

Temporal differences in eye and mouth movements classifying facial expressions of smiles

Abstract: In our previous work (S. Nishio and K. Koyama, 1997), we examined the effect of temporal differences in eye and mouth movements on classifying facial expressions of smiles. The results showed that: (a) these differences significantly influence the classification; (b) when the mouth begins moving prior to the eyes, the expressions are taken as positive; (c) when the eyes move prior to the mouth, the expressions are taken as negative; and (d) when both movements begin simultaneously, the expressions are taken as social laughter. The above results were re-examined by increasing the number of participants. Additionally, we found that when the attention of an observer is drawn to the mouth, the rate of recognizing the facial expression of smile as positive significantly increases.

Classification of facial expressions using self-organizing maps

Abstract: Just as humans use body language or nonverbal language such as gestures and facial expressions in communication, computers will also be able to communicate with humans. In medical engineering, it is possible that recognition of facial expression can be applied to support communication with persons who have trouble communicating verbally such as infants and mental patients. The purpose of this study is to enable recognition of human emotions by facial expressions. Our observations of facial expressions found that recognizing facial expressions by identifying changes in important facial segments such as the eyebrow, the eyes and the mouth by using sequences of images is important. Self-organizing maps, which are neural networks, are used to extract features of image sequences. The image sequences of six types of facial expressions are recorded on VTR and made into image sequences consisting of 30 images per second. Gray levels of each segment are input into the self-organizing map corresponding to each segment. The neuron in the output layer, called the victory neuron, reacts to the feature nearest the input segment. Our analysis of the changes in victory neurons demonstrates that they have characteristic features which correspond to each of the six facial expressions.