![Fig. 5. Fiducial grid of facial points ([51]).](/figures/fig-5-fiducial-grid-of-facial-points-51-h5mijiny.webp)
![Fig. 6. Facial Characteristic Points ([42]).](/figures/fig-6-facial-characteristic-points-42-101txrc5.webp)
![Fig. 4. Fitted PDM model (reprinted from [32] with permission from Academic Press) ß 1997 Academic Press.](/figures/fig-4-fitted-pdm-model-reprinted-from-32-with-permission-3g4t4y5v.webp)
![Fig. 3. Aligned training set for generation of PDM model (reprinted from [32] with permission from Academic Press) ß 1997 Academic Press.](/figures/fig-3-aligned-training-set-for-generation-of-pdm-model-3hhs8xxe.webp)
![Fig. 11. The FFPs ([95]).](/figures/fig-11-the-ffps-95-2c9xdbcr.webp)
![Fig. 10. Potential Field and corresponding Potential Net ([41]).](/figures/fig-10-potential-field-and-corresponding-potential-net-41-1f3goe5o.webp)
![Fig. 9. Motion vector field represented in the deformation of two grids ([60]).](/figures/fig-9-motion-vector-field-represented-in-the-deformation-of-1ibd4977.webp)
![Fig. 12. APs (reprinted from [32] with permission from Academic Press) ß 1997 Academic Press.](/figures/fig-12-aps-reprinted-from-32-with-permission-from-academic-gf960z47.webp)
Automatic Analysis of Facial Expressions:
The State of the Art
Maja Pantic, Student Member, IEEE, and Leon J.M. Rothkrantz
AbstractÐHumans detect and interpret faces and facial expressions in a scene with little or no effort. Still, development of an
automated system that accomplishes this task is rather difficult. There are several related problems: detection of an image segment as
a face, extraction of the facial expression information, and classification of the expression (e.g., in emotion categories). A system that
performs these operations accurately and in real time would form a big step in achieving a human-like interaction between man and
machine. This paper surveys the past work in solving these problems. The capability of the human visual system with respect to these
problems is discussed, too. It is meant to serve as an ultimate goal and a guide for determining recommendations for development of
an automatic facial expression analyzer.
Index TermsÐFace detection, facial expression information extraction, facial action encoding, facial expression emotional
classification.
æ
1INTRODUCTION
A
S pointed out by Bruce [6], Takeuchi and Nagao [84],
and Hara and Kobayashi [28], human face-to-face
communication is an ideal model for designing a multi-
modal/media human-computer interface (HCI). The main
characteristics of human communication are: multiplicity
and multimodality of communication channels. A channel
is a communication medium while a modality is a sense
used to perceive signals from the outside world. Examples
of human communication channels are: auditory channel
that carries speech, auditory channel that carries vocal
intonation, visual channel that carries facial expressions,
and visual channel that carries body movements. The
senses of sight, hearing, and touch are examples of
modalities. In usual face-to-face communication, many
channels are used and different modalities are activated.
As a result, communication becomes highly flexible and
robust. Failure of one channel is recovered by another
channel and a message in one channel can be explained by
another channel. This is how a multimedia/modal HCI
should be developed for facilitating robust, natural,
efficient, and effective man-machine interaction.
Relatively few existing works combine different mod-
alities into a single system for human communicative
reaction analysis. Examples are the works of Chen et al.
[9] and De Silva et al. [15] who studied the effects of a
combined detection of facial and vocal expressions of
emotions. So far, the majority of studies treat various
human communication channels separately, as indicated by
Nakatsu [58]. Examples for the presented systems are:
emotional interpretation of human voices [35], [66], [68],
[90], emotion recognition by physiological signals pattern
recognition [67], detection and interpretation of hand
gestures [64], recognition of body movements [29], [97],
[46], and facial expression analysis (this survey).
The terms ªface-to-faceº and ªinterfaceº indicate that the
face plays an essential role in interpersonal communication.
The face is the mean to identify other members of the
species, to interpret what has been said by the means of
lipreading, and to understand someone's emotional state
and intentions on the basis of the shown facial expression.
Personality, attractiveness, age, and gender can also be seen
from someone's face. Considerable research in social
psychology has also shown that facial expressions help
coordinate conversation [4], [82], and have considerably
more effect on whether a listener feels liked or disliked than
the speaker's spoken words [55]. Mehrabian indicated that
the verbal part (i.e., spoken words) of a message contributes
only for 7 percent to the effect of the message as a whole, the
vocal part (e.g., voice intonation) contributes for 38 percent,
while facial expression of the speaker contributes for
55 percent to the effect of the spoken message [55]. This
implies that the facial expressions form the major modality
in human communication.
Recent advances in image analysis and pattern recogni-
tion open up the possibility of automatic detection and
classification of emotional and conversational facial signals.
Automating facial expression analysis could bring facial
expressions into man-machine interaction as a new mod-
ality and make the interaction tighter and more efficient.
Such a system could also make classification of facial
expressions widely accessible as a tool for research in
behavioral science and medicine. The goal of this paper is to
survey the work done in automating facial expression
analysis in facial images and image sequences. Section 2
identifies three basic problems related to facial expression
analysis. These problems are: face detection in a facial
image or image sequence, facial expression data extraction,
and facial expression classification. The capability of the
1424 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 22, NO. 12, DECEMBER 2000
. The authors are with the Department of Media, Engineering and
Mathematics, Delft University of Technology, PO Box 356, 2600 AJ
Delft, The Netherlands.
E-mail: {M.Pantic, L.J.M.Rothkrantz}@cs.tudelft.nl.
Manuscript received 29 June 1999; revised 16 March 2000; accepted 12 Sept.
2000.
Recommended for acceptance by K. Bowyer.
For information on obtaining reprints of this article, please send e-mail to:
tpami@computer.org, and reference IEEECS Log Number 110156.
0162-8828/00/$10.00 ß 2000 IEEE
human visual system with respect to these problems is
described. It defines, in some way, the expectations for an
automated system. The characteristics of an ideal auto-
mated system for facial expression analysis are given in
Section 3. Section 4 surveys the techniques presented in the
literature in the past decade for facial expression analysis by
a computer. Their characteristics are summarized in respect
to the requirements posed on the design of an ideal facial
expression analyzer. We do not attempt to provide an
exhaustive review of the past work in each of the problems
related to automatic facial expression analysis. Here, we
selectively discuss systems which deal with each of these
problems. Possible directions for future research are
discussed in Section 5. Section 6 concludes the paper.
2FACIAL EXPRESSION ANALYSIS
Our aim is to explore the issues in design and implementa-
tion of a system that could perform automated facial
expression analysis. In general, three main steps can be
distinguished in tackling the problem. First, before a facial
expression can be analyzed, the face must be detected in a
scene. Next is to devise mechanisms for extracting the facial
expression information from the observed facial image or
image sequence. In the case of static images, the process of
extracting the facial expression information is referred to as
localizing the face and its features in the scene. In the case of
facial image sequences, this process is referred to as tracking
the face and its features in the scene. At this point, a clear
distinction should be made between two terms, namely,
facial features and face model features. The facial features are
the prominent features of the faceÐeyebrows, eyes, nose,
mouth, and chin. The face model features are the features
used to represent (model) the face. The face can be
represented in various ways, e.g., as a whole unit (holistic
representation), as a set of features (analytic representation)
or as a combination of these (hybrid approach). The applied
face representation and the kind of input images determine
the choice of mechanisms for automatic extraction of facial
expression information. The final step is to define some set
of categories, which we want to use for facial expression
classification and/or facial expression interpretation, and to
devise the mechanism of categorization.
Before an automated facial expression analyzerisbuilt,one
should decide on the system's functionality. A good reference
point is the functionality of the human visual system. After all,
it is the best known facial expression analyzer. This section
discusses the three basic problems related to the process of
facial expression analysis as well as the capability of the
human visual system with respect to these.
2.1 Face Detection
For most works in automatic facial expression analysis, the
conditions under which a facial image or image sequence is
obtained are controlled. Usually, the image has the face in
frontal view. Hence, the presence of a face in the scene is
ensured and some global location of the face in the scene is
known a priori. However, determining the exact location of
the face in a digitized facial image is a more complex
problem. First, the scale and the orientation of the face can
vary from image to image. If the mugshots are taken with a
fixed camera, faces can occur in images at various sizes and
angles due to the movements of the observed person. Thus,
it is difficult to search for a fixed pattern (template) in the
image. The presence of noise and occlusion makes the
problem even more difficult.
Humans detect a facial pattern by casual inspection of
the scene. We detect faces effortlessly in a wide range of
conditions, under bad lightning conditions or from a great
distance. It is generally believed that two-gray-levels
images of 100 to 200 pixels form a lower limit for detection
of a face by a human observer [75], [8]. Another character-
istic of the human visual system is that a face is perceived as
a whole, not as a collection of the facial features. The
presence of the features and their geometrical relationship
with each other appears to be more important than the
details of the features [5]. When a face is partially occluded
(e.g., by a hand), we perceive a whole face, as if our
perceptual system fills in the missing parts. This is very
difficult (if possible at all) to achieve by a computer.
2.2 Facial Expression Data Extraction
After the presence of a face has been detected in the
observed scene, the next step is to extract the information
about the encountered facial expression in an automatic
way. If the extraction cannot be performed automatically, a
fully automatic facial expression analyzer cannot be
developed. Both, the applied face representation and the
kind of input images affect the choice of the approach to
facial expression information extraction.
One of the fundamental issues about the facial expres-
sion analysis is the representation of the visual information
that an examined face might reveal [102]. The results of
Johansson's point-light display experiments [1], [5], gave a
clue to this problem. The experiments suggest that the
visual properties of the face, regarding the information
about the shown facial expression, could be made clear by
describing the movements of points belonging to the facial
features (eyebrows, eyes, and mouth) and then by analyzing
the relationships between those movements. This triggered
the researchers of vision-based facial gesture analysis to
make different attempts to define point-based visual
properties of facial expressions. Various analytic face
representations yielded, in which the face is modeled as a
set of facial points (e.g., Fig. 6, [42], Fig. 7, [61]) or as a set of
templates fitted to the facial features such as the eyes and
the mouth. In another approach to face representation
(holistic approach), the face is represented as a whole unit.
A 3D wire-frame with a mapped texture (e.g., [86]) and a
spatio-temporal model of facial image motion (e.g., Fig. 8,
[2]) are typical examples of the holistic approaches to face
representation. The face can be also modeled using a so-
called hybrid approach, which typifies a combination of
analytic and holistic approaches to face representation. In
this approach, a set of facial points is usually used to
determine an initial position of a template that models the
face (e.g., Fig 10, [40]).
Irrespectively of the kind of the face model applied,
attempts must be made to model and then extract the
information about the displayed facial expression without
losing any (or much) of that information. Several factors
make this task complex. The first is the presence of facial
PANTIC AND ROTHKRANTZ: AUTOMATIC ANALYSIS OF FACIAL EXPRESSIONS: THE STATE OF THE ART 1425
hair, glasses, etc., which obscure the facial features. Another
problem is the variation in size and orientation of the face in
input images. This disables a search for fixed patterns in the
images. Finally, noise and occlusion are always present to
some extent.
As indicated by Ellis [23], human encoding of the visual
stimulus (face and its expression) may be in the form of a
primal sketch and may be hardwired. However, not much
else is known in terms of the nature of internal representa-
tion of a face in the human brain.
2.3 Facial Expression Classification
After the face and its appearance have been perceived, the
next step of an automated expression analyzer is to
ªidentifyº the facial expression conveyed by the face. A
fundamental issue about the facial expression classification
is to define a set of categories we want to deal with. A
related issue is to devise mechanisms of categorization.
Facial expressions can be classified in various waysÐin
terms of facial actions that cause an expression, in terms of
some nonprototypic expressions such as ªraised browsº or
in terms of some prototypic expressions such as emotional
expressions.
The Facial Action Coding System (FACS) [21] is probably
the most known study on facial activity. It is a system that
has been developed to facilitate objective measurement of
facial activity for behavioral science investigations of the
face. FACS is designed for human observers to detect
independent subtle changes in facial appearance caused by
contractions of the facial muscles. In a form of rules, FACS
provides a linguistic description of all possible, visually
detectable, facial changes in terms of 44 so-called Action
Units (AUs). Using these rules, a trained human FACS
coder decomposes a shown expression into the specific AUs
that describe the expression. Automating FACS would
make it widely accessible as a research tool in the
behavioral science, which is furthermore the theoretical
basis of multimodal/media user interfaces. This triggered
researchers of computer vision field to take different
approaches in tackling the problem. Still, explicit attempts
to automate the facial action coding as applied to automated
FACS encoding are few (see [16] or [17] for a review of the
existing methods as well as Table 7 of this survey).
Most of the studies on automated expression analysis
perform an emotional classification. As indicated by
Fridlund et al. [25], the most known and the most
commonly used study on emotional classification of facial
expressions is the cross-cultural study on existence of
ªuniversal categories of emotional expressions.º Ekman
defined six such categories, referred to as the basic emotions:
happiness, sadness, surprise, fear, anger, and disgust [19].
He described each basic emotion in terms of a facial
expression that uniquely characterizes that emotion. In the
past years, many questions arose around this study. Are the
basic emotional expressions indeed universal [33], [22], or
are they merely a stressing of the verbal communication
and have no relation with an actual emotional state [76],
[77], [26]? Also, it is not at all certain that each facial
expression able to be displayed on the face can be classified
under the six basic emotion categories. Nevertheless, most
of the studies on vision-based facial expression analysis rely
on Ekman's emotional categorization of facial expressions.
The problem of automating facial expression emotional
classification is difficult to handle for a number of reasons.
First, Ekman's description of the six prototypic facial
expressions of emotion is linguistic (and, thus, ambiguous).
There is no uniquely defined description either in terms of
facial actions or in terms of some other universally defined
facial codes. Hence, the validation and the verification of
the classification scheme to be used are difficult and crucial
tasks. Second, classification of facial expressions in to
multiple emotion categories should be feasible (e.g., raised
eyebrows and smiling mouth is a blend of surprise and
happiness, Fig. 1). Still, there is no psychological scrutiny on
this topic.
Three more issues are related to facial expression
classification in general. First, the system should be capable
of analyzing any subject, male or female of any age and
ethnicity. In other words, the classification mechanism may
not depend on physiognomic variability of the observed
person. On the other hand, each person has his/her own
maximal intensity of displaying a particular facial expres-
sion. Therefore, if the obtained classification is to be
quantified (e.g., to achieve a quantified encoding of facial
actions or a quantified emotional labeling of blended
expressions), systems which can start with a generic
expression classification and then adapt to a particular
individual have an advantage. Second, it is important to
realize that the interpretation of the body language is
situation-dependent [75]. Nevertheless, the information
about the context in which a facial expression appears is
very difficult to obtain in an automatic way. This issue has
not been handled by the currently existing systems. Finally,
there is now a growing psychological research that argues
that timing of facial expressions is a critical factor in the
interpretation of expressions [1], [5], [34]. For the research-
ers of automated vision-based expression analysis, this
suggests moving towards a real-time whole-face analysis of
facial expression dynamics.
While the human mechanisms for face detection are very
robust, the same is not the case for interpretation of facial
expressions. It is often very difficult to determine the exact
nature of the expression on a person's face. According to
Bassili [1], a trained observer can correctly classify faces
1426 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 22, NO. 12, DECEMBER 2000
Fig. 1. Expressions of blended emotions (surpriseÐhappiness).
showing six basic emotions with an average of 87 percent.
This ratio varies depending on several factors: the famil-
iarity with the face, the familiarity with the personality of
the observed person, the general experience with different
types of expressions, the attention given to the face and the
nonvisual cues (e.g., the context in which an expression
appears). It is interesting to note that the appearance of the
upper face features plays a more important role in face
interpretation as opposed to lower face features [22].
3AN IDEAL SYSTEM FOR FACIAL EXPRESSION
ANALYSIS
Before developing an automated system for facial expres-
sion analysis, one should decide on its functionality. A good
reference point is the best known facial expression
analyzerÐthe human visual system. It may not be possible
to incorporate all features of the human visual system into
an automated system, and some features may even be
undesirable, but it can certainly serve as a reference point.
A first requirement that should be posed on developing
an ideal automated facial expression analyzer is that all of
the stages of the facial expression analysis are to be
performed automatically, namely, face detection, facial
expression information extraction, and facial expression
classification. Yet, actual implementation and integration of
these stages into a system are constrained by the system's
application domain. For instance, if the system is to be used
as a tool for research in behavioral science, real-time
performance is not an essential property of the system.
On the other hand, this is crucial if the system would form a
part of an advanced user-interface. Long delays make the
interaction desynchronized and less efficient. Also, having
an explanation facility that would elucidate facial action
encoding performed by the system might be useful if the
system is employed to train human experts in using FACS.
However, such facility is superfluous if the system is to be
employed in videoconferencing or as a stress-monitoring
tool. In this paper, we are mainly concerned with two major
application domains of an automated facial expression
analyzer, namely, behavioral science research and multi-
modal/media HCI. In this section, we propose an ideal
automated facial expression analyzer (Table 1) which could
be employed in those application domains and has the
properties of the human visual system.
Considering the potential applications of an automated
facial expression analyzer, which involve continuous ob-
servation of a subject in a time interval, facial image
acquisition should proceed in an automatic way. In order to
be universal, the system should be capable of analyzing
subjects of both sexes, of any age and any ethnicity. Also, no
constraints should be set on the outlook of the observed
subjects. The system should perform robustly despite
changes in lightning conditions and distractions like
glasses, changes in hair style, and facial hair like moustache,
beard and grown-together eyebrows. Similarly to the
human visual system, an ideal system would ªfill inº
missing parts of the observed face and ªperceiveº a whole
face even when a part of it is occluded (e.g., by hand). In
most real-life situations, complete immovability of the
observed subject cannot be assumed. Hence, the system
should be able to deal with rigid head motions. Ideally, the
system would perform robust facial expression analysis
despite large changes in viewing conditions; it would be
capable of dealing with a whole range of head movements,
from frontal view to profile view acquired by a fixed
camera. This could be achieved by employing several fixed
cameras for acquiring different facial views of the examined
face (such as frontal view, and right and left profile views)
and then approximating the actual view by interpolation
among the acquired views. Having no constraints set on the
rigid head motions of the subject can also be achieved by
having a camera mounted on the subject's head and placed
in front of his/her face.
An ideal system should perform robust automatic face
detection and facial expression information extraction in the
acquired images or image sequences. Considering the state-
of-the-art in image processing techniques, inaccurate, noisy,
and missing data could be expected. An ideal system
should be capable of dealing with these inaccuracies. In
addition, certainty of the extracted facial expression
information should be taken into account.
An ideal system should be able to perform analysis of all
visually distinguishable facial expressions. Well-defined
face representation is a prerequisite for achieving this. The
face representation should be such that a particular
alteration of the face model uniquely reveals a particular
facial expression. In general, an ideal system should be able
to distinguish:
1. all possible facial expressions (a reference point is
a total of 44 facial actions defined in FACS [21]
PANTIC AND ROTHKRANTZ: AUTOMATIC ANALYSIS OF FACIAL EXPRESSIONS: THE STATE OF THE ART 1427
TABLE 1
Properties of an Ideal Analyzer
whose combinations form the complete set of facial
expressions),
2. any bilateral or unilateral facial change [21], and
3. facial expressions with a similar facial appearance
(e.g., upward pull of the upper lip and nose
wrinkling which also causes the upward pull of
the upper lip [21]).
In practice, it may not be possible to define a face model
that can satisfy both, to reflect each and every change in
facial appearance and whose features are detectable in a
facial image or image sequence. Still, the set of distinct facial
expressions that the system can distinguish should be as
copious as possible.
If the system is to be used for behavioral science research
purposes it should perform facial expression recognition as
applied to automated FACS encoding. As explained by
Bartlett et al. [16], [17], this means that it should accomplish
multiple quantified expression classification in terms of
44 AUs defined in FACS. If the system is to be used as a part
of an advanced multimodal/media HCI, the system should
be able to interpret the shown facial expressions (e.g., in
terms of emotions). Since psychological researchers dis-
agree on existence of universal categories of emotional
facial displays, an ideal system should be able to adapt the
classification mechanism according to the user's subjective
interpretation of expressions, e.g., as suggested in [40]. Also,
it is definitely not the case that each and every facial
expression able to be displayed on the face can be classified
into one and only one emotion class. Think about blended
emotional displays such as raised eyebrow and smiling
mouth (Fig. 1). This expression might be classified in two
emotion categories defined by Ekman and Friesen
[20]Ðsurprise and happiness. Yet, according to the descrip-
tions of these prototypic expressions given by Ekman and
Friesen [20], the left hand side facial expression shown in
Fig. 1 belongs ªmoreº to the surpriseÐthan to the
happiness class. For instance, in the left hand side image
the ªpercentageº of shown surprise is higher than the
ªpercentageº of shown happiness while those percentages
are approximately the same in the case of the right hand
side image. In order to obtain an accurate categorization, an
ideal analyzer should perform quantified classification of
facial expression into multiple emotion categories.
4AUTOMATIC FACIAL EXPRESSION ANALYSIS
For its utility in application domains of human behavior
interpretation and multimodal/media HCI, automatic facial
expression analysis has attracted the interest of many
computer vision researchers. Since the mid 1970s, different
approaches are proposed for facial expression analysis from
either static facial images or image sequences. In 1992,
Samal and Iyengar [79] gave an overview of the early
works. This paper explores and compares approaches to
automatic facial expression analysis that have been devel-
oped recently, i.e., in the late 1990s. Before surveying these
works in detail, we are giving a short overview of the
systems for facial expression analysis proposed in the
period of 1991 to 1995.
Table 2 summarizes the features of these systems in
respect to the requirements posed on design of an ideal
facial expression analyzer. None of these systems performs
a quantified expression classification in terms of facial
actions. Also, except the system proposed by Moses et al.
[57], no system listed in Table 2 performs in real-time.
Therefore, these properties (i.e., columns 15 and 20) have
been excluded from Table 2 (. stands for ªyes,º X stands for
ªno,º and - represents a missing entry). A missing entry
either means that it is not reported on the issue or that the
issue is not applicable to the system in question. An
inapplicable issue, for instance, is the issue of dealing with
rigid head motions and inaccurate facial data in the cases
where the input data were hand measured (e.g., [40]). Some
of the methods listed in Table 2 do not perform automatic
facial data extraction (see column 8); the others achieve this
by using facial motion analysis [52], [100], [101], [73], [57].
Except the method proposed by Kearney and McKenzie
[40], which performs facial expression classification in
1428 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 22, NO. 12, DECEMBER 2000
TABLE 2
Early Methods for Automatic Facial Expression Analysis