Article
Reference
Short-term emotion assessment in a recall paradigm
CHANEL, Guillaume, et al.
Abstract
The work presented in this paper aims at assessing human emotions using peripheral as well
as electroencephalographic (EEG) physiological signals on short-time periods. Three specific
areas of the valence–arousal emotional space are defined, corresponding to negatively
excited, positively excited, and calm-neutral states. An acquisition protocol based on the recall
of past emotional life episodes has been designed to acquire data from both peripheral and
EEG signals. Pattern classification is used to distinguish between the three areas of the
valence–arousal space. The performance of several classifiers has been evaluated on 10
participants and different feature sets: peripheral features, EEG time–frequency features, EEG
pairwise mutual information (MI) features. Comparison of results obtained using either
peripheral or EEG signals confirms the interest of using EEGs to assess valence and arousal
in emotion recall conditions. The obtained accuracy for the three emotional classes is 63%
using EEG time–frequency features, which is better than the results obtained from previous
studies using EEG and similar classes. [...]
CHANEL, Guillaume, et al. Short-term emotion assessment in a recall paradigm. International
journal of human-computer studies, 2009, vol. 67, no. 8, p. 607-627
DOI : 10.1016/j.ijhcs.2009.03.005
Available at:
http://archive-ouverte.unige.ch/unige:47415
Disclaimer: layout of this document may differ from the published version.
1 / 1
www.elsevier.com/locate/ijhcs
Author’s Accepted Manuscript
Short-term emotion assessment in a recall paradigm
GuillaumeChanel,JoepKierkels,MohammadSoleymani,
Thierry Pun
PII: S1071-5819(09)00043-3
DOI: doi:10.1016/j.ijhcs.2009.03.005
Reference: YIJHC1519
To appear in: Int. J. Human–Computer Studies
Received date: 4 J uly 2008
Revised date: 19 March 2009
Accepted date: 27 March 2009
Cite this article as: Guillaume Chanel, Joep Kierkels, Mohammad Soleymani and Thierry
Pun, Short-term emotion assessment in a recall paradigm,Int. J. Human–ComputerStudies
(2009), doi:10.1016/j.ijhcs.2009.03.005
This is a PDF file of an unedited manuscript that h as been accepted for publication. As
a service to our customers we are providing this early version of the manuscript. The
manuscript will undergo copyediting, typesetting, and review of the resulting galley proof
before it is publishedinitsfinalcitableform.Please notethatduringt he productionprocess
errors may be discoveredwhichcould affect the content, and all legaldisclaimers that apply
to the journal pertain.
Accepted manuscript
1
Short-term emotion assessment in a recall
paradigm
Guillaume Chanel
*
, Joep Kierkels, Mohammad Soleymani, Thierry Pun
Computer Science Departement – University of Geneva
Route de Drize 7
CH - 1227 Carouge, Switzerland
Abstract
The work presented in this paper aims at assessing human emotions using
peripheral as well as electroencephalographic (EEG) physiological signals on short-
time periods. Three specific areas of the valence-arousal emotional space are
defined, corresponding to negatively excited, positively excited, and calm-neutral
states. An acquisition protocol based on the recall of past emotional life episodes has
been designed to acquire data from both peripheral and EEG signals. Pattern
classification is used to distinguish between the three areas of the valence-arousal
space. The performance of several classifiers has been evaluated on ten participants
and different feature sets: peripheral features, EEG time-frequency features, EEG
pairwise mutual information features. Comparison of results obtained using either
peripheral or EEG signals confirms the interest of using EEG’s to assess valence and
arousal in emotion recall conditions. The obtained accuracy for the three emotional
classes is 63% using EEG time-frequency features which is better than the results
obtained from previous studies using EEG and similar classes. Fusion of the different
feature sets at the decision level using a summation rule also showed to improve
accuracy to 70%. Furthermore, the rejection of non confident samples finally led to a
classification accuracy of 80% for the three classes.
Keywords: emotion assessment and classification, affective computing, signal
processing.
1 Introduction
Emotions are part of any natural communication between humans, generally as non-
verbal cues. Until recently affective communication was not implemented in human
computer interfaces. Nowadays researchers in human-computer interaction (HCI)
have recognized the importance of emotional aspects and started to include them in
the design of new interfaces using one of two possible approaches, either as
evaluation indicators or as components to be inserted in the human-computer loop.
The first approach consist in using emotion assessment as a tool for evaluating
attractiveness, appreciation and user experience of software (Hazlett and Benedek,
2007). Such an assessment can be done by using different self-report methods
(Isomursu et al., 2007) or by inferring emotional states from others measures such as
physiological signals (Mandryk et al., 2006; Picard and Daily, 2005).
The second approach aims at bringing the machine closer to the human by including
emotional content in the communication and is known as affective computing (Picard,
1997). According to Picard, affective computing “proposes to give computers the
ability to recognize [and] express […] emotions”. Synthetic expression of emotions
*
Corresponding author. Tel.: (+41) 22 379 01 83; fax: (+41) 22 379 0250.
E-mail address: guillaume.chanel@unige.ch.
Accepted manuscript
2
can be achieved by enabling avatars or simpler agents to have facial expressions,
different tones of voice, and empathic behaviours (Brave et al., 2005; Xu et al.,
2006). Detection of human emotions can be realized by monitoring facial expressions
(Cohen et al., 2003; Cowie et al., 2001; Fasel and Luettin, 2003), speech (Cowie,
2000; Ververidis and Kotropoulos, 2006) , postures (Coulson, 2004; Kapoor et al.,
2007) and physiological signals (see Section 2). Fusion of these different modalities
to improve the recognition accuracy has also been studied (Kapoor et al., 2007; Kim
et al., 2005; Pantic and Rothkrantz, 2003; Zeng et al., 2008).The present work
focuses on the emotion assessment aspect, especially from different physiological
signals.
Fig. 1 presents a framework describing how emotion assessment could be integrated
in human-computer interfaces. As proposed by Norman (1990) the interaction with a
machine, from the point of view of the user, can be decomposed in execution /
evaluation cycles. After identifying his/her goals, the user starts an execution stage. It
consists in formulating his intentions, specifying the necessary sequence of actions
and executing those actions. Next, the computer executes the given commands and
output results through the available modalities. The second stage is the evaluation
which is realized by: perceiving computer outputs, interpreting them and evaluating
the outcome (i.e. are the goals satisfied?).
<Figure 1>
According to the cognitive theory of emotions, emotions are issued from a cognitive
process called appraisal that evaluates a stimulus according to several criteria such
as goal relevance and consequences of the event (Cornelius, 1996; Sander et al.,
2005; Scherer, 2001). For this reason, an emotional evaluation step, corresponding
to the appraisal process, was added in Fig. 1 at the evaluation stage. Elicitation of
emotions is known to be related to changes in several components of the organism
such as physiological, motor and behavioral components (Scherer, 2001). It is thus
possible to consider those changes as emotional cues that can be used to
automatically detect the elicited emotion after being recorded by the adequate
sensors. For this purpose it is necessary to extract features of interest from the
recorded signals and perform classification using previously trained models from data
including the associated emotion. The detected emotion can then be used to adapt
the interaction by modifying command execution. The information presented on the
output modalities can also directly be influenced by the emotional adaptation, for
instance by synthesizing an emotional response on screens and speakers.
Several applications can be derived from this framework, some of them going beyond
human-computer interfaces to reach human-machine interfaces in general and even
human-human interfaces. In gaming, emotion assessment can be used for better
understanding of the playing conditions that lead to emotional activation (Mandryk
and Atkins, 2007) and for maintaining involvement of a player by adapting game
difficulty or content (Chanel et al., 2008; Chen, 2007; Rani et al., 2005). Learning in a
computer mediated environment can elicit various types of feeling that are currently
not handled. Detection of frustration for instance, if followed by proper adaptation of
the learning strategy (Choi et al., 2007; Kapoor et al., 2007), will certainly help to
maintain learner’s interest. Identification of critical states, such as stress, panic, and
boredom, can be very useful in situations such as driving or when performing
dangerous operations (Benoit et al., 2006; Healey, 2000). Another possible
Accepted manuscript
3
application is the use of emotion recognition to help severely disabled persons
express their feeling.
In addition to the cognitive theory, several theories of emotions have developed over
the past century (Cornelius, 1996). These different views gave rise to different
models of emotions. The most famous models are the basic emotions (Ekman et al.,
1987) and the valence-arousal space (Russell, 1980). Basic emotions are defined as
the emotions that are common across cultures and selected by nature because of
their high survival functions. The valence-arousal space allows for a continuous
representation of emotions on two axes: valence, ranging from unpleasant to
pleasant, and arousal, ranging from calm to excited.
As stated before, there is extensive literature regarding emotion assessment from
speech and facial expressions. However these two modalities suffer from several
disadvantages. Firstly, they are not always available: the user may not look at the
camera or speak all the time. Secondly, they do not always reflect the true emotional
state of the user, since facial and voice expressions are often faked because of
social rules or by deliberate choice. Finally, they should be regarded as the output of
the emotional process, meaning that the emotion could have been elicited prior to the
speech or expression.
An alternative is to use physiological signals both from the central and peripheral
system. The Jamesian theory (Cornelius, 1996) emphasizes the importance of
peripheral signals as it suggests there is some specific pattern of physiology for each
particular emotion. Even though this statement is often disputed in the psychology
literature (Stemmler et al., 2001) and the cognitive theory suggest that physiological
signals are also an output of the emotional process, several studies from human
computer interaction have shown the usefulness of peripheral activity for emotion
assessment in diverse conditions (see Section 2.1). Those studies mainly use signals
duration of the order of the minute to obtain accurate emotion assessment, however
shorter durations are necessary for real time applications. The cognitive theory
stresses the importance of the central nervous system i.e. the brain. It has been
shown that correlates of emotions can be observed from brain activity, especially in
the pre-frontal cortex and the amygdala (Adolphs et al., 2003; Damasio et al., 2000;
Davidson, 2003; Davidson et al., 2000; Rolls, 2000). Brain activity can be measured
using several techniques such as Electroencephalography (EEG) and others. Since it
is the least intrusive and comparatively cheap, EEG is certainly one of the most
usable modality for real and everyday applications. Brain activity can also be useful
for short term emotion assessment since activity related to an emotional stimulus
should occur shortly after the stimulus onset. Despite of this, the few studies trying to
use EEG signals for emotion assessment did not obtain convincing results.
The objective of this study is to investigate the use of EEG modality, peripheral
modality and fusion of these two for emotion assessment on short time periods. This
work will thus focus on the red parts of Fig. 1. Previous studies related to emotion
assessment from physiological signals and novelties of the present study are detailed
in Section 2. A protocol based on the recall of past emotional episodes was used to
reliably induce emotions from three areas of the valence-arousal space. This protocol
was used instead of a real HCI framework because it allows for better control over
when an emotional episode starts (timing) and over the strength and valence of the
recalled emotion (content). The protocol, the signal acquisition system and the
algorithms developed to extract different feature sets from the recorded signals are
