Emotion and Adaptation

Automated analysis of human affective behavior has attracted increasing attention from researchers in psychology, computer science, linguistics, neuroscience, and related disciplines. However, the existing methods typically handle only deliberately displayed and exaggerated expressions of prototypical emotions despite the fact that deliberate behaviour differs in visual appearance, audio profile, and timing from spontaneously occurring behaviour. To address this problem, efforts to develop algorithms that can process naturally occurring human affective behaviour have recently emerged. Moreover, an increasing number of efforts are reported toward multimodal fusion for human affect analysis including audiovisual fusion, linguistic and paralinguistic fusion, and multi-cue visual fusion based on facial expressions, head movements, and body gestures. This paper introduces and surveys these recent advances. We first discuss human emotion perception from a psychological perspective. Next we examine available approaches to solving the problem of machine understanding of human affective behavior, and discuss important issues like the collection and availability of training and test data. We finally outline some of the scientific and engineering challenges to advancing human affect sensing technology.

A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions

Since emotions are expressed through a combination of verbal and non-verbal channels, a joint analysis of speech and gestures is required to understand expressive human communication. To facilitate such investigations, this paper describes a new corpus named the “interactive emotional dyadic motion capture database” (IEMOCAP), collected by the Speech Analysis and Interpretation Laboratory (SAIL) at the University of Southern California (USC). This database was recorded from ten actors in dyadic sessions with markers on the face, head, and hands, which provide detailed information about their facial expressions and hand movements during scripted and spontaneous spoken communication scenarios. The actors performed selected emotional scripts and also improvised hypothetical scenarios designed to elicit specific types of emotions (happiness, anger, sadness, frustration and neutral state). The corpus contains approximately 12 h of data. The detailed motion capture information, the interactive setting to elicit authentic emotions, and the size of the database make this corpus a valuable addition to the existing databases in the community for the study and modeling of multimodal and expressive human communication.

/pdf/iemocap-interactive-emotional-dyadic-motion-capture-database-2ffyt1sb5i.pdf

IEMOCAP: interactive emotional dyadic motion capture database

Two channels have been distinguished in human interaction: one transmits explicit messages, which may be about anything or nothing; the other transmits implicit messages about the speakers themselves. Both linguistics and technology have invested enormous efforts in understanding the first, explicit channel, but the second is not as well understood. Understanding the other party's emotions is one of the key tasks associated with the second, implicit channel. To tackle that task, signal processing and analysis techniques have to be developed, while, at the same time, consolidating psychological and linguistic analyses of emotion. This article examines basic issues in those areas. It is motivated by the PKYSTA project, in which we aim to develop a hybrid system capable of using information from faces and voices to recognize people's emotions.

Emotion recognition in human-computer interaction

Survey on speech emotion recognition: Features, classification schemes, and databases

We present underpinning neural structures to represent various components of emotional dispositions. After a description of simple models of the dynamics, an even simpler multi-layer perceptron model is presented with three outputs. This leads to a high level of success in recognition of emotional dispositions from a database of faces, as well as an interpretation in terms of the underlying neural system. By extending the discrete classification approach to continuous variables in a three-dimensional state-space the recognition performance can be improved and the causes for classification errors can be studied. Key-Words: facial analysis, emotion recognition IMACS/IEEE CSCC'99 Proceedings, Pages:5341-5346

Towards a neural based theory of emotional dispositions

An earlier model introduced by the authors (1999) for fast associative memory has shown to be an efficient solution to the storage of binary patterns and the recall from incomplete input. We extend this model to include more biologically realistic constraints to serve as a model for the hippocampus. Among the constraints considered are the limited overall connectivity between the neurons and the distributed processing in a sequence of layered topographically connected maps. Although not all biophysical and modulatory effects from various sources have been incorporated into the present model, the emergent computational function of the hippocampus as a fast storage mechanism with reliable retrieval and pattern completion abilities from partial cues is the main subject of our study. We show that the proposed multiple layer mechanism employing a sparse code and a k-winner-take-all mechanism for the storage and retrieval of binary patterns can be matched to the functional layers of the hippocampus, thereby predicting computational roles for each map and an overall processing principle.

W. Fellenz

Papers

Towards a neural based theory of emotional dispositions

The hidden layer associative memory model of hippocampus