Emotion and Adaptation

Automated analysis of human affective behavior has attracted increasing attention from researchers in psychology, computer science, linguistics, neuroscience, and related disciplines. However, the existing methods typically handle only deliberately displayed and exaggerated expressions of prototypical emotions despite the fact that deliberate behaviour differs in visual appearance, audio profile, and timing from spontaneously occurring behaviour. To address this problem, efforts to develop algorithms that can process naturally occurring human affective behaviour have recently emerged. Moreover, an increasing number of efforts are reported toward multimodal fusion for human affect analysis including audiovisual fusion, linguistic and paralinguistic fusion, and multi-cue visual fusion based on facial expressions, head movements, and body gestures. This paper introduces and surveys these recent advances. We first discuss human emotion perception from a psychological perspective. Next we examine available approaches to solving the problem of machine understanding of human affective behavior, and discuss important issues like the collection and availability of training and test data. We finally outline some of the scientific and engineering challenges to advancing human affect sensing technology.

A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions

Since emotions are expressed through a combination of verbal and non-verbal channels, a joint analysis of speech and gestures is required to understand expressive human communication. To facilitate such investigations, this paper describes a new corpus named the “interactive emotional dyadic motion capture database” (IEMOCAP), collected by the Speech Analysis and Interpretation Laboratory (SAIL) at the University of Southern California (USC). This database was recorded from ten actors in dyadic sessions with markers on the face, head, and hands, which provide detailed information about their facial expressions and hand movements during scripted and spontaneous spoken communication scenarios. The actors performed selected emotional scripts and also improvised hypothetical scenarios designed to elicit specific types of emotions (happiness, anger, sadness, frustration and neutral state). The corpus contains approximately 12 h of data. The detailed motion capture information, the interactive setting to elicit authentic emotions, and the size of the database make this corpus a valuable addition to the existing databases in the community for the study and modeling of multimodal and expressive human communication.

/pdf/iemocap-interactive-emotional-dyadic-motion-capture-database-2ffyt1sb5i.pdf

IEMOCAP: interactive emotional dyadic motion capture database

Two channels have been distinguished in human interaction: one transmits explicit messages, which may be about anything or nothing; the other transmits implicit messages about the speakers themselves. Both linguistics and technology have invested enormous efforts in understanding the first, explicit channel, but the second is not as well understood. Understanding the other party's emotions is one of the key tasks associated with the second, implicit channel. To tackle that task, signal processing and analysis techniques have to be developed, while, at the same time, consolidating psychological and linguistic analyses of emotion. This article examines basic issues in those areas. It is motivated by the PKYSTA project, in which we aim to develop a hybrid system capable of using information from faces and voices to recognize people's emotions.

Emotion recognition in human-computer interaction

Survey on speech emotion recognition: Features, classification schemes, and databases

We describe an empirical approach to identifying the kind of task that an emotion recognition system could usefully address. Three levels of information are elicited – a basic emotion vocabulary, a basic representation in ‘evaluation-activation space’ of the meaning of each word, and a richer ‘schema’ representation. The results confirm that an approach of this kind is feasible. Key-Words: emotion recognition, semantics, neural network CSCC'99 Proceedings, Pages:5311-5316

What a neural net needs to know about emotion words

We propose a framework for the processing of face image sequences and speech, using different dynamic techniques to extract appropriate features for emotion recognition. The features will be used by a hybrid classification procedure, employing neural network techniques and fuzzy logic, to accumulate the evidence for the presence of an emotional expression of the face and the speaker's voice.

On emotion recognition of faces and of speech using neural networks, fuzzy logic and the ASSESS system

We split the rule extraction task into a subsymbolic and a symbolic phase and present a set of neural networks for filling the former. Under the two general commitments of: (i) having a learning algorithm that is sensitive to feedback signals coming from the latter phase, and (ii) extracting Boolean variables whose meaning is determined by the further symbolic processing, we introduce three unsupervised learning algorithms and show related numerical examples for a multilayer perceptron, recurrent neural networks, and a specially devised vector quantizer.

A general framework for symbol and rule extraction in neural networks

We present a simple modification to binary associative memories employing a designated intermediate layer representation which reduces crosstalk between stored patterns and enhances performance and average storage capacity of the network. It is shown that the hidden layer associative network is less dependent on the number of ones in the storage patterns and shows an improved retrieval capacity from incomplete retrieval patterns.

W. Fellenz

Papers

Emotion recognition in human-computer interaction

What a neural net needs to know about emotion words

On emotion recognition of faces and of speech using neural networks, fuzzy logic and the ASSESS system

A general framework for symbol and rule extraction in neural networks

Enhancing the storage capacity of binary associative memories