Automated analysis of human affective behavior has attracted increasing attention from researchers in psychology, computer science, linguistics, neuroscience, and related disciplines. However, the existing methods typically handle only deliberately displayed and exaggerated expressions of prototypical emotions despite the fact that deliberate behaviour differs in visual appearance, audio profile, and timing from spontaneously occurring behaviour. To address this problem, efforts to develop algorithms that can process naturally occurring human affective behaviour have recently emerged. Moreover, an increasing number of efforts are reported toward multimodal fusion for human affect analysis including audiovisual fusion, linguistic and paralinguistic fusion, and multi-cue visual fusion based on facial expressions, head movements, and body gestures. This paper introduces and surveys these recent advances. We first discuss human emotion perception from a psychological perspective. Next we examine available approaches to solving the problem of machine understanding of human affective behavior, and discuss important issues like the collection and availability of training and test data. We finally outline some of the scientific and engineering challenges to advancing human affect sensing technology.

A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions

Survey on speech emotion recognition: Features, classification schemes, and databases

This survey describes recent progress in the field of Affective Computing (AC), with a focus on affect detection. Although many AC researchers have traditionally attempted to remain agnostic to the different emotion theories proposed by psychologists, the affective technologies being developed are rife with theoretical assumptions that impact their effectiveness. Hence, an informed and integrated examination of emotion theories from multiple areas will need to become part of computing practice if truly effective real-world systems are to be achieved. This survey discusses theoretical perspectives that view emotions as expressions, embodiments, outcomes of cognitive appraisal, social constructs, products of neural circuitry, and psychological interpretations of basic feelings. It provides meta-analyses on existing reviews of affect detection systems that focus on traditional affect detection modalities like physiology, face, and voice, and also reviews emerging research on more novel channels such as text, body language, and complex multimodal systems. This survey explicitly explores the multidisciplinary foundation that underlies all AC applications by describing how AC researchers have incorporated psychological theories of emotion and how these theories affect research questions, methods, results, and their interpretations. In this way, models and methods can be compared, and emerging insights from various disciplines can be more expertly integrated.

/pdf/affect-detection-an-interdisciplinary-review-of-models-49pcvdc6da.pdf

Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications

https://hal.inria.fr/inria-00459653/document

A survey of vision-based methods for action representation, segmentation and recognition

http://www.ffri.hr/~ibrdar/komunikacija/seminari/Ververidis,%202006%20-%20Emotional%20speech%20recognition.pdf

Emotional speech recognition: Resources, features, and methods

In this contribution we introduce speech emotion recognition by use of continuous hidden Markov models. Two methods are propagated and compared throughout the paper. Within the first method a global statistics framework of an utterance is classified by Gaussian mixture models using derived features of the raw pitch and energy contour of the speech signal. A second method introduces increased temporal complexity applying continuous hidden Markov models considering several states using low-level instantaneous features instead of global statistics. The paper addresses the design of working recognition engines and results achieved with respect to the alluded alternatives. A speech corpus consisting of acted and spontaneous emotion samples in German and English language is described in detail. Both engines have been tested and trained using this equivalent speech corpus. Results in recognition of seven discrete emotions exceeded 86% recognition rate. As a basis of comparison the similar judgment of human deciders classifying the same corpus at 79.8% recognition rate was analyzed.

Hidden Markov model-based speech emotion recognition

In this paper we introduce a novel approach to the combination of acoustic features and language information for a most robust automatic recognition of a speaker's emotion. Seven discrete emotional states are classified throughout the work. Firstly a model for the recognition of emotion by acoustic features is presented. The derived features of the signal-, pitch-, energy, and spectral contours are ranked by their quantitative contribution to the estimation of an emotion. Several different classification methods including linear classifiers, Gaussian mixture models, neural nets, and support vector machines are compared by their performance within this task. Secondly an approach to emotion recognition by the spoken content is introduced applying belief network based spotting for emotional key-phrases. Finally the two information sources are integrated in a soft decision fusion by using a neural net. The gain is evaluated and compared to other advances. Two emotional speech corpora used for training and evaluation are described in detail and the results achieved applying the propagated novel advance to speaker emotion recognition are presented and discussed.

Speech emotion recognition combining acoustic features and linguistic information in a hybrid support vector machine-belief network architecture

Herein we present a comparison of novel concepts for a robust fusion of prosodic and verbal cues in speech emotion recognition. Thereby 276 acoustic features are extracted out of a spoken phrase. For linguistic content analysis we use the Bag-of-Words text representation. This allows for integration of acoustic and linguistic features within one vector prior to a final classification. Extensive feature selection by filter- and wrapper based methods is fulfilled. Likewise optimal sets via SVM-SFFS and single feature relevance by information gain ratio calculation are presented. Overall classification is realised by diverse ensemble approaches. Among base classifiers Kernel Machines, Decision Trees, Bayesian classifiers, and memory-based learners are found. Acoustics only tests ran on a database comprising 39 speakers for speaker independent accuracy analysis. Additionally the public Berlin Emotional Speech database is used. A further database of 4,221 movie related phrases forms the basis of acoustic and linguistic information analysis evaluation. Overall remarkable performance in the discrimination of seven discrete emotions could be observed.

Speaker Independent Emotion Recognition by Early Fusion of Acoustic and Linguistic Features within Ensembles

We suggest a novel approach to affect recognition based on acoustic and linguistic analysis of spoken utterances. In order to achieve maximum discrimination power within robust integration of these information sources, a fusion on the feature level is introduced. Considering classification, we use meta-classifiers, such as StackingC and Boosting, for a stabilized performance, and a combination of classifiers within ensembles. Extensive comparisons of diverse base-classifiers, including support vector machines, neural networks, stochastic models, and decision trees, are fulfilled. 381 acoustic features are extracted and their relevance is calculated by a sequential forward floating search in comparison to reduction by principal component analysis. Several variants for linguistic feature calculation are described and ranked, including bunch-of-words, n-grams, salience, and mutual information. Furthermore, reduction by stopping and stemming or filter-based selection methods is evaluated, reducing 2,334 linguistic features. Seven discrete emotions described in the MPEG-4 standard are recognized within an existing recognition engine. The presented results are based on two large databases of 4,336 acted and real emotion samples from movies, chat and car interaction dialogues. A significant gain and an outstanding overall performance are observed by this novel fusion and use of ensembles.

Meta-classifiers in acoustic and linguistic feature fusion-based affect recognition

A new and general stochastic approach to find and identify dynamic gestures in continuous video streams is presented. Hidden Markov models (HMMs) are used to solve this combined problem of temporal segmentation and classification in an integral way. Basically, an improved normalized Viterbi algorithm allows one to continuously observe the output scores of the HMMs at every time step. Characteristic peaks in the output scores of the respective models indicate the presence of gestures. Our experiments in the domain of hand gesture spotting provided excellent recognition results and very low temporal detection delays.

M. Lang

Papers

Hidden Markov model-based speech emotion recognition

Speech emotion recognition combining acoustic features and linguistic information in a hybrid support vector machine-belief network architecture

Speaker Independent Emotion Recognition by Early Fusion of Acoustic and Linguistic Features within Ensembles

Meta-classifiers in acoustic and linguistic feature fusion-based affect recognition

Spotting dynamic hand gestures in video image sequences using hidden Markov models