Data Mining Practical Machine Learning Tools and Techniques

Emotion and Adaptation

Since emotions are expressed through a combination of verbal and non-verbal channels, a joint analysis of speech and gestures is required to understand expressive human communication. To facilitate such investigations, this paper describes a new corpus named the “interactive emotional dyadic motion capture database” (IEMOCAP), collected by the Speech Analysis and Interpretation Laboratory (SAIL) at the University of Southern California (USC). This database was recorded from ten actors in dyadic sessions with markers on the face, head, and hands, which provide detailed information about their facial expressions and hand movements during scripted and spontaneous spoken communication scenarios. The actors performed selected emotional scripts and also improvised hypothetical scenarios designed to elicit specific types of emotions (happiness, anger, sadness, frustration and neutral state). The corpus contains approximately 12 h of data. The detailed motion capture information, the interactive setting to elicit authentic emotions, and the size of the database make this corpus a valuable addition to the existing databases in the community for the study and modeling of multimodal and expressive human communication.

/pdf/iemocap-interactive-emotional-dyadic-motion-capture-database-2ffyt1sb5i.pdf

IEMOCAP: interactive emotional dyadic motion capture database

The cultural origins of human cognition

Creativity: Flow and the Psychology of Discovery and Invention

During spontaneous conversations, multiple modalities such as gesture, posture and gaze are combined in sophisticated ways for different functions such as spatial references. In the French research community, there is a lack of spontaneous multimodal corpora for the French language. This paper describes the multi-level scheme that we have defined for the annotation of gesture, posture and gaze. We explain how we applied it for the annotation of a corpus of spontaneous French conversations. Two types of analyses were made on the resulting annotations. Firstly, intercoder agreement was computed on the annotations of gesture space. Secondly, we clustered chunks of postures into categories. Such research will enable the joint study of multiple nonverbal modalities such as the relations between gesture semantics and posture.

https://hal.archives-ouvertes.fr/hal-00488832/document

Multi-level Annotations of Nonverbal Behaviors in French Spontaneous Conversation

Recent advances in the specification of the multimodal behavior of Embodied Conversational Agents (ECA) have proposed a direct and deterministic one-step mapping from high-level specifications of dialog state or agent emotion onto low-level specifications of the multimodal behavior to be displayed by the agent (e.g. facial expression, gestures, vocal utterance). The difference of abstraction between these two levels of specification makes difficult the definition of such a complex mapping. In this paper we propose an intermediate level of specification based on combinations between modalities (e.g. redundancy, complementarity). We explain how such intermediate level specifications can be described using XML in the case of deictic expressions. We define algorithms for parsing such descriptions and generating the corresponding multimodal behavior of 2D cartoon-like conversational agents. Some random selection has been introduced in these algorithms in order to induce some "natural variations" in the agent's behavior. We conclude on the usefulness of this approach for the design of ECA.

Algorithms for controlling cooperation between output modalities in 2D embodied conversational agents

The present study aimed to examine multidimensional factors that contribute to a poor performance in a public speakingtask. An adapted version of the Trier Social Stress Test (TSST) as used to elicit psychosocial stress among 43 university students andmultidimensional assessments were involved to investigate acute stress responses by psychological measures (i.e. personality, affect,appraisal, coping), physiological measures (i.e. cortisol; Dehydroepiandrosterone: DHEA; ratio of cortisol/DHEA) and behaviouralmeasures (voice, postural control). Our results showed that psychological factors seemed to be the most sensitive to stressperformance. A mediation effect was detected between psychological factors and objective performance. Cortisol to DHEA ratio alsoshowed to be associated with speaking performance. This study added evidence to the literature with regards to a multidimensionalway to study human stress response and may help individuals use functional coping to improve their performance and better adapt tostressful situations.

/pdf/predicting-a-failure-of-public-speaking-performance-using-30mwjv1ugs.pdf

Predicting a Failure of Public Speaking Performance Using Multidimensional Assessment

In the context of designing multimodal social interactions for Human-Computer Interaction and for Computer-Mediated Communication, we conducted an experimental study to investigate how participants combine voice expressions with tactile stimulation to evaluate emotional valence (EV). In this study, audio and tactile stimuli were presented separately, and then presented together. Audio stimuli comprised positive and negative voice expressions, and tactile stimuli consisted of different levels of air jet tactile stimulation performed on the arm of the participants. Participants were asked to evaluate communicated EV on a continuous scale. Information Integration Theory was used to model multimodal valence perception process. Analyses showed that participants generally integrated both sources of information to evaluate EV. The main integration rule was averaging rule. The predominance of a modality over the other modality was specific to each individual.

/pdf/modeling-emotional-valence-integration-from-voice-and-touch-265awr8zdm.pdf

Modeling Emotional Valence Integration From Voice and Touch

. People react with Rapid Facial Reactions (RFRs) when presented with human facial emotional expressions. Recent studies show that RFRs are not always congruent with emotional cues. The processes underlying RFRs are still being debated. In our study described herein, we manipulate the context of perception and its influence on RFRs. We use a subliminal affective priming task with emotional labels. Facial electromyography (EMG) (frontalis, corrugator, zygomaticus, and depressor) was recorded while participants observed static facial expressions (joy, fear, anger, sadness, and neutral expression) preceded/not preceded by a subliminal word (JOY, FEAR, ANGER, SADNESS, or NEUTRAL). For the negative facial expressions, when the priming word was congruent with the facial expression, participants displayed congruent RFRs (mimicry). When the priming word was incongruent, we observed a suppression of mimicry. Happiness was not affected by the priming word. RFRs thus appear to be modulated by the context and...

Jean-Claude Martin

Papers

Multi-level Annotations of Nonverbal Behaviors in French Spontaneous Conversation

Algorithms for controlling cooperation between output modalities in 2D embodied conversational agents

Predicting a Failure of Public Speaking Performance Using Multidimensional Assessment

Modeling Emotional Valence Integration From Voice and Touch

Suppression of Facial Mimicry of Negative Facial Expressions in an Incongruent Context