Thanks to the decreasing cost of whole-body sensing technology and its increasing reliability, there is an increasing interest in, and understanding of, the role played by body expressions as a powerful affective communication channel. The aim of this survey is to review the literature on affective body expression perception and recognition. One issue is whether there are universal aspects to affect expression perception and recognition models or if they are affected by human factors such as culture. Next, we discuss the difference between form and movement information as studies have shown that they are governed by separate pathways in the brain. We also review psychological studies that have investigated bodily configurations to evaluate if specific features can be identified that contribute to the recognition of specific affective states. The survey then turns to automatic affect recognition systems using body expressions as at least one input modality. The survey ends by raising open questions on data collecting, labeling, modeling, and setting benchmarks for comparing automatic recognition systems.

https://web4.cs.ucl.ac.uk/uclic/people/n.berthouze/KleinsmithBerthouze12.pdf

Affective Body Expression Perception and Recognition: A Survey

In this paper, we report a real-time gesture driven interactive system with multimodal feedback for performing arts, especially dance. The system consists of two major parts., a gesture recognition engine and a multimodal feedback engine. The gesture recognition engine provides real-time recognition of the performer's gesture based on the 3D marker coordinates from a marker-based motion capture system. According to the recognition results, the multimodal feedback engine produces associated visual and audio feedback to the performer. This interactive system is simple to implement and robust to errors in 3D marker data. Satisfactory interactive dance performances have been successfully created and presented using the reported system

/pdf/a-gesture-driven-multimodal-interactive-dance-system-29cqb4bvu4.pdf

A gesture-driven multimodal interactive dance system

The goal of the EyesWeb project is to develop a modular system for the real-time analysis of body movement and gesture. Such information can be used to control and generate sound, music, and visual media, and to control actuators (e.g., robots). Another goal of the project is to explore and develop models of interaction by extending music language toward gesture and visual languages, with a particular focus on the understanding of affect and expressive content in gesture. For example, we attempt to distinguish the expressive content from two instances of the same movement

EyesWeb: Toward Gesture and Affect Recognition in Interactive Dance and Music Systems

Since telecommunication can never equal the richness of face-to-face interaction on its own terms, the most interesting examples of networked music go beyond the paradigm of musicians playing together in a virtual room. The Open Sound Control protocol has facilitated dozens of such innovative networked music projects. First the protocol itself is described, followed by some theoretical limits on communication latency and what they mean for music making. Then a representative list of some of the projects that take advantage of the protocol is presented, describing each project in terms of the paradigm of musical interaction that it provides.

/pdf/open-sound-control-an-enabling-technology-for-musical-31bief5974.pdf

Open Sound Control: an enabling technology for musical networking

This paper presents ongoing research on the modelling of expressive gesture in multimodal interaction and on the development of multimodal interactive systems explicitly taking into account the role of non-verbal expressive gesture in the communication process. In this perspective, a particular focus is on dance and music as first-class conveyors of expressive and emotional content. Research outputs include (i) computational models of expressive gesture, (ii) validation by means of continuous ratings on spectators exposed to real artistic stimuli, and (iii) novel hardware and software components for the EyesWeb open platform (www.eyesweb.org), such as the recently developed Expressive Gesture Processing Library. The paper starts with a definition of expressive gesture. A unifying framework for the analysis of expressive gesture is then proposed. Finally, two experiments on expressive gesture in dance and music are discussed. This research work has been supported by the EU IST project MEGA (Multisensory Expressive Gesture Applications, www.megaproject.org) and the EU MOSART TMR Network.

Multimodal Analysis of Expressive Gesture in Music and Dance Performances

The EyesWeb project concerns the development of a system for real-time analysis of full-body movement and gesture of one or more humans, with a particular focus on affect or emotion content. Such information is used to control and generate sound, music, visual media, and to control actuators (e.g., robots). The main goal is to explore extensions of music language toward gesture and visual languages. The paper describes the state of the art of the project, the design of the current EyesWeb system modules (hardware and software), their experiments in public events, and discusses ongoing developments.

EyesWeb-toward gesture and affect recognition in dance/music interactive systems

We propose a method to realize extractions of emotional information from human gestures in real time. The basic ideas of understanding the human motion is based on Rudolf Laban's theoretical work. The system we developed consists of two parts: image processing; and motion analysis. The former part works to obtain human movement by using two color CCD cameras, while the latter is involved in mapping physical parameters onto emotional information. We are trying to realize a sort of "KANSEI extractor" as the last goal. Humans can understand the emotional mode, such as happy, angry and sad, in which mode the actor performs. It seems that human movement carries emotional information although it is difficult to express the emotional effects with particular physical parameters. From this point of view, we aim to extract emotional information in dance performance with the aid of a computer system based on Laban's theory of movement. In our proposed method, human motions are mapped onto low-dimensional parameter space in real time introducing two spaces: the general space- and personal space.

KANSEI analysis of dance performance

This paper presents some our recent research on computational models and algorithms for real-time analysis of full-body human movement. The focus here is on techniques to extract in real-time expressive cues relevant to KANSEI and emotional content in human expressive gesture, e.g., in dance and music performances. Expressive gesture can contribute to new perspectives for the design of interactive systems. The EyesWeb open software platform is a main concrete result from our research work. EyesWeb is used in interactive applications, including music and other artistic productions, museum interactive exhibits, therapy and rehabilitation, based on the paradigm of expressive gesture. EyesWeb is freely available from www.eyesweb.org.

Interactive systems design: a KANSEI-based approach

We present recent developments on EyesWeb, a research project aiming at providing models and tools for integrating movement, music and visual languages in a multimodal perspective. The project includes a hardware and software platform to support the user (i) in the development and experimenting of computational models of expressive content communication and of gesture mapping strategies, and (ii) in fast development and experiment cycles of interactive performance setups. An intuitive visual programming language allows to map – at different levels gestural components with integrated music, visual, and mobile scenery. System original features, such as its realtime patch scheduler supporting active modules and userdefined datatypes, are shortly described in the paper. The EyesWeb platform consists of a serie of integrated hardware and software modules which can be easily interconnected and extended by users.

Riccardo Trocca

Papers

EyesWeb: Toward Gesture and Affect Recognition in Interactive Dance and Music Systems

EyesWeb-toward gesture and affect recognition in dance/music interactive systems

KANSEI analysis of dance performance

Interactive systems design: a KANSEI-based approach

A real-time platform for interactive dance and music systems