Social Signal Processing is the research domain aimed at bridging the social intelligence gap between humans and machines. This paper is the first survey of the domain that jointly considers its three major aspects, namely, modeling, analysis, and synthesis of social behavior. Modeling investigates laws and principles underlying social interaction, analysis explores approaches for automatic understanding of social exchanges recorded with different sensors, and synthesis studies techniques for the generation of social behavior via various forms of embodiment. For each of the above aspects, the paper includes an extensive survey of the literature, points to the most important publicly available resources, and outlines the most fundamental challenges ahead.

/pdf/bridging-the-gap-between-social-animal-and-unsocial-machine-28vclv3t3o.pdf

Bridging the Gap between Social Animal and Unsocial Machine: A Survey of Social Signal Processing

https://publications.idiap.ch/downloads/papers/2010/Gatica-Perez_JIVC_2009.pdf

Automatic nonverbal analysis of social interaction in small groups

Computer vision for assistive technologies

This paper targets the automatic detection of personality traits in a meeting environment by means of audio and visual features; information about the relational context is captured by means of acoustic features designed to that purpose. Two personality traits are considered: Extraversion (from the Big Five) and the Locus of Control. The classification task is applied to thin slices of behaviour, in the form of 1-minute sequences. SVM were used to test the performances of several training and testing instance setups, including a restricted set of audio features obtained through feature selection. The outcomes improve considerably over existing results, provide evidence about the feasibility of the multimodal analysis of personality, the role of social context, and pave the way to further studies addressing different features setups and/or targeting different personality traits.

Multimodal recognition of personality traits in social interactions

A digital residential entertainment system is disclosed recording video data of an event. The apparatuses include a processor communicating with memory. The memory stores video data of the event captured by a camera, and the video data includes a series of picture frames of the event. A loop buffer also stores video data of the event captured by the camera. A rule-based engine stored in the memory uses a set of rules to store the contents of the loop buffer in the memory. The apparatus utilizes the loop buffer to provide video data prior to occurrence of the event.

Digital residential entertainment system

This paper describes an automatically annotated multimodal corpus of multi-party meetings. The corpus provides for each subject involved in the experimental sessions information on her/his social behavior and personality traits, as well as audiovisual cues (speech rate, pitch and energy, head orientation, head, hand and body fidgeting). The corpus is based on the audio and video recordings of thirteen sessions, which took place in a lab setting equipped with cameras and microphones. Our main concern in collecting this corpus was to investigate the possibility of creating a system capable of automatically analyzing social behaviors and predicting personality traits using audio-visual cues.

Multimodal corpus of multi-party meetings for automatic social behavior analysis and personality traits detection

. During the last two decades we have witnessed great improvements in ICT hardware and software technologies. Three-dimensional content is starting to become commonplace now in many applications. Although for many years 3D technologies have been used in the generation of assets by researchers and experts, nowadays these tools are starting to become commercially available to every citizen. This is especially the case for smartphones, that are powerful enough and sufficiently widespread to perform a huge variety of activities (e.g. paying, calling, communication, photography, navigation, localization, etc.), including just very recently the possibility of running 3D reconstruction pipelines. The REPLICATE project is tackling this particular issue, and it has an ambitious vision to enable ubiquitous 3D creativity via the development of tools for mobile 3D-assets generation on smartphones/tablets. This article presents the REPLICATE project’s concept and some of the ongoing activities, with particular attention being paid to advances made in the first year of work. Thus the article focuses on the system architecture definition, selection of optimal frames for 3D cloud reconstruction, automated generation of sparse and dense point clouds, mesh modelling techniques and post-processing actions. Experiments so far were concentrated on indoor objects and some simple heritage artefacts, however, in the long term we will be targeting a larger variety of scenarios and communities.

/pdf/a-smartphone-based-3d-pipeline-for-the-creative-industry-the-fj3j0bo90l.pdf

A smartphone-based 3d pipeline for the creative industry– the replicate eu project

This paper focuses on the integration of acoustic and visual information for people tracking. The system presented relies on a probabilistic framework within which information from multiple sources is integrated at an intermediate stage. An advantage of the method proposed is that of using a generative approach which supports easy and robust integration of multi source information by means of sampled projection instead of triangulation. The system described has been developed in the EU funded CHIL Project research activities. Experimental results from the CLEAR evaluation workshop are reported.

A generative approach to audio-visual person tracking

This article presents a pipeline that enables multiple users to collaboratively acquire images with monocular smartphones and derive a 3D point cloud using a remote reconstruction server. A set of key images are automatically selected from each smartphone's camera video feed as multiple users record different viewpoints of an object, concurrently or at different time instants. Selected images are automatically processed and registered with an incremental Structure from Motion (SfM) algorithm in order to create a 3D model. Our incremental SfM approach enables on-the-fly feedback to the user to be generated about current reconstruction progress. Feedback is provided in the form of a preview window showing the current 3D point cloud, enabling users to see if parts of a surveyed scene need further attention/coverage whilst they are still in situ. We evaluate our 3D reconstruction pipeline by performing experiments in uncontrolled and unconstrained real-world scenarios. Datasets are publicly available.

Cloud-based collaborative 3D reconstruction using smartphones

This paper describes a real-time system developed for the derivation of low-level visual cues targeted at the recognition of simple hand, head and body gestures. A novel, adaptive background subtraction technique is presented together with a tool for monitoring repetitive movements, e.g. fidgeting. To monitor subtle body movements in an unconstrained environment, active cameras with pan, tilt and zoom capabilities must be employed to track an individuals actions more closely. This paper then explores a means of detecting small and large scale human activity within images produced from active cameras that may be reoriented during monitoring.

Paul Chippendale

Papers

Multimodal corpus of multi-party meetings for automatic social behavior analysis and personality traits detection

A smartphone-based 3d pipeline for the creative industry– the replicate eu project

A generative approach to audio-visual person tracking

Cloud-based collaborative 3D reconstruction using smartphones

Towards Automatic Body Language Annotation