It is commonly assumed that a person’s emotional state can be readily inferred from his or her facial movements, typically called emotional expressions or facial expressions. This assumption influences legal judgments, policy decisions, national security protocols, and educational practices; guides the diagnosis and treatment of psychiatric illness, as well as the development of commercial applications; and pervades everyday social interactions as well as research in other scientific fields such as artificial intelligence, neuroscience, and computer vision. In this article, we survey examples of this widespread assumption, which we refer to as the common view, and we then examine the scientific evidence that tests this view, focusing on the six most popular emotion categories used by consumers of emotion research: anger, disgust, fear, happiness, sadness, and surprise. The available scientific evidence suggests that people do sometimes smile when happy, frown when sad, scowl when angry, and so on, as proposed by the common view, more than what would be expected by chance. Yet how people communicate anger, disgust, fear, happiness, sadness, and surprise varies substantially across cultures, situations, and even across people within a single situation. Furthermore, similar configurations of facial movements variably express instances of more than one emotion category. In fact, a given configuration of facial movements, such as a scowl, often communicates something other than an emotional state. Scientists agree that facial movements convey a range of information and are important for social communication, emotional or otherwise. But our review suggests an urgent need for research that examines how people actually move their faces to express emotions and other social information in the variety of contexts that make up everyday life, as well as careful study of the mechanisms by which people perceive instances of emotion in one another. We make specific research recommendations that will yield a more valid picture of how people move their faces to express emotions and how they infer emotional meaning from facial movements in situations of everyday life. This research is crucial to provide consumers of emotion research with the translational information they require.

https://journals.sagepub.com/doi/pdf/10.1177/1529100619832930

Emotional Expressions Reconsidered: Challenges to Inferring Emotion From Human Facial Movements:

Synchrony refers to individuals' temporal coordination during social interactions. The analysis of this phenomenon is complex, requiring the perception and integration of multimodal communicative signals. The evaluation of synchrony has received multidisciplinary attention because of its role in early development, language learning, and social connection. Originally studied by developmental psychologists, synchrony has now captured the interest of researchers in such fields as social signal processing, robotics, and machine learning. This paper emphasizes the current questions asked by synchrony evaluation and the state-of-the-art related methods. First, we present definitions and functions of synchrony in youth and adulthood. Next, we review the noncomputational and computational approaches of annotating, evaluating, and modeling interactional synchrony. Finally, the current limitations and future research directions in the fields of developmental robotics, social robotics, and clinical studies are discussed.

/pdf/interpersonal-synchrony-a-survey-of-evaluation-methods-1u1qbicpkx.pdf

Interpersonal Synchrony: A Survey of Evaluation Methods across Disciplines

Conversational agents (CAs) are software-based systems designed to interact with humans using natural language and have attracted considerable research interest in recent years. Following the Computers Are Social Actors paradigm, many studies have shown that humans react socially to CAs when they display social cues such as small talk, gender, age, gestures, or facial expressions. However, research on social cues for CAs is scattered across different fields, often using their specific terminology, which makes it challenging to identify, classify, and accumulate existing knowledge. To address this problem, we conducted a systematic literature review to identify an initial set of social cues of CAs from existing research. Building on classifications from interpersonal communication theory, we developed a taxonomy that classifies the identified social cues into four major categories (i.e., verbal, visual, auditory, invisible) and ten subcategories. Subsequently, we evaluated the mapping between the identified social cues and the categories using a card sorting approach in order to verify that the taxonomy is natural, simple, and parsimonious. Finally, we demonstrate the usefulness of the taxonomy by classifying a broader and more generic set of social cues of CAs from existing research and practice. Our main contribution is a comprehensive taxonomy of social cues for CAs. For researchers, the taxonomy helps to systematically classify research about social cues into one of the taxonomy's categories and corresponding subcategories. Therefore, it builds a bridge between different research fields and provides a starting point for interdisciplinary research and knowledge accumulation. For practitioners, the taxonomy provides a systematic overview of relevant categories of social cues in order to identify, implement, and test their effects in the design of a CA.

A Taxonomy of Social Cues for Conversational Agents

The goal of this review was to provide a brief overview of recent developments in the domain of emotional mimicry research. We argue that emotional signals are intrinsically meaningful within a social relationship, which is crucial for understanding the functionality and boundary conditions of emotional mimicry. On the basis of a review of the literature on facial mimicry of emotion displays, we conclude that the classic matched motor hypothesis does not hold for emotional mimicry. We alternatively propose a contextual view of emotional mimicry, which states that emotional mimicry depends on the social context: we only mimic emotional signals that are interpreted to promote affiliation goals and not necessarily what we see. As a further consequence, we are less likely to mimic strangers and we do not mimic people we do not like nor emotions that signal antagonism.

Emotional mimicry: why and when we mimic emotions

The opinion mining and human-agent interaction communities are currently addressing sentiment analysis from different perspectives that comprise, on the one hand, disparate sentiment-related phenomena and computational representations, and on the other hand, different detection and dialog management methods. In this paper we identify and discuss the growing opportunities for cross-disciplinary work that may increase individual advances. Sentiment/opinion detection methods used in human-agent interaction are indeed rare and, when they are employed, they are not different from the ones used in opinion mining and consequently not designed for socio-affective interactions (timing constraint of the interaction, sentiment analysis as an input and an output of interaction strategies). To support our claims, we present a comparative state of the art which analyzes the sentiment-related phenomena and the sentiment detection methods used in both communities and makes an overview of the goals of socio-affective human-agent strategies. We propose then different possibilities for mutual benefit, specifying several research tracks and discussing the open questions and prospects. To show the feasibility of the general guidelines proposed we also approach them from a specific perspective by applying them to the case of the Greta embodied conversational agents platform and discuss the way they can be used to make a more significative sentiment analysis for human-agent interactions in two different use cases: job interviews and dialogs with museum visitors.

Sentiment Analysis: From Opinion Mining to Human-Agent Interaction

Laughter is an important communicative signal in human-human communication. However, very few attempts have been made to model laughter animation synthesis for virtual characters. This paper reports our work to model hilarious laughter. We have developed a generator for face and body motions that takes as input the sequence of pseudo-phonemes of laughter and each pseudo-phoneme's duration time. Lip and jaw movements are further driven by laughter prosodic features. The proposed generator first learns the relationship between input signals (pseudo-phoneme and acoustic features) and human motions; then the learnt generator can be used to produce automatically laughter animation in real time. Lip and jaw motion synthesis is based on an extension of Gaussian Models, the contextual Gaussian Model. Head and eyebrow motion synthesis is based on selecting and concatenating motion segments from motion capture data of human laughter while torso and shoulder movements are driven from head motion by a PD controller. Our multimodal behaviors generator of laughter has been evaluated through perceptive study involving the interaction of a human and an agent telling jokes to each other.

Laughter animation synthesis

GRETA: Towards an interactive conversational virtual Companion

https://escholarship.org/content/qt1np4f1br/qt1np4f1br.pdf?t=os603p

Beyond backchannels: co-construction of dyadic stancce by reciprocal reinforcement of smiles between virtual agents.

Synchrony is claimed by psychology as a crucial parameter of any social interaction. In dialog interactions, the synchrony between non-verbal behaviours of interactants is claimed to account for the quality of the interaction: to give to human a feeling of natural interaction, an agent must be able to synchronise on appropriate time. The synchronisation occurring during non-verbal iteractions has recently been modelised as a phenonomenon emerging from the coupling between interactants. We propose here, and test in simulation, a dynamical model of verbal communication which links the emergence of synchrony between non-verbal behaviours to the level of meaning exchanged through words by interactants: if partners of a dyad understand each other, synchrony emerges, whereas if they do not understand, synchrony is disrupted. In addition to retrieve the fact that synchrony emergence within a dyad of agents depends on their level of shared understanding, our tests pointed two noteworthy properties of synchronisation phenomenons: first, as well as synchrony accounts for mutual understanding and good interaction, di-synchrony accounts for misunderstanding; second, synchronisation and di-synchronisation emerging from mutual understanding are very quick phenomenons.

SHARED UNDERSTANDING AND SYNCHRONY EMERGENCE - Synchrony as an Indice of the Exchange of Meaning between Dialog Partners

In this paper we propose an extension of the current SAIBA architecture The new parts of the architecture should manage the generation of Embodied Conversational Agents’ reactive behaviors during an interaction with users both while speaking and listening General Terms

Ken Prepin

Papers

Laughter animation synthesis

GRETA: Towards an interactive conversational virtual Companion

Beyond backchannels: co-construction of dyadic stancce by reciprocal reinforcement of smiles between virtual agents.

SHARED UNDERSTANDING AND SYNCHRONY EMERGENCE - Synchrony as an Indice of the Exchange of Meaning between Dialog Partners

Reactive behaviors in SAIBA architecture