A review of affective computing

TLDR: This first of its kind, comprehensive literature review of the diverse field of affective computing focuses mainly on the use of audio, visual and text information for multimodal affect analysis, and outlines existing methods for fusing information from different modalities.

About: This article is published in Information Fusion.The article was published on 2017-09-01 and is currently open access. It has received 969 citations till now. The article focuses on the topics: Affective computing & Modality (human–computer interaction).

Figures

Figure 3: The Hourglass of Emotions.

Figure 4: C3D for extracting spatio-temporal generic video features.

Table 6: Study characteristics of recent papers on multimodal analysis.

Figure 7: Sentic computing framework.

Table 4: State of the art of visual-audio affect recognition

Figure 12: Percentage of research articles in multimodal affect analysis using different fusion methods over the years (Legenda: F=Feature-Level Fusion; D=Decision-Level Fusion; H=Hybrid Fusion; M=Model-Based Fusion).

Frequently asked questions

Q1. What contributions have the authors mentioned in the paper "A review of affective computing: from unimodal analysis to multimodal fusion" ?

This is the primary motivation behind their first of its kind, comprehensive literature review of the diverse field of affective computing. Furthermore, existing literature surveys lack a detailed discussion of state of the art in multimodal affect analysis frameworks, which this review aims to address. In this paper, the authors focus mainly on the use of audio, visual and text information for multimodal affect analysis, since around 90 % of the relevant literature appears to cover these three modalities. As part of this review, the authors carry out an extensive study of different categories of state-of-the-art fusion techniques, followed by a critical analysis of potential performance improvements with multimodal analysis compared to unimodal analysis. A comprehensive overview of these two complementary fields aims to form the building blocks for readers, to better understand this challenging and exciting research field. 

Q2. What future works have the authors mentioned in the paper "A review of affective computing: from unimodal analysis to multimodal fusion" ?

One important area of future research is to investigate novel approaches for advancing their understanding of the temporal dependency between utterances, i. e., the effect of utterance at time t on the utterance at time t+1. The progress in text classification research can play a major role in future of the multimodal affect analysis research. Future research should focus on answering this question. The use of deep learning for multimodal fusion can also be an important future work. 

Q3. What is the primary advantage of analyzing videos over textual analysis?

The primary advantage of analyzing videos over textual analysis, for detecting emotions and sentiments from opinions, is the surplus of behavioral cues. 

Q4. What was the acoustic feature used to generate the feature representation of the entire dataset?

For acoustic features, low-level acoustic features were extracted at frame level on each utterance and used to generate feature representation of the entire dataset, using the OpenSMILE toolkit. 

Q5. What are the common unsupervised methods for sentiment analysis?

Whilst machine learning methods, for supervised training of the sentiment analysis system, are predominant in literature, a number of unsupervised methods such as linguistic patterns can also be found. 

Q6. What is the main channel for forming an impression of the subject’s present state of mind?

Across the ages of people involved, and the nature of conversations, facial expressions are the primary channel for forming an impression of the subject’s present state of mind. 

Q7. What was the effect of the feature adaptation scheme on the emotion recognition system?

The results on uncontrolled recordings (i.e., speech downloaded from a video-sharing website) revealed that the feature adaptation scheme significantly improved the unweighted and weighted accuracies of the emotion recognition system. 

Q8. What is the percentage of studies that report visual modality as superior to audio?

In their literature survey, the authors have found more than 90% of studies reported visual modality as superior to audio and other modalities. 

Q9. How accurate was the synchronization of the audio and video signals?

To accommodate research in audio-visual fusion, the audio and video signals were synchronized with an accuracy of 25micro-seconds.