Automated affective computing in the wild setting is a challenging problem in computer vision. Existing annotated databases of facial expressions in the wild are small and mostly cover discrete emotions (aka the categorical model). There are very limited annotated facial databases for affective computing in the continuous dimensional model (e.g., valence and arousal). To meet this need, we collected, annotated, and prepared for public distribution a new database of facial emotions in the wild (called AffectNet). AffectNet contains more than 1,000,000 facial images from the Internet by querying three major search engines using 1250 emotion related keywords in six different languages. About half of the retrieved images were manually annotated for the presence of seven discrete facial expressions and the intensity of valence and arousal. AffectNet is by far the largest database of facial expression, valence, and arousal in the wild enabling research in automated facial expression recognition in two different emotion models. Two baseline deep neural networks are used to classify images in the categorical model and predict the intensity of valence and arousal. Various evaluation metrics show that our deep neural network baselines can perform better than conventional machine learning methods and off-the-shelf facial expression recognition systems.

AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild

Gilles Fauconnier & Mark Turner, " The way we think: conceptual blending and the mind's hidden complexities"

The automatic recognition of spontaneous emotions from speech is a challenging task. On the one hand, acoustic features need to be robust enough to capture the emotional content for various styles of speaking, and while on the other, machine learning algorithms need to be insensitive to outliers while being able to model the context. Whereas the latter has been tackled by the use of Long Short-Term Memory (LSTM) networks, the former is still under very active investigations, even though more than a decade of research has provided a large set of acoustic descriptors. In this paper, we propose a solution to the problem of ‘context-aware’ emotional relevant feature extraction, by combining Convolutional Neural Networks (CNNs) with LSTM networks, in order to automatically learn the best representation of the speech signal directly from the raw time representation. In this novel work on the so-called end-to-end speech emotion recognition, we show that the use of the proposed topology significantly outperforms the traditional approaches based on signal processing techniques for the prediction of spontaneous and natural emotions on the RECOLA database.

/pdf/adieu-features-end-to-end-speech-emotion-recognition-using-a-2iy5o26o9g.pdf

Adieu features? End-to-end speech emotion recognition using a deep convolutional recurrent network

▶ Addresses the accessibility, usability, and, ultimately, acceptability of Information Society Technologies by anyone, anywhere, at anytime, and through any media and device. ▶ Focuses on theoretical, methodological, and empirical research, of both technological and non-technological nature. ▶ Features papers that report on theories, methods, tools, empirical results, reviews, case studies, and best-practice examples.

Universal access in the information society

Analyzing human multimodal language is an emerging area of research in NLP Intrinsically this language is multimodal (heterogeneous), sequential and asynchronous; it consists of the language (words), visual (expressions) and acoustic (paralinguistic) modalities all in the form of asynchronous coordinated sequences From a resource perspective, there is a genuine need for large scale datasets that allow for in-depth studies of this form of language In this paper we introduce CMU Multimodal Opinion Sentiment and Emotion Intensity (CMU-MOSEI), the largest dataset of sentiment analysis and emotion recognition to date Using data from CMU-MOSEI and a novel multimodal fusion technique called the Dynamic Fusion Graph (DFG), we conduct experimentation to exploit how modalities interact with each other in human multimodal language Unlike previously proposed fusion techniques, DFG is highly interpretable and achieves competative performance when compared to the previous state of the art

/pdf/multimodal-language-analysis-in-the-wild-cmu-mosei-dataset-4rb7cjcv5e.pdf

Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph

We present in this paper a new multimodal corpus of spontaneous collaborative and affective interactions in French: RECOLA, which is being made available to the research community. Participants were recorded in dyads during a video conference while completing a task requiring collaboration. Different multimodal data, i.e., audio, video, ECG and EDA, were recorded continuously and synchronously. In total, 46 participants took part in the test, for which the first 5 minutes of interaction were kept to ease annotation. In addition to these recordings, 6 annotators measured emotion continuously on two dimensions: arousal and valence, as well as social behavior labels on live dimensions. The corpus allowed us to take self-report measures of users during task completion. Methodologies and issues related to affective corpus construction are briefly reviewed in this paper. We further detail how the corpus was constructed, i.e., participants, procedure and task, the multimodal recording setup, the annotation of data and some analysis of the quality of these annotations.

/pdf/introducing-the-recola-multimodal-corpus-of-remote-2xjay9kg47.pdf

Introducing the RECOLA multimodal corpus of remote collaborative and affective interactions

https://doc.rero.ch/record/28590/files/Sonderegger_Sauer2010.pdf

The influence of design aesthetics in usability testing: effects on user performance and perceived usability.

https://doc.rero.ch/record/28587/files/SauerSonderegger2009.pdf

The influence of prototype fidelity and aesthetics of design in usability tests: effects on user behaviour, subjective evaluation and emotion.

Microsaccades are involuntary, small-magnitude saccadic eye movements that occur during attempted visual fixation. Recent research has found that attention can modulate microsaccade dynamics, but few studies have addressed the effects of task difficulty on microsaccade parameters, and those have obtained contradictory results. Further, no study to date has investigated the influence of task difficulty on microsaccade production during the performance of non-visual tasks. Thus, the effects of task difficulty on microsaccades, isolated from sensory modality, remain unclear. Here we investigated the effects of task difficulty on microsaccades during the performance of a non-visual, mental arithmetic task with two levels of complexity. We found that microsaccade rates decreased and microsaccade magnitudes increased with increased task difficulty. We propose that changes in microsaccade rates and magnitudes with task difficulty are mediated by the effects of varying attentional inputs on the rostral superior colliculus activity map.

http://smc.neuralcorrelate.com/files/inpressfiles/ejn12395.pdf

Andreas Sonderegger

Papers

Introducing the RECOLA multimodal corpus of remote collaborative and affective interactions

The influence of design aesthetics in usability testing: effects on user performance and perceived usability.

The influence of prototype fidelity and aesthetics of design in usability tests: effects on user behaviour, subjective evaluation and emotion.

Task difficulty in mental arithmetic affects microsaccadic rates and magnitudes

The influence of age in usability testing.