Obtaining large, human labelled speech datasets to train models for emotion recognition is a notoriously challenging task, hindered by annotation cost and label ambiguity. In this work, we consider the task of learning embeddings for speech classification without access to any form of labelled audio. We base our approach on a simple hypothesis: that the emotional content of speech correlates with the facial expression of the speaker. By exploiting this relationship, we show that annotations of expression can be transferred from the visual domain (faces) to the speech domain (voices) through cross-modal distillation. We make the following contributions: (i) we develop a strong teacher network for facial emotion recognition that achieves the state of the art on a standard benchmark; (ii) we use the teacher to train a student, tabula rasa, to learn representations (embeddings) for speech emotion recognition without access to labelled audio data; and (iii) we show that the speech emotion embedding can be used for speech emotion recognition on external benchmark datasets. Code, models and data are available.

https://dl.acm.org/doi/pdf/10.1145/3240508.3240578

Emotion Recognition in Speech using Cross-Modal Transfer in the Wild

This paper presents a multimodal emotion recognition system, which is based on the analysis of audio and visual cues. From the audio channel, Mel-Frequency Cepstral Coefficients, Filter Bank Energies and prosodic features are extracted. For the visual part, two strategies are considered. First, facial landmarks’ geometric relations, i.e., distances and angles, are computed. Second, we summarize each emotional video into a reduced set of key-frames, which are taught to visually discriminate between the emotions. In order to do so, a convolutional neural network is applied to key-frames summarizing videos. Finally, confidence outputs of all the classifiers from all the modalities are used to define a new feature space to be learned for final emotion label prediction, in a late fusion/stacking fashion. The experiments conducted on the SAVEE, eNTERFACE’05, and RML databases show significant performance improvements by our proposed system in comparison to current alternatives, defining the current state-of-the-art in all three databases.

Audio-Visual Emotion Recognition in Video Clips

Social-Media-Marketing Social-Media-Marketing

People express emotions through different modalities. Utilization of both verbal and nonverbal communication channels allows to create a system in which the emotional state is expressed more clearly and therefore easier to understand. Expanding the focus to several expression forms can facilitate research on emotion recognition as well as human–machine interaction. This article presents analysis of audiovisual information to recognize human emotions. A cross-corpus evaluation is done using three different databases as the training set (SAVEE, eNTERFACE’05 and RML) and AFEW (database simulating real-world conditions) as a testing set. Emotional speech is represented by commonly known audio and spectral features as well as MFCC coefficients. The SVM algorithm has been used for classification. In case of facial expression, faces in key frames are found using Viola–Jones face recognition algorithm and facial image emotion classification done by CNN (AlexNet). Multimodal emotion recognition is based on decision-level fusion. The performance of emotion recognition algorithm is compared with the validation of human decision makers.

/pdf/audiovisual-emotion-recognition-in-wild-1s7ck5uemr.pdf

Audiovisual emotion recognition in wild

Human facial emotion recognition (FER) has attracted the attention of the research community for its promising applications. Mapping different facial expressions to the respective emotional states are the main task in FER. The classical FER consists of two major steps: feature extraction and emotion recognition. Currently, the Deep Neural Networks, especially the Convolutional Neural Network (CNN), is widely used in FER by virtue of its inherent feature extraction mechanism from images. Several works have been reported on CNN with only a few layers to resolve FER problems. However, standard shallow CNNs with straightforward learning schemes have limited feature extraction capability to capture emotion information from high-resolution images. A notable drawback of the most existing methods is that they consider only the frontal images (i.e., ignore profile views for convenience), although the profile views taken from different angles are important for a practical FER system. For developing a highly accurate FER system, this study proposes a very Deep CNN (DCNN) modeling through Transfer Learning (TL) technique where a pre-trained DCNN model is adopted by replacing its dense upper layer(s) compatible with FER, and the model is fine-tuned with facial emotion data. A novel pipeline strategy is introduced, where the training of the dense layer(s) is followed by tuning each of the pre-trained DCNN blocks successively that has led to gradual improvement of the accuracy of FER to a higher level. The proposed FER system is verified on eight different pre-trained DCNN models (VGG-16, VGG-19, ResNet-18, ResNet-34, ResNet-50, ResNet-152, Inception-v3 and DenseNet-161) and well-known KDEF and JAFFE facial image datasets. FER is very challenging even for frontal views alone. FER on the KDEF dataset poses further challenges due to the diversity of images with different profile views together with frontal views. The proposed method achieved remarkable accuracy on both datasets with pre-trained models. On a 10-fold cross-validation way, the best achieved FER accuracies with DenseNet-161 on test sets of KDEF and JAFFE are 96.51% and 99.52%, respectively. The evaluation results reveal the superiority of the proposed FER system over the existing ones regarding emotion detection accuracy. Moreover, the achieved performance on the KDEF dataset with profile views is promising as it clearly demonstrates the required proficiency for real-life applications.

https://www.mdpi.com/2079-9292/10/9/1036/pdf

Facial Emotion Recognition Using Transfer Learning in the Deep CNN

Since the success of any tourism business is determined by tourism planning, development research and marketing, the first thing we review in this article is GIS application for tourism planning. Both tourism and IT increasingly provide strategic opportunities and powerful tools for economic growth, redistribution of wealth and development of equity around the globe. GIS technology offers great opportunities for the development of modern tourism applications using maps. This technology integrates common database operations such as query with the unique visualization and geographic analysis benefits offered by maps. GIS is used for bringing the georeferenced data (spatial and non spatial) of geographic location Zlatibor and Zlatar into digital maps. Each object is assigned to a thematic layer. Each layer combines related objects like roads, building, protected areas or watercourses. In this research the authors used GIS in three types of applications such as inventory, analysis and evaluation of plan based on tourism development.

/pdf/the-application-of-gis-and-its-components-in-tourism-50o99d3k3q.pdf

The Application of GIS and ITS Components in Tourism

In this paper is presented a novel multimodal emotion recognition system which is based on the analysis of audio and visual cues. MFCC-based features are extracted from the audio channel and facial landmark geometric relations are computed from visual data. Both sets of features are learnt separately using state-of-the-art classifiers. In addition, we summarise each emotion video into a reduced set of key-frames, which are learnt in order to visually discriminate emotions by means of a Convolutional Neural Network. Finally, confidence outputs of all classifiers from all modalities are used to define a new feature space to be learnt for final emotion prediction, in a late fusion/stacking fashion. The conducted experiments on eNTERFACE'05 database show significant performance improvements of our proposed system in comparison to state-of-the-art approaches.

Fusion of classifier predictions for audio-visual emotion recognition

Micro-expressions (MEs) are very short, rapid, difficult to control and subtle which reveal hidden emotions. Spotting and recognition of MEs are very difficult for humans. Lately, researchers have tried to develop automatically MEs detection and recognition algorithms, however the biggest obstacle is the lack of a suitable datasets. Previous studies mainly focus on posed rather than spontaneous videos, and the obtained performances were low. To address these challenges, firstly we made a hidden sadness database, which includes 13 video clips elicited from students, who were watching very sad scenes from the movie in the University environment. Secondly, a new approach for automatic hidden sadness detection algorithm is proposed. Finally, Support Vector Machine and Random Forest classifiers are applied, since it has been shown that they provide state-of-the-art accuracy for the facial expression recognition problem. Two experiments were conducted, one with all extracted features from the face, and the other with only eye region features. The best results are achieved with Random Forest algorithm using all face features, with the recognition rate of 95.72%. For further improvement of the performance, we plan to integrate the deep Convolutional Neural Network algorithm, due to its grow popularity in the visual recognition.

Automatic Hidden Sadness Detection Using Micro-Expressions

Artificial intelligence (AI) impacts numerous aspects of life in the form of smart devices and smart applications, designed to understand consumer behaviour, needs, and preferences in order to deliver customized experiences. AI has been one of the primary drivers of innovation in marketing. Marketers are already leveraging the advantages of AI to gain valuable insights into customers, competitors and markets. Besides, AI automate tasks, reduce costs, and improve workflows. This paper examines the current and potential applications of AI within marketing by providing comprehensive overview of existing academic research.

/pdf/benefits-of-artificial-intelligence-and-machine-learning-in-y0edxfotw9.pdf

Angelina Njeguš

Papers

Audio-Visual Emotion Recognition in Video Clips

The Application of GIS and ITS Components in Tourism

Fusion of classifier predictions for audio-visual emotion recognition

Automatic Hidden Sadness Detection Using Micro-Expressions

Benefits of Artificial Intelligence and Machine Learning in Marketing