A new approach for product evaluation based on integration of EEG and eye-tracking

Sentiment analysis and emotion detection in conversation are becoming hot topics in regard to several applications. With the development of the social robot, social network, and intelligent voice assistant, emotion detection is attracting more attention as a key component in these research fields. Many approaches have been proposed to handle this problem in recent years. However, these previous approaches focus on either the temporal change information of the conversation or the semantic correlation information of the dialogue but ignore the combination of temporal information and semantic correlation information. In this paper, we propose an incremental graph convolution network (I-GCN) to handle emotion detection in conversation. We first utilize the graph structure to represent conversation at different times, which can represent the semantic correlation information of utterances. Then, we apply the incremental graph structure to imitate the process of dynamic conversation, which can preserve the temporal change information of conversation. Especially, for the first step of the process, we creatively propose utterance-level GCN (U-GCN) and speaker-level GCN (S-GCN) to learn the features of utterances for emotion detection. U-GCN focuses on the correlations among utterances and applies the multi-head attention model to find latent correlation information among utterances, which aims to further enhance the guidance of semantic relevance for feature learning. S-GCN focuses on the correlation between speaker and utterances, which can provide a different angle to guide the feature learning of utterances. In the learning of model parameters, we constantly utilize the new utterances to fine-tune the parameters of GNN for enhancement of the contribution of temporal change information. Detailed evaluations of the proposed method on three published conversation corpuses demonstrate the great effectiveness of our approach over several conventional competitive baselines. 

I-GCN: Incremental Graph Convolution Network for Conversation Emotion Detection

The future emotion prediction of users on social media has been attracting increasing attention from academics. Previous studies on predicting future emotion have focused on the characteristics of individuals’ emotion changes; however, the role of the individual’s neighbors has not yet been thoroughly researched. To fill this gap, a surrounding-aware individual emotion prediction model (SAEP) based on a deep encoder–decoder architecture is proposed to predict individuals’ future emotions. In particular, two memory-based attention networks are constructed: The time-evolving attention network and the surrounding attention network to extract the features of the emotional changes of users and neighbors, respectively. Then, these features are incorporated into the emotion prediction task. In addition, a novel variant LSTM is introduced as the encoder of the proposed model, which can effectively extract complex patterns of users’ emotional changes from irregular time series. Extensive experimental results show that the proposed approach outperforms five alternative methods. The SAEP approach has improved by approximately 4.21–14.84% micro F1 on a dataset built from Twitter and 7.30–13.41% on a dataset built from Microblog. Further analyses validate the effectiveness of the proposed time-evolving context and surrounding context, as well as the factors that may affect the prediction results.

SAEP: A Surrounding-Aware Individual Emotion Prediction Model Combined with T-LSTM and Memory Attention Mechanism

Abstract In recent years, reinforcement learning has been successfully applied to dialogue systems. However, in the face of task-oriented dialogue systems where policies are difficult to optimize, states are difficult to track, and tasks are multiple and compound, task-oriented dialogue systems based on reinforcement learning have problems such as poor collaboration, non-unique learning goals, and non-stationarity due to the lack of collaboration among agents. In this paper, we propose a multi-agent cooperative dialogue (MACD) algorithm for task-oriented dialogue systems, in which, for the information interaction between multi-agent in task-oriented dialogue systems, a deep neural network approach is used to integrate the observations of multiple single agents and obtain joint observations to achieve information sharing among single agents, to solve the non-stationarity caused by the lack of joint information of multi-agent. For multi-agent policy learning task-oriented dialogue systems, the multi-agent deep deterministic policy gradient (MADDPG) architecture is applied to the policy selection of task-oriented dialogue systems to solve the problem of lack of joint policy learning of multi-agent in task-oriented dialogue systems; the observation integration of single agents and multi-agent policy learning are effectively combined to solve the problem of poor multi-agent collaboration in task-oriented dialogue systems. By verifying and analyzing reinforcement learning algorithms such as MACD, REINFORCE, DQN, and QMIX in the MultiWOZ corpus, the experimental results show that the algorithms effectively improve the success rate of multi-agent working together to complete dialogue tasks, reduce the number of invalid dialogues in dialogue turns, and outperform common reinforcement learning algorithms in terms of agents' information interaction and joint policy learning in the composite task dialogue scenario. 

/pdf/a-multi-agent-collaborative-algorithm-for-task-oriented-1rc830wc.pdf

A multi-agent collaborative algorithm for task-oriented dialogue systems

Feedback is a crucial process in education because it helps learners identify their weaknesses whilst motivating them to continue to learn. Existing systems often only provide a score or rating with basic explanations. Although some systems provide detailed feedback, they require manual input from teachers. This paper proposes a real-time feedback visualisation system (called BETTER) for supporting emotional speech training which uses a visual dashboard to provide the learner with immediate feedback in the form of written, audio, and visual feedback. The AI-based feedback system utilises pitch tracking, transcriptions, and audio modifications in addition to one-dimensional convolutional neural networks (CNNs) to categorise speech into emotional states. A preliminary experiment was conducted involving a speech expert and 8 non-native speakers to assess their cognitive load, technology acceptance, and satisfaction while using the system. 

BETTER: An Automatic feedBack systEm for supporTing emoTional spEech tRaining

In this article, we propose a novel reinforcement learning network that keeps track of the gradual emotional changes from every utterance throughout the conversation and uses this information for each utterance’s emotion detection. Concretely, we first establish an agent and, then, utilize sliding windows to extract the accumulated emotional information before the current utterance. We define the concatenation of accumulated emotional information and the contextual information as the state of the reinforcement learning framework. The action of the established agent is formulated as the emotional label of the current utterance. On this basis, we formulate the progressive emotional interaction process throughout the conversation as a sequential decision problem and solve it with the reinforcement learning framework. Detailed evaluations on the published multimodal MELD dataset demonstrate the effectiveness of our approach.

Qiankun Han

Papers

Emotion Detection for Conversations Based on Reinforcement Learning Framework