Objective Electroencephalography (EEG) analysis has been an important tool in neuroscience with applications in neuroscience, neural engineering (e.g. Brain-computer interfaces, BCI's), and even commercial applications. Many of the analytical tools used in EEG studies have used machine learning to uncover relevant information for neural classification and neuroimaging. Recently, the availability of large EEG data sets and advances in machine learning have both led to the deployment of deep learning architectures, especially in the analysis of EEG signals and in understanding the information it may contain for brain functionality. The robust automatic classification of these signals is an important step towards making the use of EEG more practical in many applications and less reliant on trained professionals. Towards this goal, a systematic review of the literature on deep learning applications to EEG classification was performed to address the following critical questions: (1) Which EEG classification tasks have been explored with deep learning? (2) What input formulations have been used for training the deep networks? (3) Are there specific deep learning network structures suitable for specific types of tasks? Approach A systematic literature review of EEG classification using deep learning was performed on Web of Science and PubMed databases, resulting in 90 identified studies. Those studies were analyzed based on type of task, EEG preprocessing methods, input type, and deep learning architecture. Main results For EEG classification tasks, convolutional neural networks, recurrent neural networks, deep belief networks outperform stacked auto-encoders and multi-layer perceptron neural networks in classification accuracy. The tasks that used deep learning fell into five general groups: emotion recognition, motor imagery, mental workload, seizure detection, event related potential detection, and sleep scoring. For each type of task, we describe the specific input formulation, major characteristics, and end classifier recommendations found through this review. Significance This review summarizes the current practices and performance outcomes in the use of deep learning for EEG classification. Practical suggestions on the selection of many hyperparameters are provided in the hope that they will promote or guide the deployment of deep learning to EEG datasets in future research.

https://iopscience.iop.org/article/10.1088/1741-2552/ab0ab5/pdf

Deep learning for electroencephalogram (EEG) classification tasks: a review.

Automated emotion recognition (AEE) is an important issue in various fields of activities which use human emotional reactions as a signal for marketing, technical equipment, or human–robot interaction. This paper analyzes scientific research and technical papers for sensor use analysis, among various methods implemented or researched. This paper covers a few classes of sensors, using contactless methods as well as contact and skin-penetrating electrodes for human emotion detection and the measurement of their intensity. The results of the analysis performed in this paper present applicable methods for each type of emotion and their intensity and propose their classification. The classification of emotion sensors is presented to reveal area of application and expected outcomes from each method, as well as their limitations. This paper should be relevant for researchers using human emotion evaluation and analysis, when there is a need to choose a proper method for their purposes or to find alternative decisions. Based on the analyzed human emotion recognition sensors and methods, we developed some practical applications for humanizing the Internet of Things (IoT) and affective computing systems.

Human Emotion Recognition: Review of Sensors and Methods.

Emotion recognition based on multi-channel electroencephalograph (EEG) signals is becoming increasingly attractive. However, the conventional methods ignore the spatial characteristics of EEG signals, which also contain salient information related to emotion states. In this paper, a deep learning framework based on a multiband feature matrix (MFM) and a capsule network (CapsNet) is proposed. In the framework, the frequency domain, spatial characteristics, and frequency band characteristics of the multi-channel EEG signals are combined to construct the MFM. Then, the CapsNet model is introduced to recognize emotion states according to the input MFM. Experiments conducted on the dataset for emotion analysis using EEG, physiological, and video signals (DEAP) indicate that the proposed method outperforms most of the common models. The experimental results demonstrate that the three characteristics contained in the MFM were complementary and the capsule network was more suitable for mining and utilizing the three correlation characteristics.

https://www.mdpi.com/1424-8220/19/9/2212/pdf

Emotion Recognition from Multiband EEG Signals Using CapsNet

As a challenging pattern recognition task, automatic real-time emotion recognition based on multi-channel EEG signals is becoming an important computer-aided method for emotion disorder diagnose in neurology and psychiatry. Traditional machine learning approaches require to design and extract various features from single or multiple channels based on comprehensive domain knowledge. Consequently, these approaches may be an obstacle for non-domain experts. On the contrast, deep learning approaches have been used successfully in many recent literatures to learn features and classify different types of data. In this paper, baseline signals are considered and a simple but effective pre-processing method has been proposed to improve the recognition accuracy. Meanwhile, a hybrid neural network which combines `Convolutional Neural Network (CNN)’ and `Recurrent Neural Network (RNN)’ has been applied to classify human emotion states by effectively learning compositional spatial-temporal representation of raw EEG streams. The CNN module is used to mine the inter-channel correlation among physically adjacent EEG signals by converting the chain-like EEG sequence into 2D-like frame sequence. The LSTM module is adopted to mine contextual information. Experiments are carried out in a segment-level emotion identification task, on the DEAP benchmarking dataset. Our experimental results indicate that the proposed pre-processing method can increase emotion recognition accuracy by 32% approximately and the model achieves a high performance with a mean accuracy of 90.80% and 91.03% on valence and arousal classification task respectively.

Emotion Recognition from Multi-Channel EEG through Parallel Convolutional Recurrent Neural Network

Emotion recognition based on electroencephalography (EEG) is a significant task in the brain-computer interface field. Recently, many deep learning-based emotion recognition methods are demonstrated to outperform traditional methods. However, it remains challenging to extract discriminative features for EEG emotion recognition, and most methods ignore useful information in channel and time. This paper proposes an attention-based convolutional recurrent neural network (ACRNN) to extract more discriminative features from EEG signals and improve the accuracy of emotion recognition. First, the proposed ACRNN adopts a channel-wise attention mechanism to adaptively assign the weights of different channels, and a CNN is employed to extract the spatial information of encoded EEG signals. Then, to explore the temporal information of EEG signals, extended self-attention is integrated into an RNN to recode the importance based on intrinsic similarity in EEG signals. We conducted extensive experiments on the DEAP and DREAMER databases. The experimental results demonstrate that the proposed ACRNN outperforms state-of-the-art methods.

EEG-based Emotion Recognition via Channel-wise Attention and Self Attention

The aim of this study is to recognize human emotions by electroencephalographic (EEG) signals. The innovation of our research methods involves two aspects: First, we integrate the spatial characteristics, frequency domain, and temporal characteristics of the EEG signals, and map them to a two-dimensional image. With these images, we build a series of EEG Multidimensional Feature Image (EEG MFI) sequences to represent the emotion variation with EEG signals. Second, we construct a hybrid deep neural network to deal with the EEG MFI sequences to recognize human emotional states where the hybrid deep neural network combined the Convolution Neural Networks (CNN) and Long Short-Term-Memory (LSTM) Recurrent Neural Networks (RNN). Empirical research is carried out with the open-source dataset DEAP (a Dataset for Emotion Analysis using EEG, Physiological, and video signals) using our method, and the results demonstrate the significant improvements over current state-of-the-art approaches in this ﬁeld. The average emotion classification accuracy of each subject with CLRNN (the hybrid neural networks that we proposed in this study) is 75.21%.

Human Emotion Recognition with Electroencephalographic Multidimensional Features by Hybrid Deep Neural Networks

Brain informatics adopts a systematic methodology to study clinical diagnosis and pathology of mental disorders from macro, meso and micro points of view. Systematic studies produce multi-level and multi-aspect health big data of mental disorders. Effectively integrating multi-level mental health big data and providing multi-level and content-oriented data services for different types of users by an open and extendable mode becomes new requirements on the acquisition and computing of mental health big data. The existing Data-Brain-based brain data center cannot meet these new requirements. This paper proposes a brain and health big data center and develops a content-oriented cloud service architecture, namely, wisdom as a service, for realizing the brain and health big data center. The illustrative example demonstrates significance and usefulness of the proposed approach.

Wisdom as a Service for Mental Health Care

This study aims to build a smart hospital information framework to deal with various medical information. First, a ubiquitous smart hospital information system model under WaaS (Wisdom as a Service) architecture is proposed. With this model, medical information can be organized into different levels. In this model, novel methods of organizing medical information (offline computing) and smart services delivering (online computing) are defined. The methods are used to supply medical knowledge recommendation services to patients according to their personalized condition and context. As a use case, a smart medical knowledge recommendation system, namely SKeWa, was built to reveal the usefulness of the model and the methods.

A Smart Hospital Information System for Mental Disorders

Emotion is considered as a critical aspect of human brain behavior. In this paper, we investigate human normal emotion variation for a long period without stimuli. Eight subjects participated in the experiment for seven days. The EEG signal and POMS scale of the subjects were collected in the experiment. After data collection and preprocessing, Pearson correlation analysis and multiple linear regression analysis were carried out between EEG features and POMS emotion components. The results of Pearson correlation analysis show that the correlation coefficient of EEG features and POMS emotion component range from 0.367 to 0.610 at 0.01 significant levels. Based on this, multiple linear regression models are built between POMS emotion components and EEG features. With these models, the POMS scales of the subjects can be predicted such that the R2 between the prediction scale and real scale ranges from 0.329 to 0.772; the emotion of ‘Depression-Dejection’ has the lowest R2 (0.329); and the ‘Negative Emotion’ has the highest R2 (0.772).

Human Emotion Variation Analysis Based on EEG Signal and POMS Scale

The invention discloses a complexity analysis method based on a multi-scale window as a threshold. The research of an electroencephalogram signal mainly focuses on a conventional analysis method for time frequency features in the past, and more and more researches adopt nonlinear methods in recent years. The first steps of most nonlinear methods all involve the coarse graining problem, and excessive coarse graining causes loss of effective information in the electroencephalogram signal. In order to solve the problem, the complexity analysis method based on the multi-scale window is proposed. The method comprises the steps of firstly performing filtration processing on the electroencephalogram signal; extracting an effective frequency band; obtaining a window with different scales; taking a mid-value in the window as the threshold; and obtaining a unique threshold for each signal point to perform binarization processing and coarse graining. According to the method, the effectiveness of performing coarse graining by taking a mean value as the threshold before is improved and the effective information in the signal can be correctly extracted.

Youjun Li

Papers

Human Emotion Recognition with Electroencephalographic Multidimensional Features by Hybrid Deep Neural Networks

Wisdom as a Service for Mental Health Care

A Smart Hospital Information System for Mental Disorders

Human Emotion Variation Analysis Based on EEG Signal and POMS Scale

Complexity analysis method based on multi-scale window as threshold