Automatic emotion recognition has become a trending research topic in the past decade. While works based on facial expressions or speech abound, recognizing affect from body gestures remains a less explored topic. We present a new comprehensive survey hoping to boost research in the field. We first introduce emotional body gestures as a component of what is commonly known as ”body language” and comment general aspects as gender differences and culture dependence. We then define a complete framework for automatic emotional body gesture recognition. We introduce person detection and comment static and dynamic body pose estimation methods both in RGB and 3D. We then comment the recent literature related to representation learning and emotion recognition from images of emotionally expressive gestures. We also discuss multi-modal approaches that combine speech or face with body gestures for improved emotion recognition. While pre-processing methodologies (e.g., human detection and pose estimation) are nowadays mature technologies fully developed for robust large scale analysis, we show that for emotion recognition the quantity of labelled data is scarce. There is no agreement on clearly defined output spaces and the representations are shallow and largely based on naive geometrical representations.

/pdf/survey-on-emotional-body-gesture-recognition-3ljhg1x54b.pdf

Survey on Emotional Body Gesture Recognition

Despite the importance of sign language recognition systems, there is a lack of a Systematic Literature Review and a classification scheme for it. This is the first identifiable academic literature review of sign language recognition systems. It provides an academic database of literature between the duration of 2007–2017 and proposes a classification scheme to classify the research articles. Three hundred and ninety six research articles were identified and reviewed for their direct relevance to sign language recognition systems. One hundred and seventeen research articles were subsequently selected, reviewed and classified. Each of 117 selected papers was categorized on the basis of twenty five sign languages and were further compared on the basis of six dimensions (data acquisition techniques, static/dynamic signs, signing mode, single/double handed signs, classification technique and recognition rate). The Systematic Literature Review and classification process was verified independently. Literature findings of this paper indicate that the major research on sign language recognition has been performed on static, isolated and single handed signs using camera. Overall, it will be hoped that the study may provide readers and researchers a roadmap to guide future research and facilitate knowledge accumulation and creation into the field of sign language recognition.

Sign Language Recognition Systems: A Decade Systematic Literature Review

Hand gesture recognition consists of identifying the class and the instant of occurrence of a given movement of the hand. The solutions to this problem have many applications in science and technology. In this paper, we propose a model for hand gesture recognition in real time. This model takes as input the surface electromyography (EMG) measured on the muscles of the forearm by the Myo armband. For any user, the proposed model can learn to recognize any gesture of the hand through a training process. As part of this process a user needs to record 5 times, during 2 s each, the EMG on his forearm, close to the elbow, while performing the gesture to recognize. The ?-nearest neighbor and the dynamic time warping algorithms are used for classifying the EMGs seen through a window. As part of the proposed model, we also include a detector of muscle activity that speeds the time of processing up and improves the accuracy of the recognition. We tested the proposed model at recognizing the 5 gestures defined by the proprietary recognition system of the Myo armband, achieving an accuracy of 89.5%. Finally, we also demonstrated that the model proposed in this work outperforms other systems, including the recognition system of the Myo.

Real-time hand gesture recognition using the Myo armband and muscle activity detection

Automatic emotion recognition has become a trending research topic in the past decade. While works based on facial expressions or speech abound, recognizing affect from body gestures remains a less explored topic. We present a new comprehensive survey hoping to boost research in the field. We first introduce emotional body gestures as a component of what is commonly known as "body language" and comment general aspects as gender differences and culture dependence. We then define a complete framework for automatic emotional body gesture recognition. We introduce person detection and comment static and dynamic body pose estimation methods both in RGB and 3D. We then comment the recent literature related to representation learning and emotion recognition from images of emotionally expressive gestures. We also discuss multi-modal approaches that combine speech or face with body gestures for improved emotion recognition. While pre-processing methodologies (e.g. human detection and pose estimation) are nowadays mature technologies fully developed for robust large scale analysis, we show that for emotion recognition the quantity of labelled data is scarce, there is no agreement on clearly defined output spaces and the representations are shallow and largely based on naive geometrical representations.

The aim of an automatic video-based facial expression recognition system is to detect and classify human facial expressions from image sequence. An integrated automatic system often involves two components: 1) peak expression frame detection and 2) expression feature extraction. In comparison with the image-based expression recognition system, the video-based recognition system often performs online detection, which prefers low-dimensional feature representation for cost-effectiveness. Moreover, effective feature extraction is needed for classification. Many recent recognition systems often incorporate rich additional subjective information and thus become less efficient for real-time application. In our facial expression recognition system, first, we propose the double local binary pattern (DLBP) to detect the peak expression frame from the video. The proposed DLBP method has a much lower-dimensional size and can successfully reduce detection time. Besides, to handle the illumination variations in LBP, logarithm-laplace (LL) domain is further proposed to get a more robust facial feature for detection. Finally, the Taylor expansion theorem is employed in our system for the first time to extract facial expression feature. We propose the Taylor feature pattern (TFP) based on the LBP and Taylor expansion to obtain an effective facial feature from the Taylor feature map. Experimental results on the JAFFE and Cohn-Kanade data sets show that the proposed TFP method outperforms some state-of-the-art LBP-based feature extraction methods for facial expression feature extraction and can be suited for real-time applications.

Facial Expression Recognition From Image Sequence Based on LBP and Taylor Expansion

This novel work is aimed at the study of emotion recognition from gestures using Kinect sensor. The Kinect sensor along with Software Development Kit (SDK) generates the human skeleton represented by 3-dimensional coordinates corresponding to twenty body joints. Using the co-ordinates of eleven such joints from the upper body and the hands, a set of nine features based on the distances, accelerations and angles between the different joints have been extracted. These features are able to uniquely identify gestures corresponding to five basic human emotional states, namely, `Anger', `Fear', `Happiness', `Sadness' and `Relaxation'. The goal of the proposed system is to classify an emotion based on body gesture. A comparison of classification using binary decision tree, ensemble decision tree, k-nearest neighbour, support vector machine with radial basis function kernel and neural network classifier based on back-propagation learning is made, in terms of average classification accuracy and computation time. A high overall recognition rate of 90.83% is obtained from the ensemble decision tree.

A study on emotion recognition from body gestures using Kinect sensor

This work proposes gesture recognition algorithm for Indian Classical Dance Style using Kinect sensor. This device generates the skeleton of human body from which twenty different junction 3-dimensional coordinates are obtained. Here we require only eleven coordinates for the proposed work. Basically six joints coordinates about right and left hands and five upper body joint coordinates are processed. A unique system of feature extraction have been used to distinguish between `Anger', `Fear', `Happiness', `Sadness' and `Relaxation'. This system checks whether the emotion is positive or negative with its intensity information. A total of twenty three features have been extracted based on the distance between different parts of the upper human body, the velocity and acceleration generated along with the angle between different joints. The proposed algorithm gives a high recognition rate of 86.8% using SVM.

Gesture Recognition from Indian Classical Dance Using Kinect Sensor

This paper presents a method for automatic hand gesture recognition of 'Bharatanatyam' dance using Fuzzy L membership function based approach. Here, a 3-stage system has been designed. In the first stage, the hand of the dancer from background is isolated using Texture based segmentation and thus the contour of the hand is extracted by using Sobel edge detection technique. In the next stage, centre point of the boundary is located and based on this eight spatial distances are calculated. These distances are normalized by dividing the maximum distance value. In the final stage, fuzzy L Membership values are calculated for each distance and matching of an unknown hand gesture is done with the known hand gestures from the database based on L fuzzy membership function. The proposed algorithm gives overall an accuracy of 85.1% and timing complexity is 2.563 sec in an Intel Pentium Dual Core processor running Mat lab R011b for each hand gesture. This simple yet effective code is very useful for e-learning of 'Bharatanatyam' dance.

Fuzzy L Membership Function Based Hand Gesture Recognition for Bharatanatyam Dance

A simple method to detect gestures revealing muscle and joint pain is described in this paper. Kinect Sensor is used for data acquisition. This sensor only processes twenty joint coordinates in three dimensional space for feature extraction. The recognition part is achieved using a neural network optimized by Levenberg-Marquardt learning rule. A high recognition rate of 91.9% is achieved using the proposed method. This is also better than several algorithms previously used for elder person gesture recognition works.

Neural Network Based Gesture Recognition for Elderly Health Care Using Kinect Sensor

This work aims at designing a fuzzy matching algorithm that would automatically recognize an unknown ballet posture from seventeen fundamental ballet dance primitives. A novel and simple 7-stage system is proposed to achieve the desired objective. Minimized skeletons of the dance postures are generated after performing skin color segmentation on them. Straight line approximation on the minimized skeletons with the help of chain code and sampling generate their equivalent stick figure diagrams. Significant straight lines from the stick figure diagrams are considered, their fuzzy membership with respect to the 4 quadrants are evaluated. Finally with the help of the evaluated data, a fuzzy T-norm operator determines the proximity of a generated dance posture with the seventeen fundamental dance primitives.

Amit Konar

Papers

A study on emotion recognition from body gestures using Kinect sensor

Gesture Recognition from Indian Classical Dance Using Kinect Sensor

Fuzzy L Membership Function Based Hand Gesture Recognition for Bharatanatyam Dance

Neural Network Based Gesture Recognition for Elderly Health Care Using Kinect Sensor

Fuzzy image matching for posture recognition in ballet dance