This article proposes a novel framework for the real-time capture, assessment, and visualization of ballet dance movements as performed by a student in an instructional, virtual reality (VR) setting. The acquisition of human movement data is facilitated by skeletal joint tracking captured using the popular Microsoft (MS) Kinect camera system, while instruction and performance evaluation are provided in the form of 3D visualizations and feedback through a CAVE virtual environment, in which the student is fully immersed. The proposed framework is based on the unsupervised parsing of ballet dance movement into a structured posture space using the spherical self-organizing map (SSOM). A unique feature descriptor is proposed to more appropriately reflect the subtleties of ballet dance movements, which are represented as gesture trajectories through posture space on the SSOM. This recognition subsystem is used to identify the category of movement the student is attempting when prompted (by a virtual instructor) to perform a particular dance sequence. The dance sequence is then segmented and cross-referenced against a library of gestural components performed by the teacher. This facilitates alignment and score-based assessment of individual movements within the context of the dance sequence. An immersive interface enables the student to review his or her performance from a number of vantage points, each providing a unique perspective and spatial context suggestive of how the student might make improvements in training. An evaluation of the recognition and virtual feedback systems is presented.

An Approach to Ballet Dance Training through MS Kinect and Visualization in a CAVE Virtual Reality Environment

A naïve Bayes baseline for early gesture recognition

Human Computer Interaction would be much more smooth with the implementation of rapid recognition, the aim of which is to recognize the hand gesture before it is completed In this paper, a rapid recognition for dynamic hand gestures using leap motion is proposed The database contains the three-dimensional motion trajectory of the numbers and the alphabet (36 gestures in total) which captured by utilizing a leap motion In order to enhance the effectiveness of rapid recognition, the SVM algorithm is utilized in the paper Experimental results show high recognition rate and accuracy of the proposed system

Rapid recognition of dynamic hand gestures using leap motion

In this paper we present a robust motion recognition framework for both motion capture and RGB-D sensor data. We extract four different types of features and apply a temporal difference operation to form the final feature vector for each frame in the motion sequences. The frames are classified with the extreme learning machine, and the final class of an action is obtained by majority voting. We test our framework with both motion capture and Kinect data and compare the results of different features. The experiments show that our approach can accurately classify actions with both sources of data. For 40 actions of motion capture data, we achieve 92.7% classification accuracy with real-time performance.

/pdf/classification-of-rgb-d-and-motion-capture-sequences-using-531mkv9b30.pdf

Classification of RGB-D and Motion Capture Sequences Using Extreme Learning Machine

This article presents a framework for real-time analysis and visualization of ballet dance movements performed within a Cave Virtual Reality Environment (CAVE). A Kinect sensor captures and extracts dance-based movement features, from which a topology preserved "posture space"' is constructed using a spherical self-organizing map (SSOM). Recordings of dance movements are parsed into gestural elements by projection onto the SSOM to form unique trajectories in posture space. Dependencies between postures in a trajectory are modeled using a Markovian empirical transition matrix, which is then used to recognize attempted movements. This allows for quantitative assessment and feedback of a student's performance, delivered using concurrent, localized visualizations together with a performance score based on incremental dynamic time warping (IDTW).

A Machine Intelligence Approach to Virtual Ballet Training

We propose a new approach for early gesture recognition. Early gesture recognition is a method to recognize sequential posture patterns at their beginning parts. Using early gesture recognition, we can reduce the delay and increase the interactivity of the system. The key issue of early recognition problem is how to recognize the beginning part of gesture. Generally, a reference gesture consists of a sequence of postures. Therefore, to realize early recognition we have to select beginning parts of the posture sequences. In this paper, we propose a method to obtain such beginning parts of the posture sequences, and demonstrate its effectiveness through some experiments.

Adaptive template method for early recognition of gestures

We propose a new gesture recognition method which is called "early recognition". Early recognition is a method to recognize sequential patterns at their beginning parts. Therefore, in the case of gesture recognition, we can get a recognition result of human gestures before the gestures have finished. We realize early recognition by using sparse codes of Self-Organizing Map.

Early Recognition of Gesture Patterns Using Sparse Code of Self-Organizing Map

We propose an approach to achieve early recognition of gesture patterns. We assume that there are two people who interact with a machine, a robot or something. In such a situation, a gesture of a person often has a relationship with a gesture of another person. We exploit such a relationship to realize early recognition of gesture patterns. Early recognition is a method to recognize sequential patterns at their beginning parts. Therefore, in the case of gesture recognition, we can get a recognition result of human gestures before the gestures have finished. Recent years, some approaches have been proposed. In this paper, we expand the application range of early recognition to multiple people based on the co-occurrence of gesture patterns. In our approach, we use Self-Organizing Map to represent gesture patterns of each person, and associative memory based approach learns the relationship between cooccurring gestures. In the experiments, we have found that our proposed method achieved the early recognition more accurately and earlier than the traditional approach.

Early recognition based on co-occurrence of gesture patterns

We propose an approach to achieving early recognition of gesture patterns. Early recognition is a method for recognizing sequential patterns at their earliest stage. Therefore, in the case of gesture recognition, we can get a recognition result for human gestures before the gestures are finished. The most difficult problem in early recognition is knowing when the system has determined the result. Most traditional approaches suffer from this problem, since gestures are often ambiguous. At the start of a gesture, in particular, it is very difficult to determinate the recognition result since insufficient input data have been observed. Therefore, we have improved on the traditional approach by using a self-organizing map.

Manabu Kawashima

Papers

Adaptive template method for early recognition of gestures

Early Recognition of Gesture Patterns Using Sparse Code of Self-Organizing Map

Early recognition based on co-occurrence of gesture patterns

Improvement of early recognition of gesture patterns based on a self-organizing map