With the invention of the low-cost Microsoft Kinect sensor, high-resolution depth and visual (RGB) sensing has become available for widespread use. The complementary nature of the depth and visual information provided by the Kinect sensor opens up new opportunities to solve fundamental problems in computer vision. This paper presents a comprehensive review of recent Kinect-based computer vision algorithms and applications. The reviewed approaches are classified according to the type of vision problems that can be addressed or enhanced by means of the Kinect sensor. The covered topics include preprocessing, object tracking and recognition, human activity analysis, hand gesture analysis, and indoor 3-D mapping. For each category of methods, we outline their main algorithmic contributions and summarize their advantages/differences compared to their RGB counterparts. Finally, we give an overview of the challenges in this field and future research trends. This paper is expected to serve as a tutorial and source of references for Kinect-based computer vision researchers.

Enhanced Computer Vision With Microsoft Kinect Sensor: A Review

In this paper, we present a novel approach for human action recognition with histograms of 3D joint locations (HOJ3D) as a compact representation of postures. We extract the 3D skeletal joint locations from Kinect depth maps using Shotton et al.'s method [6]. The HOJ3D computed from the action depth sequences are reprojected using LDA and then clustered into k posture visual words, which represent the prototypical poses of actions. The temporal evolutions of those visual words are modeled by discrete hidden Markov models (HMMs). In addition, due to the design of our spherical coordinate system and the robust 3D skeleton estimation from Kinect, our method demonstrates significant view invariance on our 3D action dataset. Our dataset is composed of 200 3D sequences of 10 indoor activities performed by 10 individuals in varied views. Our method is real-time and achieves superior results on the challenging 3D action dataset. We also tested our algorithm on the MSR Action 3D dataset and our algorithm outperforms Li et al. [25] on most of the cases.

View invariant human action recognition using histograms of 3D joints

Human activity recognition from 3D data: A review

A new statistic has been developed to quantify the amount of regularity in data. This statistic, ApEn (approximate entropy), appears to have potential application throughout medicine, notably in electrocardiogram and related heart rate data analyses and in the analysis of endocrine hormone release pulsatility. The focus of this article is ApEn. We commence with a simple example of what we are trying to discern. We then discuss exact regularity statistics and practical difficulties of using them in data analysis. The mathematic formula development for ApEn concludes the Solution section. We next discuss the two key input requirements, followed by an account of a pilot study successfully applying ApEn to neonatal heart rate analysis. We conclude with the important topic of ApEn as a relative (not absolute) measure, potential applications, and some caveats about appropriate usage of ApEn. Appendix A provides example ApEn and entropy computations to develop intuition about these measures. Appendix B contains a Fortran program for computing ApEn. This article can be read from at least three viewpoints. The practitioner who wishes to use a "black box" to measure regularity should concentrate on the exact formula, choices for the two input variables, potential applications, and caveats about appropriate usage. The physician who wishes to apply ApEn to heart rate analysis should particularly note the pilot study discussion. The more mathematically inclined reader will benefit from discussions of the relative (comparative) property of ApEn and from Appendix A.

A regularity statistic for medical data analysis

Although widely used in many applications, accurate and efficient human action recognition remains a challenging area of research in the field of computer vision. Most recent surveys have focused on narrow problems such as human action recognition methods using depth data, 3D-skeleton data, still image data, spatiotemporal interest point-based methods, and human walking motion recognition. However, there has been no systematic survey of human action recognition. To this end, we present a thorough review of human action recognition methods and provide a comprehensive overview of recent approaches in human action recognition research, including progress in hand-designed action features in RGB and depth data, current deep learning-based action feature representation methods, advances in human⁻object interaction recognition methods, and the current prominent research topic of action detection methods. Finally, we present several analysis recommendations for researchers. This survey paper provides an essential reference for those interested in further research on human action recognition.

A Comprehensive Survey of Vision-Based Human Action Recognition Methods.

Robust human activity recognition from depth video using spatiotemporal multi-fused features

Recent advancements in depth video sensors technologies have made human activity recognition (HAR) realizable for elderly monitoring applications. Although conventional HAR utilizes RGB video sensors, HAR could be greatly improved with depth video sensors which produce depth or distance information. In this paper, a depth-based life logging HAR system is designed to recognize the daily activities of elderly people and turn these environments into an intelligent living space. Initially, a depth imaging sensor is used to capture depth silhouettes. Based on these silhouettes, human skeletons with joint information are produced which are further used for activity recognition and generating their life logs. The life-logging system is divided into two processes. Firstly, the training system includes data collection using a depth camera, feature extraction and training for each activity via Hidden Markov Models. Secondly, after training, the recognition engine starts to recognize the learned activities and produces life logs. The system was evaluated using life logging features against principal component and independent component features and achieved satisfactory recognition rates against the conventional approaches. Experiments conducted on the smart indoor activity datasets and the MSRDailyActivity3D dataset show promising results. The proposed system is directly applicable to any elderly monitoring system, such as monitoring healthcare problems for elderly people, or examining the indoor activities of people at home, office or hospital.

https://www.mdpi.com/1424-8220/14/7/11735/pdf

A depth video sensor-based life-logging human activity recognition system for elderly care in smart indoor environments.

Video-based human activity recognition systems have potential contributions to various applications such as smart homes and healthcare services. In this work, we present a novel depth video-based translation and scaling invariant human activity recognition (HAR) system utilizing R transformation of depth silhouettes. To perform HAR in indoor settings, an invariant HAR method is critical to freely perform activities anywhere in a camera view without translation and scaling problems of human body silhouettes. We obtain such invariant features via R transformation on depth silhouettes. Furthermore, in R transforming depth silhouettes, shape information of human body reflected in depth values is encoded into the features. In R transformation, 2D feature maps are computed first through Radon transform of each depth silhouette followed by computing 1D feature profile through R transform to get the translation and scaling invariant features. Then, we apply Principle Component Analysis (PCA) for dimension reduction and Linear Discriminant Analysis (LDA) to make the features more prominent, compact and robust. Finally, Hidden Markov Models (HMMs) are used to train and recognize different human activities. Our proposed system shows superior recognition rate over the conventional approaches, reaching up to the mean recognition rate of 93.16% for six typical human activities whereas the conventional PC and IC-based depth silhouettes achieved only 74.83% and 86.33% ,while binary silhouettes-based R transformation approach achieved 67.08% respectively. Our experimental results show that the proposed method is robust, reliable, and efficient in recognizing the daily human activities.

Depth video-based human activity recognition system using translation and scaling invariant features for life logging at smart home

Human activity recognition (HAR) is an emerging methodology essential for smart homes with practical applications such as personal lifecare and healthcare services for the elderly and disabled people. In this work, we present a novel HAR methodology utilizing the recognized body parts of human depth silhouettes and Hidden Markov Models (HMMs). We first create a database of synthetic depth silhouettes and their corresponding body parts labelled silhouettes of various human activities to train random forests (RFs). With the trained RFs, a set of 23 body parts are recognized from incoming depth silhouettes, yielding a set of centroids from the identified body parts. From the dynamics of these centroids, motion parameters are computed: a set of magnitude and directional angle features. Finally, the spatio-temporal dynamics of these motion features of various activities are used to train HMMs. We have performed HAR with the trained HMMs for six typical home activities and obtained the mean recognition rate of ...

Human Activity Recognition via Recognized Body Parts of Human Depth Silhouettes for Residents Monitoring Services at Smart Home

Recent development in depth sensors opens up new challenging task in the field of computer vision research areas, including human-computer interaction, computer games and surveillance systems. This paper addresses shape and motion features approach to observe, track and recognize human silhouettes using a sequence of RGB-D images. Under our proposed activity recognition framework, the required procedure includes: detecting human silhouettes from the image sequence, we remove noisy effects from background and track human silhouettes using temporal continuity constraints of human motion information for each activity, extracting the shape and motion features to identify richer motion information and then these features are clustered and fed into Hidden Markov Model (HMM) to train, model and recognize human activities based on transition and emission probabilities values. During experimental results, we demonstrate this approach on two challenging depth video datasets: one based on our own annotated database and other based on public database (i.e., MSRAction3D). Our approach shows significant recognition results over the state of the art algorithms.

Ahmad Jalal

Papers

Robust human activity recognition from depth video using spatiotemporal multi-fused features

A depth video sensor-based life-logging human activity recognition system for elderly care in smart indoor environments.

Depth video-based human activity recognition system using translation and scaling invariant features for life logging at smart home

Human Activity Recognition via Recognized Body Parts of Human Depth Silhouettes for Residents Monitoring Services at Smart Home

Shape and Motion Features Approach for Activity Tracking and Recognition from Kinect Video Camera