Data fusion and multiple classifier systems for human activity detection and health monitoring: Review and open research directions

Call for papers for Special Issue of ACM Transactions on Multimedia Computing, Communications and Applications on Interactive Digital Television

ACM Transactions on Multimedia Computing, Communications and Applications (ACM TOMCCAP)

In the last decade, Human Activity Recognition (HAR) has become a vibrant research area, especially due to the spread of electronic devices such as smartphones, smartwatches and video cameras present in our daily lives. In addition, the advance of deep learning and other machine learning algorithms has allowed researchers to use HAR in various domains including sports, health and well-being applications. For example, HAR is considered as one of the most promising assistive technology tools to support elderly’s daily life by monitoring their cognitive and physical function through daily activities. This survey focuses on critical role of machine learning in developing HAR applications based on inertial sensors in conjunction with physiological and environmental sensors.

Human Activity Recognition Using Inertial, Physiological and Environmental Sensors: A Comprehensive Survey

In this paper, we present a method (Action-Fusion) for human action recognition from depth maps and posture data using convolutional neural networks (CNNs). Two input descriptors are used for action representation. The first input is a depth motion image that accumulates consecutive depth maps of a human action, whilst the second input is a proposed moving joints descriptor which represents the motion of body joints over time. In order to maximize feature extraction for accurate action classification, three CNN channels are trained with different inputs. The first channel is trained with depth motion images (DMIs), the second channel is trained with both DMIs and moving joint descriptors together, and the third channel is trained with moving joint descriptors only. The action predictions generated from the three CNN channels are fused together for the final action classification. We propose several fusion score operations to maximize the score of the right action. The experiments show that the results of fusing the output of three channels are better than using one channel or fusing two channels only. Our proposed method was evaluated on three public datasets: 1) Microsoft action 3-D dataset (MSRAction3D); 2) University of Texas at Dallas-multimodal human action dataset; and 3) multimodal action dataset (MAD) dataset. The testing results indicate that the proposed approach outperforms most of existing state-of-the-art methods, such as histogram of oriented 4-D normals and Actionlet on MSRAction3D. Although MAD dataset contains a high number of actions (35 actions) compared to existing action RGB-D datasets, this paper surpasses a state-of-the-art method on the dataset by 6.84%.

/pdf/deep-convolutional-neural-networks-for-human-action-4w1wknymnj.pdf

Deep Convolutional Neural Networks for Human Action Recognition Using Depth Maps and Postures

Multi-sensor information fusion based on machine learning for real applications in human activity recognition: State-of-the-art and research challenges

Automatic activity recognition systems aim to capture the state of the user and its environment by exploiting heterogeneous sensors, and permit continuous monitoring of numerous physiological signals, where these sensors are attached to the subject's body. This can be immensely useful in healthcare applications, for automatic and intelligent daily activity monitoring for elderly people. In this paper, we present novel data analytic scheme for intelligent Human Activity Recognition (AR) using smartphone inertial sensors based on information theory based feature ranking algorithm and classifiers based on random forests, ensemble learning and lazy learning. Extensive experiments with a publicly available database 1 of human activity with smart phone inertial sensors show that the proposed approach can indeed lead to development of intelligent and automatic real time human activity monitoring for eHealth application scenarios for elderly, disabled and people with special needs.

Smart Phone Based Data Mining for Human Activity Recognition

Body Sensor Networks aim to capture the state of the user and its environment by utilizing from information heterogeneous sensors, and allow continuous monitoring of numerous physiological signals, where these sensors are attached to the subject's body. This can be immensely useful in activity recognition for identity verification, health and ageing and sport and exercise monitoring applications. In this paper, the application of body sensor networks for automatic and intelligent daily activity monitoring for elderly people, using wireless body sensors and smartphone inertial sensors has been reported. The scheme uses information theory-based feature ranking algorithms and classifiers based on random forests, ensemble learning and lazy learning. Extensive experiments using different publicly available datasets of human activity show that the proposed approach can assist in the development of intelligent and automatic real time human activity monitoring technology for eHealth application scenarios for elderly, disabled and people with special needs.

Body sensor networks for human activity recognition

In this article, we propose a novel multimedia sensor fusion approach based on heterogeneous sensors for biometric access control applications. The proposed fusion technique uses multiple acoustic and visual sensors for extracting dominant biometric cues, and combines them with nondominant cues. The performance evaluation of the proposed fusion protocol and a novel cascaded authentication approach using a 3D stereovision database shows a significant improvement in performance and robustness, with equal error rates of 42.9p (audio only), 32p (audio + 3D face + 2D lip features), 15p (audio + 3D face + 2D eye features), and 7.3p (audio-3D face + 2D lip + 2D eye-eyebrows) respectively.

Multimedia sensor fusion for retrieving identity in biometric access control systems

In this paper, we propose a deep learning-based spam detection model. This model is a combination of the Word Embedding technique and Neural Network algorithm. Word Embedding allows a distributed representation of words in the feature space where word's meaning and word analogy can be represented. Deep neural network is used to learn features of text documents represented in the embedding space and use these features to classify text documents. This model architecture is expected to be able to effectively detect spams in various types of text documents as well as in large document corpus.

Deep Learning Based Spam Detection System

In this article, we propose a novel multimodal data analytics scheme for human activity recognition. Traditional data analysis schemes for activity recognition using heterogeneous sensor network setups for e-Health application scenarios are usually a heuristic process, involving underlying domain knowledge. Relying on such explicit knowledge is problematic when aiming to created automatic, unsupervised monitoring and tracking of different activities, and detection of abnormal events. Experiments on a publicly available OPPORTUNITY activity recognition database from UCI machine learning repository demonstrates the potential of our approach to address next generation unsupervised automatic classification and detection approaches for remote activity recognition for novel, eHealth application scenarios, such as monitoring and tracking of elderly, disabled and those with special needs.

Matthew White

Papers

Smart Phone Based Data Mining for Human Activity Recognition

Body sensor networks for human activity recognition

Multimedia sensor fusion for retrieving identity in biometric access control systems

Deep Learning Based Spam Detection System

Multimodal activity recognition based on automatic feature discovery