Showing papers on "Activity recognition published in 2023"

Multiscale Deep Feature Learning for Human Activity Recognition Using Wearable Sensors

Abstract: Deep convolutional neural networks (CNNs) achieve state-of-the-art performance in wearable human activity recognition (HAR), which has become a new research trend in ubiquitous computing scenario. Increasing network depth or width can further improve accuracy. However, in order to obtain the optimal HAR performance on mobile platform, it has to consider a reasonable tradeoff between recognition accuracy and resource consumption. Improving the performance of CNNs without increasing memory and computational burden is more beneficial for HAR. In this article, we first propose a new CNN that uses hierarchical-split (HS) idea for a large variety of HAR tasks, which is able to enhance multiscale feature representation ability via capturing a wider range of receptive fields of human activities within one feature layer. Experiments conducted on benchmarks demonstrate that the proposed HS module is an impressive alternative to baseline models with similar model complexity, and can achieve higher recognition performance (e.g., 97.28%, 93.75%, 99.02%, and 79.02% classification accuracies) on UCI-HAR, PAMAP2, WISDM, and UNIMIB-SHAR. Extensive ablation studies are performed to evaluate the effect of the variations of receptive fields on classification performance. Finally, we demonstrate that multiscale receptive fields can help to learn more discriminative features (achieving 94.10% SOTA accuracy) in weakly labeled HAR dataset.

Multi-ResAtt: Multilevel Residual Network With Attention for Human Activity Recognition Using Wearable Sensors

Abstract: Human activity recognition (HAR) applications have received much attention due to their necessary implementations in various domains, including Industry 5.0 applications such as smart homes, e-health, and various Internet of Things applications. Deep learning (DL) techniques have shown impressive performance in different classification tasks, including HAR. Accordingly, in this article, we develop a comprehensive HAR system based on a novel DL architecture called Multi-ResAtt (multilevel residual network with attention). This model incorporates initial blocks and residual modules aligned in parallel. Multi-ResAtt learns data representations on the inertial measurement units level. Multi-ResAtt integrates a recurrent neural network with attention to extract time-series features and perform activity recognition. We consider complex human activities collected from wearable sensors to evaluate the Multi-ResAtt using three public datasets, Opportunity; UniMiB-SHAR; and PAMAP2. Additionally, we compared the proposed Multi-ResAtt to several DL models and existing HAR systems, and it achieved significant performance.

An Efficient ResNetSE Architecture for Smoking Activity Recognition from Smartwatch

Abstract: Smoking is a major cause of cancer, heart disease and other afflictions that lead to early mortality. An effective smoking classification mechanism that provides insights into individual smoking habits would assist in implementing addiction treatment initiatives. Smoking activities often accompany other activities such as drinking or eating. Consequently, smoking activity recognition can be a challenging topic in human activity recognition (HAR). A deep learning framework for smoking activity recognition (SAR) employing smartwatch sensors was proposed together with a deep residual network combined with squeeze-and-excitation modules (ResNetSE) to increase the effectiveness of the SAR framework. The proposed model was tested against basic convolutional neural networks (CNNs) and recurrent neural networks (LSTM, BiLSTM, GRU and BiGRU) to recognize smoking and other similar activities such as drinking, eating and walking using the UT-Smoke dataset. Three different scenarios were investigated for their recognition performances using standard HAR metrics (accuracy, F1-score and the area under the ROC curve). Our proposed ResNetSE outperformed the other basic deep learning networks, with maximum accuracy of 98.63%.

RF-Based Human Activity Recognition Using Signal Adapted Convolutional Neural Network

Abstract: Human Activity Recognition (HAR) plays a critical role in a wide range of real-world applications, and it is traditionally achieved via wearable sensing. Recently, to avoid the burden and discomfort caused by wearable devices, device-free approaches exploiting RF signals arise as a promising alternative for HAR. Most of the latest device-free approaches require training a large deep neural network model in either time or frequency domain, entailing extensive storage to contain the model and intensive computations to infer activities. Consequently, even with some major advances on device-free HAR, current device-free approaches are still far from practical in real-world scenarios where the computation and storage resources possessed by, for example, edge devices, are limited. Therefore, we introduce HAR-SAnet which is a novel RF-based HAR framework. It adopts an original signal adapted convolutional neural network architecture: instead of feeding the handcraft features of RF signals into a classifier, HAR-SAnet fuses them adaptively from both time and frequency domains to design an end-to-end neural network model. We apply point-wise grouped convolution and depth-wise separable convolutions to confine the model scale and to speed up the inference execution time. The experiment results show that the recognition accuracy of HAR-SAnet outperforms state-of-the-art algorithms and systems.

ElderSim: A Synthetic Data Generation Platform for Human Action Recognition in Eldercare Applications

Abstract: To train deep learning models for vision-based action recognition of elders’ daily activities, we need large-scale activity datasets acquired under various daily living environments and conditions. However, most public datasets used in human action recognition either differ from or have limited coverage of elders’ activities in many aspects, making it challenging to recognize elders’ daily activities well by only utilizing existing datasets. Recently, such limitations of available datasets have actively been compensated by generating synthetic data from realistic simulation environments and using those data to train deep learning models. In this paper, based on these ideas we develop ElderSim, an action simulation platform that can generate synthetic data on elders’ daily activities. For 55 kinds of frequent daily activities of the elders, ElderSim generates realistic motions of synthetic characters with various adjustable data-generating options and provides different output modalities including RGB videos, two- and three-dimensional skeleton trajectories. We then generate KIST SynADL, a large-scale synthetic dataset of elders’ activities of daily living, from ElderSim and use the data in addition to real datasets to train three state-of-the-art human action recognition models. From the experiments following several newly proposed scenarios that assume different real and synthetic dataset configurations for training, we observe a noticeable performance improvement by augmenting our synthetic data. We also offer guidance with insights for the effective utilization of synthetic data to help recognize elders’ daily activities.

A Fine-Tuning Based Approach for Daily Activity Recognition between Smart Homes

Abstract: Daily activity recognition between different smart home environments faces some challenges, such as an insufficient amount of data and differences in data distribution. However, a deep network requires a large amount of labeled data for training. Additionally, inconsistent data distribution can lead to over-fitting of network learning. Additionally, the time cost of training the network from scratch is too high. In order to solve the above problems, this paper proposes a fine-tuning method suitable for daily activity recognition, which is the first application of this method in our field. Firstly, we unify the sensor space and activity space to reduce the variability in heterogeneous environments. Then, the Word2Vec algorithm is used to transform the activity samples into digital vectors recognizable by the network. Finally, the deep network is fine-tuned to transfer knowledge and complete the recognition task. Additionally, we try to train the network on public datasets. The results show that the network trained on a small dataset also has good transferability. It effectively improves the recognition accuracy and reduces the time cost and heavy data annotation.

Human Activity Recognition Using Smartphones With WiFi Signals

Abstract: In this article, we present a work using a smartphone with an off-the-shelf WiFi router for human activity recognition with various scales. The router serves as a hotspot for transmitting WiFi packets. The smartphone is configured with customized firmware and developed software for capturing WiFi channel state information (CSI) data. We extract the features from the CSI data associated with specific human activities, and utilize the features to classify the activities using machine learning models. To evaluate the system performance, we test 20 types of human activities with different scales including seven small motions, four medium motions, and nine big motions. We recruit 60 participants and spend 140 hours for data collection at various experimental settings, and have 36 000 data points collected in total. Furthermore, for comparison, we adopt three distinct machine learning models, including convolutional neural networks (CNNs), decision tree, and long short-term memory. The results demonstrate that our system can predict these human activities with an overall accuracy of 97.25%. Specifically, our system achieves a mean accuracy of 97.57% for recognizing small-scale motions that are particularly useful for gesture recognition. We then consider the adaptability of the machine learning algorithms in classifying the motions, where CNN achieves the best predicting accuracy. As a result, our system enables human activity recognition in a more ubiquitous and mobile fashion that can potentially enhance a wide range of applications such as gesture control, sign language recognition, etc.

Wrapper-based deep feature optimization for activity recognition in the wearable sensor networks of healthcare systems

Abstract: Abstract The Human Activity Recognition (HAR) problem leverages pattern recognition to classify physical human activities as they are captured by several sensor modalities. Remote monitoring of an individual’s activities has gained importance due to the reduction in travel and physical activities during the pandemic. Research on HAR enables one person to either remotely monitor or recognize another person’s activity via the ubiquitous mobile device or by using sensor-based Internet of Things (IoT). Our proposed work focuses on the accurate classification of daily human activities from both accelerometer and gyroscope sensor data after converting into spectrogram images. The feature extraction process follows by leveraging the pre-trained weights of two popular and efficient transfer learning convolutional neural network models. Finally, a wrapper-based feature selection method has been employed for selecting the optimal feature subset that both reduces the training time and improves the final classification performance. The proposed HAR model has been tested on the three benchmark datasets namely, HARTH, KU-HAR and HuGaDB and has achieved 88.89%, 97.97% and 93.82% respectively on these datasets. It is to be noted that the proposed HAR model achieves an improvement of about 21%, 20% and 6% in the overall classification accuracies while utilizing only 52%, 45% and 60% of the original feature set for HuGaDB, KU-HAR and HARTH datasets respectively. This proves the effectiveness of our proposed wrapper-based feature selection HAR methodology.

Artificial Intelligence Based Approach for Classification of Human Activities Using MEMS Sensors Data

Abstract: The integration of Micro Electronic Mechanical Systems (MEMS) sensor technology in smartphones has greatly improved the capability for Human Activity Recognition (HAR). By utilizing Machine Learning (ML) techniques and data from these sensors, various human motion activities can be classified. This study performed experiments and compiled a large dataset of nine daily activities, including Laying Down, Stationary, Walking, Brisk Walking, Running, Stairs-Up, Stairs-Down, Squatting, and Cycling. Several ML models, such as Decision Tree Classifier, Random Forest Classifier, K Neighbors Classifier, Multinomial Logistic Regression, Gaussian Naive Bayes, and Support Vector Machine, were trained on sensor data collected from accelerometer, gyroscope, and magnetometer embedded in smartphones and wearable devices. The highest test accuracy of 95% was achieved using the random forest algorithm. Additionally, a custom-built Bidirectional Long-Short-Term Memory (Bi-LSTM) model, a type of Recurrent Neural Network (RNN), was proposed and yielded an improved test accuracy of 98.1%. This approach differs from traditional algorithmic-based human activity detection used in current wearable technologies, resulting in improved accuracy.

AMIR: Active Multimodal Interaction Recognition from Video and Network Traffic in Connected Environments

Abstract: Activity recognition using video data is widely adopted for elder care, monitoring for safety and security, and home automation. Unfortunately, using video data as the basis for activity recognition can be brittle, since models trained on video are often not robust to certain environmental changes, such as camera angle and lighting changes. There has been a proliferation of network-connected devices in home environments. Interactions with these smart devices are associated with network activity, making network data a potential source for recognizing these device interactions. This paper advocates for the synthesis of video and network data for robust interaction recognition in connected environments. We consider machine learning-based approaches for activity recognition, where each labeled activity is associated with both a video capture and an accompanying network traffic trace. We develop a simple but effective framework AMIR (Active Multimodal Interaction Recognition) 1 that trains independent models for video and network activity recognition respectively, and subsequently combines the predictions from these models using a meta-learning framework. Whether in lab or at home, this approach reduces the amount of “paired” demonstrations needed to perform accurate activity recognition, where both network and video data are collected simultaneously. Specifically, the method we have developed requires up to 70.83% fewer samples to achieve 85% F1 score than random data collection, and improves accuracy by 17.76% given the same number of samples. CCS

Internet of Things Centric-Based Multiactivity Recognition in Smart Home Environment

Abstract: In recent times, numerous human activity recognition (HAR) schemes have been proposed with embedding sensors, wearable devices, smart phones, and vision and ambient sensors. Though the systems have shown better performance they are mostly standalone and still lack the ability to share, host, and perform real-time analysis and visualization of activity data. The Internet of Things (IoT) paradigm has a solution to render the limitations and this will pave the way for HAR in the smart home environment. Thus in this article, an IoT-centric multiactivity recognition system is proposed and deployed on the cloud platform for activity data tracking in the smart home environment. The proposed system collects the real-time data collected using IMU sensors and transmitted to the IoT-Edge Server via Wi-Fi where the data has been fused and classified using light-weight deep learning models. This system has a provision of a Web-based dashboard which is helpful for the home dwellers to monitor the activities in the remote. The performance evaluation justified that the developed system can measure IoT-based activity recognition with greater efficiency in terms of accuracy and F1-score in a shorter response time as of deployment in the cloud platform to detect the activity.

DSFNet: A Distributed Sensors Fusion Network for Action Recognition

Abstract: Human action recognition (HAR) has become a hot topic in the field of computer vision and pattern recognition due to its wide range of application prospects. In most deep learning and multisensor data-based action recognition works, sequences from sensors are first stacked into 2-D images and then fed into convolutional neural network (CNN) models to extract features. This approach based on data-level fusion learns different sensor data features but loses the uniqueness of different sensor data. In this article, we propose a new deep learning model called distributed sensors fusion network (DSFNet). For the property that acceleration sequences can reflect motion trends, we first learn the features of different sensors and then use the Transformer encoder module to model the dependencies of multisensor actions and extract features. For the property that angular velocity can reflect the direction and velocity of the local pose, we first learn the features of a single sensor in different motion directions in time sequence and then learn the output feature maps of multisensor features in time sequence and extract the features. Finally, the outputs of different data modalities are used for decision-level fusion, which significantly improves the performance of the model. We evaluate the performance of the proposed model on the self-built dataset Changzhou University: a comprehensive multi-modal human action dataset (CZU-MHAD), and the experimental results show that the DSFNet model outperforms the existing methods.

Activity-Based Person Identification Using Multimodal Wearable Sensor Data

Abstract: Wearable devices equipped with a variety of sensors facilitate the measurement of physiological and behavioral characteristics. Activity-based person identification is considered an emerging and fast-evolving technology in security and access control fields. Wearables, such as smartphones, Apple Watch, and Google glass can continuously sense and collect activity-related information of users, and activity patterns can be extracted for differentiating different people. Although various human activities have been widely studied, few of them (gaits and keystrokes) have been used for person identification. In this article, we performed person identification using two public benchmark data sets (UCI-HAR and WISDM2019), which are collected from several different activities using multimodal sensors (accelerometer and gyroscope) embedded in wearable devices (smartphone and smartwatch). We implemented eight classifiers, including an multivariate squeeze-and-excitation network (MSENet), time-series transformer (TST), temporal convolutional network (TCN), CNN-LSTM, ConvLSTM, XGBoost, decision tree, and $k$ -nearest neighbor. The proposed MSENet can model the relationship between different sensor data. It achieved the best person identification accuracies under different activities of 91.31% and 97.79%, respectively, for the public data sets of UCI-HAR and WISDM2019. We also investigated the effects of sensor modality, human activity, feature fusion, and window size for sensor signal segmentation. Compared to the related work, our approach has achieved the state of the art.

A Multi-Label Based Physical Activity Recognition via Cascade Classifier

Abstract: Physical activity recognition is a field that infers human activities used in machine learning techniques through wearable devices and embedded inertial sensors of smartphones. It has gained much research significance and promising prospects in the fields of medical rehabilitation and fitness management. Generally, datasets with different wearable sensors and activity labels are used to train machine learning models, and most research has achieved satisfactory performance for these datasets. However, most of the methods are incapable of recognizing the complex physical activity of free living. To address the issue, we propose a cascade classifier structure for sensor-based physical activity recognition from a multi-dimensional perspective, with two types of labels that work together to represent an exact type of activity. This approach employed the cascade classifier structure based on a multi-label system (Cascade Classifier on Multi-label, CCM). The labels reflecting the activity intensity would be classified first. Then, the data flow is divided into the corresponding activity type classifier according to the output of the pre-layer prediction. The dataset of 110 participants has been collected for the experiment on PA recognition. Compared with the typical machine learning algorithms of Random Forest (RF), Sequential Minimal Optimization (SMO) and K Nearest Neighbors (KNN), the proposed method greatly improves the overall recognition accuracy of ten physical activities. The results show that the RF-CCM classifier has achieved 93.94% higher accuracy than the 87.93% obtained from the non-CCM system, which could obtain better generalization performance. The comparison results reveal that the novel CCM system proposed is more effective and stable in physical activity recognition than the conventional classification methods.

Federated Meta-Learning with Attention for Diversity-Aware Human Activity Recognition

Abstract: The ubiquity of smartphones equipped with multiple sensors has provided the possibility of automatically recognizing of human activity, which can benefit intelligent applications such as smart homes, health monitoring, and aging care. However, there are two major barriers to deploying an activity recognition model in real-world scenarios. Firstly, deep learning models for activity recognition use a large amount of sensor data, which are privacy-sensitive and hence cannot be shared or uploaded to a centralized server. Secondly, divergence in the distribution of sensory data exists among multiple individuals due to their diverse behavioral patterns and lifestyles, which contributes to difficulty in recognizing activity for large-scale users or ’cold-starts’ for new users. To address these problems, we propose DivAR, a diversity-aware activity recognition framework based on a federated Meta-Learning architecture, which can extract general sensory features shared among individuals by a centralized embedding network and individual-specific features by attention module in each decentralized network. Specifically, we first classify individuals into multiple clusters according to their behavioral patterns and social factors. We then apply meta-learning in the architecture of federated learning, where a centralized meta-model learns common feature representations that can be transferred across all clusters of individuals, and multiple decentralized cluster-specific models are utilized to learn cluster-specific features. For each cluster-specific model, a CNN-based attention module learns cluster-specific features from the global model. In this way, by training with sensory data locally, privacy-sensitive information existing in sensory data can be preserved. To evaluate the model, we conduct two data collection experiments by collecting sensor readings from naturally used smartphones annotated with activity information in the real-life environment and constructing two multi-individual heterogeneous datasets. In addition, social characteristics including personality, mental health state, and behavior patterns are surveyed using questionnaires. Finally, extensive empirical results demonstrate that the proposed diversity-aware activity recognition model has a relatively better generalization ability and achieves competitive performance on multi-individual activity recognition tasks.

Deep Learning for Laying Hen Activity Recognition Using Wearable Sensors

Abstract: Laying hen activities in modern intensive housing systems can dramatically influence the policies needed for the optimal management of such systems. Intermittent monitoring of different behaviors during daytime cannot provide a good overview, since daily behaviors are not equally distributed over the day. This paper investigates the application of deep learning technology in the automatic recognition of laying hen behaviors equipped with body-worn inertial measurement unit (IMU) modules in poultry systems. Motivated by the human activity recognition literature, a sophisticated preprocessing method is tailored on the time-series data of IMU, transforming it into the form of so-called activity images to be recognized by the deep learning models. The diverse range of behaviors a laying hen can exhibit are categorized into three classes: low-, medium-, and high-intensity activities, and various recognition models are trained to recognize these behaviors in real-time. Several ablation studies are conducted to assess the efficacy and robustness of the developed models against variations and limitations common for an in situ practical implementation. Overall, the best trained model on the full-feature acquired data achieves a mean accuracy of almost 100%, where the whole process of inference by the model takes less than 30 milliseconds. The results suggest that the application of deep learning technology for activity recognition of individual hens has the potential to accurately aid successful management of modern poultry systems.

WMNN: Wearables-Based Multi-Column Neural Network for Human Activity Recognition

Abstract: In recent years, human activity recognition (HAR) technologies in e-health have triggered broad interest. In literature, mainstream works focus on the body's spatial information (i.e. postures) which lacks the interpretation of key bioinformatics associated with movements, limiting the use in applications requiring comprehensively evaluating motion tasks' correctness. To address the issue, in this article, a Wearables-based Multi-column Neural Network (WMNN) for HAR based on multi-sensor fusion and deep learning is presented. Here, the Tai Chi Eight Methods were utilized as an example as in which both postures and muscle activity strengths are significant. The research work was validated by recruiting 14 subjects in total, and we experimentally show 96.9% and 92.5% accuracy for training and testing, for a total of 144 postures and corresponding muscle activities. The method is then provided with a human-machine interface (HMI), which returns users with motion suggestions (i.e. postures and muscle strength). The report demonstrates that the proposed HAR technique can enhance users' self-training efficiency, potentially promoting the development of the HAR area.

CSI-Based Joint Location and Activity Monitoring for COVID-19 Quarantine Environments

Abstract: The recent SARS-COV-2 virus, also known as COVID-19, badly affected the world’s healthcare system due to limited medical resources for a large number of infected human beings. Quarantine helps in breaking the spread of the virus for such communicable diseases. This work proposes a nonwearable/contactless system for human location and activity recognition using ubiquitous wireless signals. The proposed method utilizes the channel state information (CSI) of the wireless signals recorded through a low-cost device for estimating the location and activity of the person under quarantine. We propose to utilize a Siamese architecture with combined one-dimensional convolutional neural networks (1-D-CNNs) and bi-directional long short-term memory (Bi-LSTM) networks. The proposed method provides high accuracy for the joint task and is validated on two real-world testbeds, first, using the designed low-cost CSI recording hardware, and second, on a public dataset for joint activity and location estimation. The human activity recognition (HAR) results outperform state-of-the-art machine and deep learning methods, and localization results are comparable with the existing methods.

Semi-Supervised Adversarial Auto-Encoder to Expedite Human Activity Recognition

Abstract: The study of human activity recognition concentrates on classifying human activities and the inference of human behavior using modern sensing technology. However, the issue of domain adaptation for inertial sensing-based human activity recognition (HAR) is still burdensome. The existing requirement of labeled training data for adapting such classifiers to every new person, device, or on-body location is a significant barrier to the widespread adoption of HAR-based applications, making this a challenge of high practical importance. We propose the semi-supervised HAR method to improve reconstruction and generation. It executes proper adaptation with unlabeled data without changes to a pre-trained HAR classifier. Our approach decouples VAE with adversarial learning to ensure robust classifier operation, without newly labeled training data, under changes to the individual activity and the on-body sensor position. Our proposed framework shows the empirical results using the publicly available benchmark dataset compared to state-of-art baselines, achieving competitive improvement for handling new and unlabeled activity. The result demonstrates SAA has achieved a 5% improvement in classification score compared to the existing HAR platform.

Transfer Learning Approach for Human Activity Recognition Based on Continuous Wavelet Transform

Abstract: Over the last few years, human activity recognition (HAR) has drawn increasing interest from the scientific community. This attention is mainly attributable to the proliferation of wearable sensors and the expanding role of HAR in such fields as healthcare, sports, and human activity monitoring. Convolutional neural networks (CNN) are becoming a popular approach for addressing HAR problems. However, this method requires extensive training datasets to perform adequately on new data. This paper proposes a novel deep learning model pre-trained on scalograms generated using the continuous wavelet transform (CWT). Nine popular CNN architectures and different CWT configurations were considered to select the best performing combination, resulting in the training and evaluation of more than 300 deep learning models. On the source KU-HAR dataset, the selected model achieved classification accuracy and an F1 score of 97.48% and 97.52%, respectively, which outperformed contemporary state-of-the-art works where this dataset was employed. On the target UCI-HAPT dataset, the proposed model resulted in a maximum accuracy and F1-score increase of 0.21% and 0.33%, respectively, on the whole UCI-HAPT dataset and of 2.82% and 2.89%, respectively, on the UCI-HAPT subset. It was concluded that the usage of the proposed model, particularly with frozen layers, results in improved performance, faster training, and smoother gradient descent on small HAR datasets. However, the use of the pre-trained model on sufficiently large datasets may lead to negative transfer and accuracy degradation.

Deep-HAR: an ensemble deep learning model for recognizing the simple, complex, and heterogeneous human activities

Abstract: The recognition of human activities has become a dominant emerging research problem and widely covered application areas in surveillance, wellness management, healthcare, and many more. In real life, the activity recognition is a challenging issue because human beings are often performing the activities not only simple but also complex and heterogeneous in nature. Most of the existing approaches are addressing the problem of recognizing only simple straightforward activities (e.g. walking, running, standing, sitting, etc.). Recognizing the complex and heterogeneous human activities are a challenging research problem whereas only a limited number of existing works are addressing this issue. In this paper, we proposed a novel Deep-HAR model by ensembling the Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) for recognizing the simple, complex, and heterogeneous type activities. Here, the CNNs are used for extracting the features whereas RNNs are used for finding the useful patterns in time-series sequential data. The activities recognition performance of the proposed model was evaluated using three different publicly available datasets, namely WISDM, PAMAP2, and KU-HAR. Through extensive experiments, we have demonstrated that the proposed model performs well in recognizing all types of activities and has achieved an accuracy of 99.98%, 99.64%, and 99.98% for simple, complex, and heterogeneous activities respectively.