Parkinson’s disease (PD) is a neurological disorder with complicated and disabling motor and non-motor symptoms. The complexity of PD pathology is amplified due to its dependency on patient diaries and the neurologist’s subjective assessment of clinical scales. A significant amount of recent research has explored new cost-effective and subjective assessment methods pertaining to PD symptoms to address this challenge. This article analyzes the application areas and use of mobile and wearable technology in PD research using the PRISMA methodology. Based on the published papers, we identify four significant fields of research: diagnosis, prognosis and monitoring, predicting response to treatment, and rehabilitation. Between January 2008 and December 2021, 31,718 articles were published in four databases: PubMed Central, Science Direct, IEEE Xplore, and MDPI. After removing unrelated articles, duplicate entries, non-English publications, and other articles that did not fulfill the selection criteria, we manually investigated 1559 articles in this review. Most of the articles (45%) were published during a recent four-year stretch (2018–2021), and 19% of the articles were published in 2021 alone. This trend reflects the research community’s growing interest in assessing PD with wearable devices, particularly in the last four years of the period under study. We conclude that there is a substantial and steady growth in the use of mobile technology in the PD contexts. We share our automated script and the detailed results with the public, making the review reproducible for future publications.

/pdf/a-systematic-survey-of-research-trends-in-technology-usage-3vnnx8lh.pdf

A Systematic Survey of Research Trends in Technology Usage for Parkinson’s Disease

The ability to estimate 3D human body pose and movement, also known as human pose estimation (HPE), enables many applications for home-based health monitoring, such as remote rehabilitation training. Several possible solutions have emerged using sensors ranging from RGB cameras, depth sensors, millimeter-Wave (mmWave) radars, and wearable inertial sensors. Despite previous efforts on datasets and benchmarks for HPE, few dataset exploits multiple modalities and focuses on home-based health monitoring. To bridge the gap, we present mRI, a multi-modal 3D human pose estimation dataset with mmWave, RGB-D, and Inertial Sensors. Our dataset consists of over 160k synchronized frames from 20 subjects performing rehabilitation exercises and supports the benchmarks of HPE and action detection. We perform extensive experiments using our dataset and delineate the strength of each modality. We hope that the release of mRI can catalyze the research in pose estimation, multi-modal learning, and action understanding, and more importantly facilitate the applications of home-based health monitoring.

mRI: Multi-modal 3D Human Pose Estimation Dataset using mmWave, RGB-D, and Inertial Sensors

Parkinson’s disease (PD) is a neurodegenerative disorder that produces both motor and non-motor complications, degrading the quality of life of PD patients. Over the past two decades, the use of wearable devices in combination with machine learning algorithms has provided promising methods for more objective and continuous monitoring of PD. Recent advances in artificial intelligence have provided new methods and algorithms for data analysis, such as deep learning (DL). The aim of this article is to provide a comprehensive review of current applications where DL algorithms are employed for the assessment of motor and non-motor manifestations (NMM) using data collected via wearable sensors. This paper provides the reader with a summary of the current applications of DL and wearable devices for the diagnosis, prognosis, and monitoring of PD, in the hope of improving the adoption, applicability, and impact of both technologies as support tools. Following PRISMA (Systematic Reviews and Meta-Analyses) guidelines, sixty-nine studies were selected and analyzed. For each study, information on sample size, sensor configuration, DL approaches, validation methods and results according to the specific symptom under study were extracted and summarized. Furthermore, quality assessment was conducted according to the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) method. The majority of studies (74%) were published within the last three years, demonstrating the increasing focus on wearable technology and DL approaches for PD assessment. However, most papers focused on monitoring (59%) and computer-assisted diagnosis (37%), while few papers attempted to predict treatment response. Motor symptoms (86%) were treated much more frequently than NMM (14%). Inertial sensors were the most commonly used technology, followed by force sensors and microphones. Finally, convolutional neural networks (52%) were preferred to other DL approaches, while extracted features (38%) and raw data (37%) were similarly used as input for DL models. The results of this review highlight several challenges related to the use of wearable technology and DL methods in the assessment of PD, despite the advantages this technology could bring in the development and implementation of automated systems for PD assessment. 

Deep learning and wearable sensors for the diagnosis and monitoring of Parkinson's disease: A systematic review

Combining skeleton and RGB modalities in human action recognition (HAR) has garnered attention due to their ability to complement each other. However, previous studies did not address the challenge of recognizing fine-grained human-object interaction (HOI). To tackle this problem, this study introduces a new transformer-based architecture called Sequential Skeleton RGB Transformer (SSRT), which fuses skeleton and RGB modalities. First, SSRT leverages the strength of Long Short-Term Memory (LSTM) and a multi-head attention mechanism to extract high-level features from both modalities. Subsequently, SSRT employs a two-stage fusion method, including transformer cross-attention fusion and softmax layer late score fusion, to effectively integrate the multimodal features. Aside from evaluating the proposed method on fine-grained HOI, this study also assesses its performance on two other action recognition tasks: general HAR and cross-dataset HAR. Furthermore, this study conducts a performance comparison between a HAR model using single-modality features (RGB and skeleton) alongside multimodality features on all three action recognition tasks. To ensure a fair comparison, comparable state-of-the-art transformer architectures are employed for both the single-modality HAR model and SSRT. In terms of modality, SSRT outperforms the best-performing single-modality HAR model on all three tasks, with accuracy improved by 9.92% on fine-grained HOI recognition, 6.73% on general HAR, and 11.08% on cross-dataset HAR. Additionally, the proposed fusion model surpasses state-of-the-art multimodal fusion techniques like Transformer Early Concatenation, with an accuracy improved by 6.32% on fine-grained HOI recognition, 4.04% on general HAR, and 6.56% on cross-dataset.

SSRT: A Sequential Skeleton RGB Transformer to Recognize Fine-Grained Human-Object Interactions and Action Recognition

Parkinson’s disease (PD) is increasingly being studied using science-intensive methods due to economic, medical, rehabilitation and social reasons. Wearable sensors and Internet of Things-enabled technologies look promising for monitoring motor activity and gait in PD patients. In this study, we sought to evaluate gait characteristics by analyzing the accelerometer signal received from a smartphone attached to the head during an extended TUG test, before and after single and repeated sessions of terrestrial microgravity modeled with the condition of “dry” immersion (DI) in five subjects with PD. The accelerometer signal from IMU during walking phases of the TUG test allowed for the recognition and characterization of up to 35 steps. In some patients with PD, unusually long steps have been identified, which could potentially have diagnostic value. It was found that after one DI session, stepping did not change, though in one subject it significantly improved (cadence, heel strike and step length). After a course of DI sessions, some characteristics of the TUG test improved significantly. In conclusion, the use of accelerometer signals received from a smartphone IMU looks promising for the creation of an IoT-enabled system to monitor gait in subjects with PD.

/pdf/gait-characteristics-analyzed-with-smartphone-imu-sensors-in-2jnvzzsn.pdf

Gait Characteristics Analyzed with Smartphone IMU Sensors in Subjects with Parkinsonism under the Conditions of “Dry” Immersion

Millimeter-Wave (mmWave) radar can enable high-resolution human pose estimation with low cost and computational requirements. However, mmWave data point cloud, the primary input to processing algorithms, is highly sparse and carries significantly less information than other alternatives such as video frames. Furthermore, the scarce labeled mmWave data impedes the development of machine learning (ML) models that can generalize to unseen scenarios. We propose a fast and scalable human pose estimation (FUSE) framework that combines multi-frame representation and meta-learning to address these challenges. Experimental evaluations show that FUSE adapts to the unseen scenarios 4× faster than current supervised learning approaches and estimates human joint coordinates with about 7 cm mean absolute error.

/pdf/fast-and-scalable-human-pose-estimation-using-mmwave-point-1rohzcez.pdf

Fast and scalable human pose estimation using mmWave point cloud

Synthesis and reconstruction of 3D human head has gained increasing interests in computer vision and computer graphics recently. Existing state-of-the-art 3D generative adversarial networks (GANs) for 3D human head synthesis are either limited to near-frontal views or hard to preserve 3D consistency in large view angles. We propose PanoHead, the first 3D-aware generative model that enables high-quality view-consistent image synthesis of full heads in $360^\circ$ with diverse appearance and detailed geometry using only in-the-wild unstructured images for training. At its core, we lift up the representation power of recent 3D GANs and bridge the data alignment gap when training from in-the-wild images with widely distributed views. Specifically, we propose a novel two-stage self-adaptive image alignment for robust 3D GAN training. We further introduce a tri-grid neural volume representation that effectively addresses front-face and back-head feature entanglement rooted in the widely-adopted tri-plane formulation. Our method instills prior knowledge of 2D image segmentation in adversarial learning of 3D neural scene structures, enabling compositable head synthesis in diverse backgrounds. Benefiting from these designs, our method significantly outperforms previous 3D GANs, generating high-quality 3D heads with accurate geometry and diverse appearances, even with long wavy and afro hairstyles, renderable from arbitrary poses. Furthermore, we show that our system can reconstruct full 3D heads from single input images for personalized realistic 3D avatars.

PanoHead: Geometry-Aware 3D Full-Head Synthesis in 360°

The ability to estimate 3D human body pose and movement, also known as human pose estimation (HPE), enables many applications for home-based health monitoring, such as remote rehabilitation training. Several possible solutions have emerged using sensors ranging from RGB cameras, depth sensors, millimeter-Wave (mmWave) radars, and wearable inertial sensors. Despite previous efforts on datasets and benchmarks for HPE, few dataset exploits multiple modalities and focuses on home-based health monitoring. To bridge this gap, we present mRI 1 , a multi-modal 3D human pose estimation dataset with m mWave, R GB-D, and I nertial Sensors. Our dataset consists of over 160k synchronized frames from 20 subjects performing rehabilitation exercises and supports the benchmarks of HPE and action detection. We perform extensive experiments using our dataset and de-lineate the strength of each modality. We hope that the release of mRI can catalyze the research in pose estimation, multi-modal learning, and action understanding, and more importantly facilitate the applications of home-based health monitoring.

Sizhe An

Papers

A Systematic Survey of Research Trends in Technology Usage for Parkinson’s Disease

mRI: Multi-modal 3D Human Pose Estimation Dataset using mmWave, RGB-D, and Inertial Sensors

Fast and scalable human pose estimation using mmWave point cloud

PanoHead: Geometry-Aware 3D Full-Head Synthesis in 360°

Multi-modal 3D Human Pose Estimation using mmWave, RGB-D, and Inertial Sensors