Human action recognition has drawn significant attention because of its importance in computer vision-based applications. Action recognition based on skeleton sequences has rapidly advanced in the last decade. Conventional deep learning-based approaches are based on extracting skeleton sequences through convolutional operations. Most of these architectures are implemented by learning spatial and temporal features through multiple streams. These studies have enlightened the action recognition endeavor from various algorithmic angles. However, three common issues are observed: (1) The models are usually complicated; therefore, they have a correspondingly higher computational complexity. (2) For supervised learning models, the reliance on labels during training is always a drawback. (3) Implementing large models is not beneficial to real-time applications. To address the above issues, in this paper, we propose a multi-layer perceptron (MLP)-based self-supervised learning framework with a contrastive learning loss function (ConMLP). ConMLP does not require a massive computational setup; it can effectively reduce the consumption of computational resources. Compared with supervised learning frameworks, ConMLP is friendly to the huge amount of unlabeled training data. In addition, it has low requirements for system configuration and is more conducive to being embedded in real-world applications. Extensive experiments show that ConMLP achieves the top one inference result of 96.9% on the NTU RGB+D dataset. This accuracy is higher than the state-of-the-art self-supervised learning method. Meanwhile, ConMLP is also evaluated in a supervised learning manner, which has achieved comparable performance to the state of the art of recognition accuracy.

/pdf/conmlp-mlp-based-self-supervised-contrastive-learning-for-1ffx7yog.pdf

ConMLP: MLP-Based Self-Supervised Contrastive Learning for Skeleton Data Analysis and Action Recognition

AI technologies for Natural Languages have made tremendous progress recently.
However, commensurate progress has not been made on Sign Languages, in
particular, in recognizing signs as individual words or as complete sentences.
We introduce OpenHands, a library where we take four key ideas from the NLP
community for low-resource languages and apply them to sign languages for
word-level recognition. First, we propose using pose extracted through
pretrained models as the standard modality of data to reduce training time and
enable efficient inference, and we release standardized pose datasets for 6
different sign languages - American, Argentinian, Chinese, Greek, Indian, and
Turkish. Second, we train and release checkpoints of 4 pose-based isolated sign
language recognition models across all 6 languages, providing baselines and
ready checkpoints for deployment. Third, to address the lack of labelled data,
we propose self-supervised pretraining on unlabelled data. We curate and
release the largest pose-based pretraining dataset on Indian Sign Language
(Indian-SL). Fourth, we compare different pretraining strategies and for the
first time establish that pretraining is effective for sign language
recognition by demonstrating (a) improved fine-tuning performance especially in
low-resource settings, and (b) high crosslingual transfer from Indian-SL to few
other sign languages. We open-source all models and datasets in OpenHands with
a hope that it makes research in sign languages more accessible, available here
at https://github.com/AI4Bharat/OpenHands . 

OpenHands: Making Sign Language Recognition Accessible with Pose-based Pretrained Models across Languages

Self-supervised learning methods for 3D skeleton-based action recognition via contrastive learning have obtained competitive achievements compared to classical supervised methods. Current researches show that adding a Multilayer Perceptron (MLP) to the top of the base encoder can extract high-level and global positive representations. Using a negative memory bank to store negative samples dynamically can balance the ample storage and feature consistency. However, these methods need to consider that the MLP lacks accurate encoding of fine-grained local features, and a memory bank needs rich and diverse negative sample pairs to match positive representations from different encoders. This paper proposes a new method called Cross Momentum Contrast (CrossMoCo), composed of three parts: ST-GCN encoder, ST-GCN encoder with MLP encoder (ST-MLP encoder), and two independent negative memory banks. The two encoders encode the input data into two positive feature pairs. Learning the cross representations of the two positive pairs is helpful for the model to extract both the global and the local information. Two independent negative memory banks update the negative samples according to different positive representations from two encoders, diversifying the negative samples' distribution and making negative representations close to the positive features. The increasing classification difficulty will improve the model's ability of contrastive learning. In addition, the spatiotemporal occlusion mask data augmentation method is used to enhance positive samples' information diversity. This method takes the adjacent skeleton joints that can form a skeleton bone as a mask unit, which can reduce the information redundancy after data augmentation since adjacent joints may carry similar spatiotemporal information. Experiments on the PKU-MMD Part II dataset, the NTU RGB+D 60 dataset, and the NW-UCLA dataset show that the CrossMoCo framework with spatiotemporal occlusion mask data augmentation has achieved a comparable performance.

Contrastive 3D Human Skeleton Action Representation Learning via CrossMoCo With Spatiotemporal Occlusion Mask Data Augmentation

AI technologies for Natural Languages have made tremendous progress recently.
However, commensurate progress has not been made on Sign Languages, in
particular, in recognizing signs as individual words or as complete sentences.
We introduce OpenHands, a library where we take four key ideas from the NLP
community for low-resource languages and apply them to sign languages for
word-level recognition. First, we propose using pose extracted through
pretrained models as the standard modality of data to reduce training time and
enable efficient inference, and we release standardized pose datasets for 6
different sign languages - American, Argentinian, Chinese, Greek, Indian, and
Turkish. Second, we train and release checkpoints of 4 pose-based isolated sign
language recognition models across all 6 languages, providing baselines and
ready checkpoints for deployment. Third, to address the lack of labelled data,
we propose self-supervised pretraining on unlabelled data. We curate and
release the largest pose-based pretraining dataset on Indian Sign Language
(Indian-SL). Fourth, we compare different pretraining strategies and for the
first time establish that pretraining is effective for sign language
recognition by demonstrating (a) improved fine-tuning performance especially in
low-resource settings, and (b) high crosslingual transfer from Indian-SL to few
other sign languages. We open-source all models and datasets in OpenHands with
a hope that it makes research in sign languages more accessible, available here
at https://github.com/AI4Bharat/OpenHands .

Cross-Stream Contrastive Learning for Self-Supervised Skeleton-Based Action Recognition

3D human action recognition has received increasing attention due to its potential application in video surveillance equipment. To guarantee satisfactory performance, previous studies are mainly based on supervised methods, which have to add a large amount of manual annotation costs. In addition, general deep networks for video sequences suffer from heavy computational costs, thus cannot satisfy the basic requirement of embedded systems. In this paper, a novel Motion Guided Attention Learning (MG-AL) framework is proposed, which formulates the action representation learning as a self-supervised motion attention prediction problem. Specifically, MG-AL is a lightweight network. A set of simple motion priors (e.g., intra-joint variance, inter-frame deviation, intra-joint variance, and cross-joint covariance), which minimizes additional parameters and computational overhead, is regarded as a supervisory signal to guide the attention generation. The encoder is trained via predicting multiple self-attention tasks to capture action-specific feature representations. Extensive evaluations are performed on three challenging benchmark datasets (NTU-RGB+D 60, NTU-RGB+D 120 and NW-UCLA). The proposed method achieves superior performance compared to state-of-the-art methods, while having a very low computational cost.

Motion Guided Attention Learning for Self-Supervised 3D Human Action Recognition

/pdf/contrastive-self-supervised-learning-for-skeleton-action-22kr5pu5yl.pdf

Contrastive Self-Supervised Learning for Skeleton Action Recognition

Graph convolution networks (GCNs) have been widely used and achieved fruitful progress in the skeleton-based action recognition task. In GCNs, node interaction modeling dominates the context aggregation and, therefore, is crucial for a graph-based convolution kernel to extract representative features. In this article, we introduce a closer look at a powerful graph convolution formulation to capture rich movement patterns from these skeleton-based graphs. Specifically, we propose a novel heterogeneous graph convolution (HetGCN) that can be considered as the middle ground between the extremes of (2 + 1)-D and 3-D graph convolution. The core observation of HetGCN is that multiple information flows are jointly intertwined in a 3-D convolution kernel, including spatial, temporal, and spatial-temporal cues. Since spatial and temporal information flows characterize different cues for action recognition, HetGCN first dynamically analyzes pairwise interactions between each node and its cross-space-time neighbors and then encourages heterogeneous context aggregation among them. Considering the HetGCN as a generic convolution formulation, we further develop it into two specific instantiations (i.e., intra-scale and inter-scale HetGCN) that significantly facilitate cross-space-time and cross-scale learning on skeleton graphs. By integrating these modules, we propose a strong human action recognition system that outperforms state-of-the-art methods with the accuracy of 93.1% on NTU-60 cross-subject (X-Sub) benchmark, 88.9% on NTU-120 X-Sub benchmark, and 38.4% on kinetics skeleton.

Xuehao Gao

Papers

Motion Guided Attention Learning for Self-Supervised 3D Human Action Recognition

Contrastive Self-Supervised Learning for Skeleton Action Recognition

Learning Heterogeneous Spatial-Temporal Context for Skeleton-Based Action Recognition.