Point cloud classification plays an important role in a wide range of airborne light detection and ranging (LiDAR) applications, such as topographic mapping, forest monitoring, power line detection, and road detection. However, due to the sensor noise, high redundancy, incompleteness, and complexity of airborne LiDAR systems, point cloud classification is challenging. Traditional point cloud classification methods mostly focus on the development of handcrafted point geometry features and employ machine learning-based classification models to conduct point classification. In recent years, the advances of deep learning models have caused researchers to shift their focus towards machine learning-based models, specifically deep neural networks, to classify airborne LiDAR point clouds. These learning-based methods start by transforming the unstructured 3D point sets to regular 2D representations, such as collections of feature images, and then employ a 2D CNN for point classification. Moreover, these methods usually need to calculate additional local geometry features, such as planarity, sphericity and roughness, to make use of the local structural information in the original 3D space. Nonetheless, the 3D to 2D conversion results in information loss. In this paper, we propose a directionally constrained fully convolutional neural network (D-FCN) that can take the original 3D coordinates and LiDAR intensity as input; thus, it can directly apply to unstructured 3D point clouds for semantic labeling. Specifically, we first introduce a novel directionally constrained point convolution (D-Conv) module to extract locally representative features of 3D point sets from the projected 2D receptive fields. To make full use of the orientation information of neighborhood points, the proposed D-Conv module performs convolution in an orientation-aware manner by using a directionally constrained nearest neighborhood search. Then, we design a multiscale fully convolutional neural network with downsampling and upsampling blocks to enable multiscale point feature learning. The proposed D-FCN model can therefore process input point cloud with arbitrary sizes and directly predict the semantic labels for all the input points in an end-to-end manner. Without involving additional geometry features as input, the proposed method demonstrates superior performance on the International Society for Photogrammetry and Remote Sensing (ISPRS) 3D labeling benchmark dataset. The results show that our model achieves a new state-of-the-art performance on powerline, car, and facade categories. Moreover, to demonstrate the generalization abilities of the proposed method, we conduct further experiments on the 2019 Data Fusion Contest Dataset. Our proposed method achieves superior performance than the comparing methods and accomplishes an overall accuracy of 95.6% and an average F1 score of 0.810.

Directionally constrained fully convolutional neural network for airborne LiDAR point cloud classification

Airborne Laser Scanning (ALS) point cloud classification is a critical task in remote sensing and photogrammetry communities, which can be widely utilized in urban management, powerline surveying and forest monitoring, etc. In particular, the characteristics of ALS point clouds are distinctive in three aspects, (1) numerous geometric instances (e.g. tracts of roofs); (2) extreme scale variations between different categories (e.g. car v.s. roof); (3) discrepancy distribution along the elevation, which should be specifically focused on for ALS point cloud classification. In this paper, we propose a geometry-attentional network consisting of geometry-aware convolution, dense hierarchical architecture and elevation-attention module to embed the three characteristics effectively, which can be trained in an end-to-end manner. Evaluated on the ISPRS Vaihingen 3D Semantic Labeling benchmark, our method achieves the state-of-the-art performance in terms of average F1 score and overall accuracy (OA). Additionally, without retraining, our model trained on the above Vaihingen 3D dataset can also achieve a better result on the dataset of 2019 IEEE GRSS Data Fusion Contest 3D point cloud classification challenge (DFC 3D) than the baseline (i.e. PointSIFT), which verifies the stronger generalization ability of our model.

A geometry-attentional network for ALS point cloud classification

Airborne light detection and ranging (LiDAR) plays an increasingly significant role in urban planning, topographic mapping, environmental monitoring, power line detection and other fields thanks to its capability to quickly acquire large-scale and high-precision ground information. To achieve point cloud classification, previous studies proposed point cloud deep learning models that can directly process raw point clouds based on PointNet-like architectures. And some recent works proposed graph convolution neural network based on the inherent topology of point clouds. However, the above point cloud deep learning models only pay attention to exploring local geometric structures, yet ignore global contextual relationships among all points. In this paper, we present a global-local graph attention convolution neural network (GACNN) that can be directly applied to the classification of unstructured 3D point clouds obtained by airborne LiDAR. Specifically, we first introduce a graph attention convolution module that incorporates global contextual information and local structural features. The global attention module examines spatial relationships among all points, while the local attention module can dynamically learn convolution weights with regard to the spatial position of the local neighboring points and reweight the convolution weights by inspecting the density of each local region. Based on the proposed graph attention convolution module, we further design an end-to-end encoder-decoder network, named GACNN, to capture multiscale features of the point clouds and therefore enable more accurate airborne point cloud classification. Experiments on the ISPRS 3D labeling dataset show that the proposed model achieves a new state-of-the-art performance in terms of average F1 score (71.5%) and a satisfying overall accuracy (83.2%). Additionally, experiments further conducted on the 2019 Data Fusion Contest Dataset by comparing with other prevalent point cloud deep learning models demonstrate the favorable generalization capability of the proposed model.

Airborne LiDAR point cloud classification with global-local graph attention convolution neural network

Airborne LiDAR point cloud classification has been a long-standing problem in photogrammetry and remote sensing. Early efforts either combine hand-crafted feature engineering with machine learning-based classification models or leverage the power of conventional convolutional neural networks (CNNs) on projected feature images. Recent proposed deep learning-based methods tend to develop new convolution operators which can be directly applied on raw point clouds for representative point feature learning. Although these methods have achieved satisfying performance for the classification of airborne LiDAR point clouds, they cannot adequately recognize fine-grained local structures due to the uneven density distribution of 3D point clouds. In this paper, to address this challenging issue, we introduce a density-aware convolution module which uses the point-wise density to reweight the learnable weights of convolution kernels. The proposed convolution module can approximate continuous convolution on unevenly distributed 3D point sets. Based on this convolution module, we further develop a multi-scale CNN model with downsampling and upsampling blocks to perform per-point semantic labeling. In addition, to regularize the global semantic context, we implement a context encoding module to predict a global context encoding and formulated a context encoding regularizer to enforce the predicted context encoding to be aligned with the ground truth one. The overall network can be trained in an end-to-end fashion and directly produces the desired classification results in one network forward pass. Experiments on the ISPRS 3D Labeling Dataset and 2019 Data Fusion Contest Dataset demonstrate the effectiveness and superiority of the proposed method for airborne LiDAR point cloud classification.

DANCE-NET: Density-aware convolution networks with context encoding for airborne LiDAR point cloud classification

Advances in techniques for automated classification of point cloud data introduce great opportunities for many new and existing applications. However, with a limited number of labelled points, automated classification by a machine learning model is prone to overfitting and poor generalization. The present paper addresses this problem by inducing controlled noise (on a trained model) generated by invoking conditional random field similarity penalties using nearby features. The method is called Atrous XCRF and works by forcing a trained model to respect the similarity penalties provided by unlabeled data. In a benchmark study carried out using the ISPRS 3D labeling dataset, our technique achieves 85.0% in term of overall accuracy, and 71.1% in term of F1 score. The result is on par with the current best model for the benchmark dataset and has the highest value in term of F1 score. Additionally, transfer learning using the Bergen 2018 dataset, without model retraining, was also performed. Even though our proposal provides a consistent 3% improvement in term of accuracy, more work still needs to be done to alleviate the generalization problem on the domain adaptation and the transfer learning field.

Addressing overfitting on point cloud classification using Atrous XCRF

The classification of point clouds is a basic task in airborne laser scanning (ALS) point cloud processing. It is quite a challenge when facing complex observed scenes and irregular point distributions. In order to reduce the computational burden of the point-based classification method and improve the classification accuracy, we present a segmentation and multi-scale convolutional neural network-based classification method. Firstly, a three-step region-growing segmentation method was proposed to reduce both under-segmentation and over-segmentation. Then, a feature image generation method was used to transform the 3D neighborhood features of a point into a 2D image. Finally, feature images were treated as the input of a multi-scale convolutional neural network for training and testing tasks. In order to obtain performance comparisons with existing approaches, we evaluated our framework using the International Society for Photogrammetry and Remote Sensing Working Groups II/4 (ISPRS WG II/4) 3D labeling benchmark tests. The experiment result, which achieved 84.9% overall accuracy and 69.2% of average F1 scores, has a satisfactory performance over all participating approaches analyzed.

https://www.mdpi.com/1424-8220/18/10/3347/pdf

Segmentation and Multi-Scale Convolutional Neural Network-Based Classification of Airborne Laser Scanner Data.

Remote Human Activity Recognition (HAR) in a private residential area has a beneficial influence on the elderly population's life, since this group of people require regular monitoring of health conditions. This paper addresses the problem of continuous detection of daily human activities using mm-wave Doppler radar. Unlike most previous research, this work records the data in terms of continuous series of activities rather than individual activities. These series of activities are similar to real-life activity patterns. The Dynamic Time Warping (DTW) algorithm is used for the detection of human activities in the recorded time series of data and compared to other time-series classification methods. DTW requires less amount of labelled data. The input for DTW was provided using three strategies, and the obtained results were compared against each other. The first approach uses the pixel-level data of frames (named UnSup-PLevel). In the other two strategies, a Convolutional Variational Autoencoder (CVAE) is used to extract Un-Supervised Encoded features (UnSup-EnLevel) and Supervised Encoded features (Sup-EnLevel) from the series of Doppler frames. Results demonstrates the superiority of the Sup-EnLevel features over UnSup-EnLevel and UnSup-PLevel strategies. However, the performance of the UnSup-PLevel strategy worked surprisingly well without using annotations.

Bo Tan

Papers

Segmentation and Multi-Scale Convolutional Neural Network-Based Classification of Airborne Laser Scanner Data.

Continuous Human Activity Recognition using Radar Imagery and Dynamic Time Warping