Point cloud learning has lately attracted increasing attention due to its wide applications in many areas, such as computer vision, autonomous driving, and robotics As a dominating technique in AI, deep learning has been successfully used to solve various 2D vision problems However, deep learning on point clouds is still in its infancy due to the unique challenges faced by the processing of point clouds with deep neural networks Recently, deep learning on point clouds has become even thriving, with numerous methods being proposed to address different problems in this area To stimulate future research, this paper presents a comprehensive review of recent progress in deep learning methods for point clouds It covers three major tasks, including 3D shape classification, 3D object detection and tracking, and 3D point cloud segmentation It also presents comparative results on several publicly available datasets, together with insightful observations and inspiring future research directions

Deep Learning for 3D Point Clouds: A Survey

Deep learning with 3D data such as reconstructed point clouds and CAD models has received great research interests recently. However, the capability of using point clouds with convolutional neural network has been so far not fully explored. In this paper, we present a convolutional neural network for semantic segmentation and object recognition with 3D point clouds. At the core of our network is point-wise convolution, a new convolution operator that can be applied at each point of a point cloud. Our fully convolutional network design, while being surprisingly simple to implement, can yield competitive accuracy in both semantic segmentation and object recognition task.

/pdf/pointwise-convolutional-neural-networks-2d9bwspkar.pdf

Pointwise Convolutional Neural Networks

Remote sensing image scene classification, which aims at labeling remote sensing images with a set of semantic categories based on their contents, has broad applications in a range of fields. Propelled by the powerful feature learning capabilities of deep neural networks, remote sensing image scene classification driven by deep learning has drawn remarkable attention and achieved significant breakthroughs. However, to the best of our knowledge, a comprehensive review of recent achievements regarding deep learning for scene classification of remote sensing images is still lacking. Considering the rapid evolution of this field, this article provides a systematic survey of deep learning methods for remote sensing image scene classification by covering more than 160 papers. To be specific, we discuss the main challenges of remote sensing image scene classification and survey: first, autoencoder-based remote sensing image scene classification methods; second, convolutional neural network-based remote sensing image scene classification methods; and third, generative adversarial network-based remote sensing image scene classification methods. In addition, we introduce the benchmarks used for remote sensing image scene classification and summarize the performance of more than two dozen of representative algorithms on three commonly used benchmark datasets. Finally, we discuss the promising opportunities for further research.

Remote Sensing Image Scene Classification Meets Deep Learning: Challenges, Methods, Benchmarks, and Opportunities

We present PPF-FoldNet for unsupervised learning of 3D local descriptors on pure point cloud geometry Based on the folding-based auto-encoding of well known point pair features, PPF-FoldNet offers many desirable properties: it necessitates neither supervision, nor a sensitive local reference frame, benefits from point-set sparsity, is end-to-end, fast, and can extract powerful rotation invariant descriptors Thanks to a novel feature visualization, its evolution can be monitored to provide interpretable insights Our extensive experiments demonstrate that despite having six degree-of-freedom invariance and lack of training labels, our network achieves state of the art results in standard benchmark datasets and outperforms its competitors when rotations and varying point densities are present PPF-FoldNet achieves 9% higher recall on standard benchmarks, 23% higher recall when rotations are introduced into the same datasets and finally, a margin of >35% is attained when point density is significantly decreased

/pdf/ppf-foldnet-unsupervised-learning-of-rotation-invariant-3d-3v2qdjkpmq.pdf

PPF-FoldNet: Unsupervised Learning of Rotation Invariant 3D Local Descriptors

In this paper, we propose 3D point-capsule networks, an auto-encoder designed to process sparse 3D point clouds while preserving spatial arrangements of the input data. 3D capsule networks arise as a direct consequence of our unified formulation of the common 3D auto-encoders. The dynamic routing scheme and the peculiar 2D latent space deployed by our capsule networks bring in improvements for several common point cloud-related tasks, such as object classification, object reconstruction and part segmentation as substantiated by our extensive evaluations. Moreover, it enables new applications such as part interpolation and replacement.

/pdf/3d-point-capsule-networks-12dywy30mi.pdf

3D Point Capsule Networks

Learning and analyzing 3D point clouds with deep networks is challenging due to the sparseness and irregularity of the data. In this paper, we present a data-driven point cloud upsampling technique. The key idea is to learn multi-level features per point and expand the point set via a multi-branch convolution unit implicitly in feature space. The expanded feature is then split to a multitude of features, which are then reconstructed to an upsampled point set. Our network is applied at a patch-level, with a joint loss function that encourages the upsampled points to remain on the underlying surface with a uniform distribution. We conduct various experiments using synthesis and scan data to evaluate our method and demonstrate its superiority over some baseline methods and an optimization-based method. Results show that our upsampled points have better uniformity and are located closer to the underlying surfaces.

/pdf/pu-net-point-cloud-upsampling-network-1pub27qvln.pdf

PU-Net: Point Cloud Upsampling Network

Point clouds obtained from 3D scans are typically sparse, irregular, and noisy, and required to be consolidated. In this paper, we present the first deep learning based edge-aware technique to facilitate the consolidation of point clouds. We design our network to process points grouped in local patches, and train it to learn and help consolidate points, deliberately for edges. To achieve this, we formulate a regression component to simultaneously recover 3D point coordinates and point-to-edge distances from upsampled features, and an edge-aware joint loss function to directly minimize distances from output points to 3D meshes and to edges. Compared with previous neural network based works, our consolidation is edge-aware. During the synthesis, our network can attend to the detected sharp edges and enable more accurate 3D reconstructions. Also, we trained our network on virtual scanned point clouds, demonstrated the performance of our method on both synthetic and real point clouds, presented various surface reconstruction results, and showed how our method outperforms the state-of-the-arts.

/pdf/ec-net-an-edge-aware-point-set-consolidation-network-33vs8jhp8y.pdf

EC-Net: An Edge-Aware Point Set Consolidation Network

Point clouds acquired from range scans are often sparse, noisy, and non-uniform. This paper presents a new point cloud upsampling network called PU-GAN, which is formulated based on a generative adversarial network (GAN), to learn a rich variety of point distributions from the latent space and upsample points over patches on object surfaces. To realize a working GAN network, we construct an up-down-up expansion unit in the generator for upsampling point features with error feedback and self-correction, and formulate a self-attention unit to enhance the feature integration. Further, we design a compound loss with adversarial, uniform and reconstruction terms, to encourage the discriminator to learn more latent patterns and enhance the output point distribution uniformity. Qualitative and quantitative evaluations demonstrate the quality of our results over the state-of-the-arts in terms of distribution uniformity, proximity-to-surface, and 3D reconstruction quality.

/pdf/pu-gan-a-point-cloud-upsampling-adversarial-network-2xz32ckxot.pdf

PU-GAN: A Point Cloud Upsampling Adversarial Network

PU-GAN: a Point Cloud Upsampling Adversarial Network.

With the development of deep learning, supervised feature learning methods have achieved prominent performance in the field of aerial scene classification. However, supervised feature learning methods require a large amount of labeled training data. To address this limitation, in this article, a novel unsupervised deep feature learning method, namely, Attention generative adversarial networks (Attention GANs), is proposed for aerial scene classification. First, Attention GANs integrates the attention mechanism into GANs to enhance the representation power of the discriminator. Then, to obtain contextual information, a context-aggregation-based feature fusion architecture is designed in the discriminator. Furthermore, the generator and discriminator losses are improved on basis of the Relativistic GAN. At the same time, a content loss is formed by using the feature representations from the context-aggregation-based feature fusion architecture. In the experiments, our Attention GANs is evaluated via comprehensive experiments with four publicly available remote sensing scene data sets, i.e., the UC-Merced data set with 21 scene classes, the RSSCN7 data set with 7 scene classes, the AID data set with 30 scene classes, and the NWPU-RESISC45 data set with 45 scene classes. Experimental results demonstrate that our Attention GANs can obtain the best performance compared with the state-of-the-art methods.

Xianzhi Li

Papers

PU-Net: Point Cloud Upsampling Network

EC-Net: An Edge-Aware Point Set Consolidation Network

PU-GAN: A Point Cloud Upsampling Adversarial Network

PU-GAN: a Point Cloud Upsampling Adversarial Network.

Attention GANs: Unsupervised Deep Feature Learning for Aerial Scene Classification