Point cloud learning has lately attracted increasing attention due to its wide applications in many areas, such as computer vision, autonomous driving, and robotics As a dominating technique in AI, deep learning has been successfully used to solve various 2D vision problems However, deep learning on point clouds is still in its infancy due to the unique challenges faced by the processing of point clouds with deep neural networks Recently, deep learning on point clouds has become even thriving, with numerous methods being proposed to address different problems in this area To stimulate future research, this paper presents a comprehensive review of recent progress in deep learning methods for point clouds It covers three major tasks, including 3D shape classification, 3D object detection and tracking, and 3D point cloud segmentation It also presents comparative results on several publicly available datasets, together with insightful observations and inspiring future research directions

Deep Learning for 3D Point Clouds: A Survey

We study the problem of efficient semantic segmentation for large-scale 3D point clouds. By relying on expensive sampling techniques or computationally heavy pre/post-processing steps, most existing approaches are only able to be trained and operate over small-scale point clouds. In this paper, we introduce RandLA-Net, an efficient and lightweight neural architecture to directly infer per-point semantics for large-scale point clouds. The key to our approach is to use random point sampling instead of more complex point selection approaches. Although remarkably computation and memory efficient, random sampling can discard key features by chance. To overcome this, we introduce a novel local feature aggregation module to progressively increase the receptive field for each 3D point, thereby effectively preserving geometric details. Extensive experiments show that our RandLA-Net can process 1 million points in a single pass with up to 200x faster than existing approaches. Moreover, our RandLA-Net clearly surpasses state-of-the-art approaches for semantic segmentation on two large-scale benchmarks Semantic3D and SemanticKITTI.

/pdf/randla-net-efficient-semantic-segmentation-of-large-scale-4msnszot0z.pdf

RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Clouds

We introduce Position Adaptive Convolution (PAConv), a generic convolution operation for 3D point cloud processing. The key of PAConv is to construct the convolution kernel by dynamically assembling basic weight matrices stored in Weight Bank, where the coefficients of these weight matrices are self-adaptively learned from point positions through ScoreNet. In this way, the kernel is built in a data-driven manner, endowing PAConv with more flexibility than 2D convolutions to better handle the irregular and unordered point cloud data. Besides, the complexity of the learning process is reduced by combining weight matrices instead of brutally predicting kernels from point positions.Furthermore, different from the existing point convolution operators whose network architectures are often heavily engineered, we integrate our PAConv into classical MLP-based point cloud pipelines without changing network configurations. Even built on simple networks, our method still approaches or even surpasses the state-of-the-art models, and significantly improves baseline performance on both classification and segmentation tasks, yet with decent efficiency. Thorough ablation studies and visualizations are provided to understand PAConv. Code is released on https://github.com/CVMI-Lab/PAConv.

/pdf/paconv-position-adaptive-convolution-with-dynamic-kernel-1fw1x139tf.pdf

PAConv: Position Adaptive Convolution with Dynamic Kernel Assembling on Point Clouds

Ripe with possibilities offered by deep-learning techniques and useful in applications related to remote sensing, computer vision, and robotics, 3D point cloud semantic segmentation (PCSS) and point cloud segmentation (PCS) are attracting increasing interest. This article summarizes available data sets and relevant studies on recent developments in PCSS and PCS.

/pdf/linking-points-with-labels-in-3d-a-review-of-point-cloud-1venutor5l.pdf

Linking Points With Labels in 3D: A Review of Point Cloud Semantic Segmentation

Due to the sparsity and irregularity of the point cloud data, methods that directly consume points have become popular. Among all point-based models, graph convolutional networks (GCN) lead to notable performance by fully preserving the data granularity and exploiting point interrelation. However, point-based networks spend a significant amount of time on data structuring (e.g., Farthest Point Sampling (FPS) and neighbor points querying), which limit the speed and scalability. In this paper, we present a method, named Grid-GCN, for fast and scalable point cloud learning. Grid-GCN uses a novel data structuring strategy, Coverage-Aware Grid Query (CAGQ). By leveraging the efficiency of grid space, CAGQ improves spatial coverage while reducing the theoretical time complexity. Compared with popular sampling methods such as Farthest Point Sampling (FPS) and Ball Query, CAGQ achieves up to 50 times speed-up. With a Grid Context Aggregation (GCA) module, Grid-GCN achieves state-of-the-art performance on major point cloud classification and segmentation benchmarks with significantly faster runtime than previous studies. Remarkably, Grid-GCN achieves the inference speed of 50FPS on ScanNet using 81920 points as input. The supplementary xharlie.github.io/papers/GGCN_supCamReady.pdf and the code github.com/xharlie/Grid-GCN are released.

/pdf/grid-gcn-for-fast-and-scalable-point-cloud-learning-3kl21fxwd0.pdf

Grid-GCN for Fast and Scalable Point Cloud Learning

Standard convolution is inherently limited for semantic segmentation of point cloud due to its isotropy about features. It neglects the structure of an object, results in poor object delineation and small spurious regions in the segmentation result. This paper proposes a novel graph attention convolution (GAC), whose kernels can be dynamically carved into specific shapes to adapt to the structure of an object. Specifically, by assigning proper attentional weights to different neighboring points, GAC is designed to selectively focus on the most relevant part of them according to their dynamically learned features. The shape of the convolution kernel is then determined by the learned distribution of the attentional weights. Though simple, GAC can capture the structured features of point clouds for fine-grained segmentation and avoid feature contamination between objects. Theoretically, we provided a thorough analysis on the expressive capabilities of GAC to show how it can learn about the features of point clouds. Empirically, we evaluated the proposed GAC on challenging indoor and outdoor datasets and achieved the state-of-the-art results in both scenarios.

/pdf/graph-attention-convolution-for-point-cloud-semantic-12s36767gl.pdf

Graph Attention Convolution for Point Cloud Semantic Segmentation

Point cloud classification is quite challenging due to the influence of noise, occlusion, and the variety of types and sizes of objects. Currently, most methods mainly focus on subjectively designing and extracting features. However, the features rely on prior knowledge, and it is also difficult to accurately characterize the complex objects of point clouds. In this paper, we propose a concise multi-scale convolutional network (MSNet) for adaptive and robust point cloud classification. Both the local feature and global context are incorporated for this purpose. First, around each point, the spatial contexts of different sizes are partitioned as voxels of different scales. A voxel-based MSNet is then simultaneously applied at multiple scales to adaptively learn the discriminative local features. The class probability of a point cloud is predicted by fusing the features together across multiple scales. Finally, the predicted class probabilities of MSNet are optimized globally using the conditional random field (CRF) with a spatial consistency constraint. The proposed method was tested with data sets of mobile laser scanning (MLS), terrestrial laser scanning (TLS), and airborne laser scanning (ALS) point clouds. The experimental results show that the proposed method was able to achieve appreciable classification accuracies of 83.18%, 98.24%, and 97.02% on the MLS, TLS, and ALS data sets, respectively. The results also demonstrate that the proposed network has a strong generalization capability for classifying different kinds of point clouds under complex urban environments.

MSNet: Multi-Scale Convolutional Network for Point Cloud Classification

The combination of a camera and a 2D laser rangefinder (LRF) is widely used in robotics, mapping, and unmanned driving to simultaneously obtain the 3D geometry and color texture of a scene. However, data misregistration between the camera and the LRF frequently occurs due to the difficulty of precise installation and alignment between them. Extrinsic calibration between the camera and the LRF is necessary. In this study, a photogrammetric control field is used to perform a robust and accurate calibration between the camera and the LRF which requires only one shot. With the use of the photogrammetric control field as the common reference, extrinsic calibration between two sensors is decoupled by calibrating each sensor separately. First, using the room corner of the control field, the LRF is calibrated with respect to the control field by solving a simplified perspective-three-point problem. Then, a large number of control points are used to obtain the robust and accurate extrinsic parameters of the camera with respect to the control field. Experiments with simulated and real data were performed. The experimental results show that the proposed scheme is accurate, precise, and robust under different noise levels, and the calibration results can be utilized in indoor and outdoor applications.

Extrinsic Calibration between a Camera and a 2D Laser Rangefinder using a Photogrammetric Control Field.

Multiple two-dimensional laser rangefinders (LRFs) are applied in many applications like mobile robotics, autonomous vehicles, and three-dimensional reconstruction. The extrinsic calibration between LRFs is the first step to perform data fusion and practical application. In this paper, we proposed a simple method to calibrate LRFs based on a corner composed of three mutually perpendicular planes. In contrast to other methods that require a special pattern or assistance from other sensors, the trihedron corner needed in this method is common in daily environments. In practice, we can adjust the position of the LRFs to observe the corner until the laser scanning plane intersects with three planes of the corner. Then, we formed a Perspective-Three-Point problem to solve the position and orientation of each LRF at the common corner coordinate system. The method was validated with synthetic and real experiments, showing better performance than existing methods.

Extrinsic Calibration of Multiple Two-Dimensional Laser Rangefinders Based on a Trihedron.

The combination of area-scan camera and 2D laser rangefinder (LRF) can capture both textural and geometrical information from a scene at the same time, and has been widely used in various fields. Due to the differences of the installation position and acquisition mode, calibrating the extrinsic parameters, including the rotation and translation, of two sensors is necessary for fusing the camera image and LRF data. In this paper, a simple and flexible extrinsic calibration method is proposed by only acquiring a checkerboard trihedron once. Using checkerboard trihedron as a mobile referenced control field, the proposed method includes three steps to calibrate the extrinsic parameters. First, the rotation and translation between the trihedron and LRF are solved with a simplified perspective-three-point (P3P) solution; Second, using the collinear equation of checkerboard corners and their pixels in the image, the camera is calibrated with respect to the trihedron; Third, combining the last two steps, the rotation and translation parameters between the camera and LRF are finally calibrated with the intermediate referenced trihedron. After a lot of simulation and real experiments, the proposed method has been demonstrated to have the advantages of simple operation, strong robustness and high accuracy in real experiment.

/pdf/the-extrinsic-calibration-of-area-scan-camera-and-2d-laser-231dsqnvht.pdf

Yuchun Huang

Papers

Graph Attention Convolution for Point Cloud Semantic Segmentation

MSNet: Multi-Scale Convolutional Network for Point Cloud Classification

Extrinsic Calibration between a Camera and a 2D Laser Rangefinder using a Photogrammetric Control Field.

Extrinsic Calibration of Multiple Two-Dimensional Laser Rangefinders Based on a Trihedron.

The Extrinsic Calibration of Area-Scan Camera and 2D Laser Rangefinder (LRF) Using Checkerboard Trihedron