Point cloud learning has lately attracted increasing attention due to its wide applications in many areas, such as computer vision, autonomous driving, and robotics As a dominating technique in AI, deep learning has been successfully used to solve various 2D vision problems However, deep learning on point clouds is still in its infancy due to the unique challenges faced by the processing of point clouds with deep neural networks Recently, deep learning on point clouds has become even thriving, with numerous methods being proposed to address different problems in this area To stimulate future research, this paper presents a comprehensive review of recent progress in deep learning methods for point clouds It covers three major tasks, including 3D shape classification, 3D object detection and tracking, and 3D point cloud segmentation It also presents comparative results on several publicly available datasets, together with insightful observations and inspiring future research directions

Deep Learning for 3D Point Clouds: A Survey

We study the problem of efficient semantic segmentation for large-scale 3D point clouds. By relying on expensive sampling techniques or computationally heavy pre/post-processing steps, most existing approaches are only able to be trained and operate over small-scale point clouds. In this paper, we introduce RandLA-Net, an efficient and lightweight neural architecture to directly infer per-point semantics for large-scale point clouds. The key to our approach is to use random point sampling instead of more complex point selection approaches. Although remarkably computation and memory efficient, random sampling can discard key features by chance. To overcome this, we introduce a novel local feature aggregation module to progressively increase the receptive field for each 3D point, thereby effectively preserving geometric details. Extensive experiments show that our RandLA-Net can process 1 million points in a single pass with up to 200x faster than existing approaches. Moreover, our RandLA-Net clearly surpasses state-of-the-art approaches for semantic segmentation on two large-scale benchmarks Semantic3D and SemanticKITTI.

/pdf/randla-net-efficient-semantic-segmentation-of-large-scale-4msnszot0z.pdf

RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Clouds

We introduce Position Adaptive Convolution (PAConv), a generic convolution operation for 3D point cloud processing. The key of PAConv is to construct the convolution kernel by dynamically assembling basic weight matrices stored in Weight Bank, where the coefficients of these weight matrices are self-adaptively learned from point positions through ScoreNet. In this way, the kernel is built in a data-driven manner, endowing PAConv with more flexibility than 2D convolutions to better handle the irregular and unordered point cloud data. Besides, the complexity of the learning process is reduced by combining weight matrices instead of brutally predicting kernels from point positions.Furthermore, different from the existing point convolution operators whose network architectures are often heavily engineered, we integrate our PAConv into classical MLP-based point cloud pipelines without changing network configurations. Even built on simple networks, our method still approaches or even surpasses the state-of-the-art models, and significantly improves baseline performance on both classification and segmentation tasks, yet with decent efficiency. Thorough ablation studies and visualizations are provided to understand PAConv. Code is released on https://github.com/CVMI-Lab/PAConv.

/pdf/paconv-position-adaptive-convolution-with-dynamic-kernel-1fw1x139tf.pdf

PAConv: Position Adaptive Convolution with Dynamic Kernel Assembling on Point Clouds

Ripe with possibilities offered by deep-learning techniques and useful in applications related to remote sensing, computer vision, and robotics, 3D point cloud semantic segmentation (PCSS) and point cloud segmentation (PCS) are attracting increasing interest. This article summarizes available data sets and relevant studies on recent developments in PCSS and PCS.

/pdf/linking-points-with-labels-in-3d-a-review-of-point-cloud-1venutor5l.pdf

Linking Points With Labels in 3D: A Review of Point Cloud Semantic Segmentation

Due to the sparsity and irregularity of the point cloud data, methods that directly consume points have become popular. Among all point-based models, graph convolutional networks (GCN) lead to notable performance by fully preserving the data granularity and exploiting point interrelation. However, point-based networks spend a significant amount of time on data structuring (e.g., Farthest Point Sampling (FPS) and neighbor points querying), which limit the speed and scalability. In this paper, we present a method, named Grid-GCN, for fast and scalable point cloud learning. Grid-GCN uses a novel data structuring strategy, Coverage-Aware Grid Query (CAGQ). By leveraging the efficiency of grid space, CAGQ improves spatial coverage while reducing the theoretical time complexity. Compared with popular sampling methods such as Farthest Point Sampling (FPS) and Ball Query, CAGQ achieves up to 50 times speed-up. With a Grid Context Aggregation (GCA) module, Grid-GCN achieves state-of-the-art performance on major point cloud classification and segmentation benchmarks with significantly faster runtime than previous studies. Remarkably, Grid-GCN achieves the inference speed of 50FPS on ScanNet using 81920 points as input. The supplementary xharlie.github.io/papers/GGCN_supCamReady.pdf and the code github.com/xharlie/Grid-GCN are released.

/pdf/grid-gcn-for-fast-and-scalable-point-cloud-learning-3kl21fxwd0.pdf

Grid-GCN for Fast and Scalable Point Cloud Learning

Standard convolution is inherently limited for semantic segmentation of point cloud due to its isotropy about features. It neglects the structure of an object, results in poor object delineation and small spurious regions in the segmentation result. This paper proposes a novel graph attention convolution (GAC), whose kernels can be dynamically carved into specific shapes to adapt to the structure of an object. Specifically, by assigning proper attentional weights to different neighboring points, GAC is designed to selectively focus on the most relevant part of them according to their dynamically learned features. The shape of the convolution kernel is then determined by the learned distribution of the attentional weights. Though simple, GAC can capture the structured features of point clouds for fine-grained segmentation and avoid feature contamination between objects. Theoretically, we provided a thorough analysis on the expressive capabilities of GAC to show how it can learn about the features of point clouds. Empirically, we evaluated the proposed GAC on challenging indoor and outdoor datasets and achieved the state-of-the-art results in both scenarios.

/pdf/graph-attention-convolution-for-point-cloud-semantic-12s36767gl.pdf

Graph Attention Convolution for Point Cloud Semantic Segmentation

The recent release of worldwide SRTM 1 DEM and AW3D30 adds new members to the open global medium resolution (90–30 m ground spacing) digital elevation models. Together with the previously existing SRTM 3 and ASTER GDEM, their quality is of great interest to various scientific applications. This paper uses 1:50,000 DEM in Hubei Province of China as a reference to assess their vertical accuracy in terms of terrain types, slopes, and land cover. For ASTER GDEM and AW3D30, we further evaluate their accuracy in terms of the stack number, i.e., the number of scenes used to generate the DEM. It is found out that: (1) all of the DEMs have nearly the same horizontal offset due to the adoption of different datums; (2) the vertical accuracy varies in terms of terrain complexity, from ~5 m for plains, ~10 m for hills to ~20 m for mountains; (3) the vertical accuracy is negatively related to the tangent of terrain slope exponentially in forest areas and linearly in cultivated lands; (4) forest areas have the lowest vertical accuracy, comparing to built-up areas, wetland, and cultivate land areas while SRTM 1 and AW3D30 have the highest accuracy in all land cover classes; (5) the large elevation differences over forest areas are likely due to canopy coverage; and (6) for ASTER GDEM and AW3D30, their accuracy is in general positively related to the stack number. This study provides a practically useful quality specification and comprehensive understanding for these four global DEMs, especially the recently released worldwide SRTM 1 DEM and AW3D30.

Evaluation of Recently Released Open Global Digital Elevation Models of Hubei, China

Multi-view depth map reconstruction is regarded as a suitable approach for 3D generation of large-scale scenes due to its flexibility and scalability. However, there are challenges when this technique is applied to urban scenes where apparent man-made regular shapes may present. To address this need, this paper proposes a p lanarity constrained m ulti- v iew d epth (PMVD) map reconstruction method. Starting with image segmentation and feature matching for each input image, the main procedure is iterative optimization under the constraints of planar geometry and smoothness. A set of candidate local planes are first generated by an extended PatchMatch method. The image matching costs are then computed and aggregated by an adaptive-manifold filter (AMF), whereby the smoothness constraint is applied to adjacent pixels through belief propagation. Finally, multiple criteria are used to eliminate image matching outliers. (Vertical) aerial images, oblique (aerial) images and ground images are used for qualitative and quantitative evaluations. The experiments demonstrated that the PMVD outperforms the popular multi-view depth map reconstruction with an accuracy two times better for the aerial datasets and achieves an outcome comparable to the state-of-the-art for ground images. As expected, PMVD is able to preserve the planarity for piecewise flat structures in urban scenes and restore the edges in depth discontinuous areas.

Planarity constrained multi-view depth map reconstruction for urban scenes

Multi-view depth map reconstruction is a popular approach to generate 3D information with good flexibility and scalability. However, texture-weak regions and repeated textures challenge these methods in urban scenes. To address this need, this paper proposes an image-triangle based multi-view 3D reconstruction (IMGTR) method. Starting from constructing a density-adaptive image triangulation for each image, the main procedure is to determine the corresponding object plane for each image triangle under an objective function consisting of the image similarity measure, the smoothness constraint and the continuity constraint on the edges between adjacent triangles. Qualitative and quantitative experiments show that the proposed method can reconstruct complex urban structures more accurate and achieve higher fidelity in planar urban structures than some recently released multi-view stereo algorithms. Specifically, for the Vaihingen dataset IMGTR achieves an average improvement of 4.27% compared to PMVD, 50.94% to SURE and 54.76% to COLMAP in position accuracy.

IMGTR: Image-triangle based multi-view 3D reconstruction for urban scenes

. A complete building model reconstruction needs data collected from both air and ground. The former often has sparse coverage on building facades, while the latter usually is unable to observe the building rooftops. Attempting to solve the missing data issues in building reconstruction from single data source, we describe an approach for complete building reconstruction that integrates airborne LiDAR data and ground smartphone imagery. First, by taking advantages of GPS and digital compass information embedded in the image metadata of smartphones, we are able to find airborne LiDAR point clouds for the corresponding buildings in the images. In the next step, Structure-from-Motion and dense multi-view stereo algorithms are applied to generate building point cloud from multiple ground images. The third step extracts building outlines respectively from the LiDAR point cloud and the ground image point cloud. An automated correspondence between these two sets of building outlines allows us to achieve a precise registration and combination of the two point clouds, which ultimately results in a complete and full resolution building model. The developed approach overcomes the problem of sparse points on building facades in airborne LiDAR and the deficiency of rooftops in ground images such that the merits of both datasets are utilized.

/pdf/integrating-smartphone-images-and-airborne-lidar-data-for-37rlrp1ih0.pdf

Yaolin Hou

Papers

Graph Attention Convolution for Point Cloud Semantic Segmentation

Evaluation of Recently Released Open Global Digital Elevation Models of Hubei, China

Planarity constrained multi-view depth map reconstruction for urban scenes

IMGTR: Image-triangle based multi-view 3D reconstruction for urban scenes

Integrating smartphone images and airborne lidar data for complete urban building modelling