3D semantic scene labeling is fundamental to agents operating in the real world. In particular, labeling raw 3D point sets from sensors provides fine-grained semantics. Recent works leverage the capabilities of Neural Networks(NNs), but are limited to coarse voxel predictions and do not explicitly enforce global consistency. We present SEGCloud, an end-to-end framework to obtain 3D point-level segmentation that combines the advantages of NNs, trilinear interpolation(TI) and fully connected Conditional Random Fields (FC-CRF). Coarse voxel predictions from a 3D Fully Convolutional NN are transferred back to the raw 3D points via trilinear interpolation. Then the FC-CRF enforces global consistency and provides fine-grained semantics on the points. We implement the latter as a differentiable Recurrent NN to allow joint optimization. We evaluate the framework on two indoor and two outdoor 3D datasets (NYU V2, S3DIS, KITTI, Semantic3D.net), and show performance comparable or superior to the state-of-the-art on all datasets.

SEGCloud: Semantic Segmentation of 3D Point Clouds

Current 3D object detection methods are heavily influenced by 2D detectors. In order to leverage architectures in 2D detectors, they often convert 3D point clouds to regular grids (i.e., to voxel grids or to bird's eye view images), or rely on detection in 2D images to propose 3D boxes. Few works have attempted to directly detect objects in point clouds. In this work, we return to first principles to construct a 3D detection pipeline for point cloud data and as generic as possible. However, due to the sparse nature of the data -- samples from 2D manifolds in 3D space -- we face a major challenge when directly predicting bounding box parameters from scene points: a 3D object centroid can be far from any surface point thus hard to regress accurately in one step. To address the challenge, we propose VoteNet, an end-to-end 3D object detection network based on a synergy of deep point set networks and Hough voting. Our model achieves state-of-the-art 3D detection on two large datasets of real 3D scans, ScanNet and SUN RGB-D with a simple design, compact model size and high efficiency. Remarkably, VoteNet outperforms previous methods by using purely geometric information without relying on color images.

Deep Hough Voting for 3D Object Detection in Point Clouds

3D scene analysis in terms of automatically assigning 3D points a respective semantic label has become a topic of great importance in photogrammetry, remote sensing, computer vision and robotics. In this paper, we address the issue of how to increase the distinctiveness of geometric features and select the most relevant ones among these for 3D scene analysis. We present a new, fully automated and versatile framework composed of four components: (i) neighborhood selection, (ii) feature extraction, (iii) feature selection and (iv) classification. For each component, we consider a variety of approaches which allow applicability in terms of simplicity, efficiency and reproducibility, so that end-users can easily apply the different components and do not require expert knowledge in the respective domains. In a detailed evaluation involving 7 neighborhood definitions, 21 geometric features, 7 approaches for feature selection, 10 classifiers and 2 benchmark datasets, we demonstrate that the selection of optimal neighborhoods for individual 3D points significantly improves the results of 3D scene analysis. Additionally, we show that the selection of adequate feature subsets may even further increase the quality of the derived results while significantly reducing both processing time and memory consumption.

Semantic point cloud interpretation based on optimal neighborhoods, relevant features and efficient classifiers

In this work we address the task of the contextual classification of an airborne LiDAR point cloud. For that purpose, we integrate a Random Forest classifier into a Conditional Random Field (CRF) framework. It is a flexible approach for obtaining a reliable classification result even in complex urban scenes. In this way, we benefit from the consideration of context on the one hand and from the opportunity to use a large amount of features on the other hand. Considering the interactions in our experiments increases the overall accuracy by 2%, though a larger improvement becomes apparent in the completeness and correctness of some of the seven classes discerned in our experiments. We compare the Random Forest approach to linear models for the computation of unary and pairwise potentials of the CRF, and investigate the relevance of different features for the LiDAR points as well as for the interaction of neighbouring points. In a second step, building objects are detected based on the classified point cloud. For that purpose, the CRF probabilities for the classes are plugged into a Markov Random Field as unary potentials, in which the pairwise potentials are based on a Potts model. The 2D binary building object masks are extracted and evaluated by the benchmark ISPRS Test Project on Urban Classification and 3D Building Reconstruction. The evaluation shows that the main buildings (larger than 50 m 2 ) can be detected very reliably with a correctness larger than 96% and a completeness of 100%.

Contextual classification of lidar data and building object detection in urban areas

The depth information of RGB-D sensors has greatly simplified some common challenges in computer vision and enabled breakthroughs for several tasks. In this paper, we propose to use depth maps for object detection and design a 3D detector to overcome the major difficulties for recognition, namely the variations of texture, illumination, shape, viewpoint, clutter, occlusion, self-occlusion and sensor noises. We take a collection of 3D CAD models and render each CAD model from hundreds of viewpoints to obtain synthetic depth maps. For each depth rendering, we extract features from the 3D point cloud and train an Exemplar-SVM classifier. During testing and hard-negative mining, we slide a 3D detection window in 3D space. Experiment results show that our 3D detector significantly outperforms the state-of-the-art algorithms for both RGB and RGB-D images, and achieves about ×1.7 improvement on average precision compared to DPM and R-CNN. All source code and data are available online.

/pdf/sliding-shapes-for-3d-object-detection-in-depth-images-45ex60s8q5.pdf

Sliding Shapes for 3D Object Detection in Depth Images

The problem of laser scan analysis gained significant attention within the last decade. The standard approach to point cloud classification utilizes Markov Random Fields (MRF). Usually, a subclass of MRFs, Associative Markov Networks (AMNs), are used. In AMN the pairwise potential function is constant for a pair of different class labels. In some cases this constraint is too rigorous since it does not allow expressing some natural interactions between objects, such as “roof is likely to be above the ground”. In this work we use the general form of pairwise potentials instead. We show how to perform a tractable inference in such network using a message-passing based algorithm. Oversegmentation technique is used to subsample a scan; this helps to eliminate noise, improve efficiency and makes possible to use natural edge features. Experiments on two datasets obtained by an airborne laser scanner show that non-associative Markov networks usage leads to improvement in classification accuracy.

/pdf/non-associative-markov-networks-for-3d-point-cloud-1baekggnz8.pdf

Non-associative markov networks for 3d point cloud classification

. We present a method for automatic object localization and recognition in 3D point clouds representing outdoor urban scenes. The method is based on the implicit shape models (ISM) framework, which recognizes objects by voting for their center locations. It requires only few training examples per class, which is an important property for practical use. We also introduce and evaluate an improved version of the spin image descriptor, more robust to point density variation and uncertainty in normal direction estimation. Our experiments reveal a significant impact of these modifications on the recognition performance. We compare our results against the state-of-the-art method and get significant improvement in both precision and recall on the Ohio dataset, consisting of combined aerial and terrestrial LiDAR scans of 150,000 m2 of urban area in total.

/pdf/implicit-shape-models-for-object-detection-in-3d-point-4np8jzcab5.pdf

Implicit shape models for object detection in 3d point clouds

We address the problem of object class segmentation of 3D point clouds. Each point of a cloud should be assigned a class label determined by the category of the object it belongs to. Non-associative Markov networks have been applied to this task recently. Indeed, they impose more flexible constraints on segmentation results in contrast to the associative ones. We show how to train non-associative Markov networks in a principled manner using the structured Support Vector Machine (SVM) formalism. In contrast to prior work we use the kernel trick which makes our method one of the first non-linear methods for max-margin Markov Random Field training applied to 3D point cloud segmentation. We evaluate our method on airborne and terrestrial laser scans. In comparison to the other non-linear training techniques our method shows higher accuracy.

/pdf/cutting-plane-training-of-non-associative-markov-network-for-1a23idxvph.pdf

Cutting-Plane Training of Non-associative Markov Network for 3D Point Cloud Segmentation

Some embodiments of the invention include a method for providing a 3D-point cloud using a geodetic surveying instrument. Some embodiments also include an image capturing unit. A method, according to some embodiments may include scanning a surrounding with the surveying instrument according to a defined scanning region with at least partly covering the object and generating a scanning point cloud corresponding to the scanning region with reference to a surveying coordinate system which is defined by the surveying instrument and generating a first image on side of the surveying instrument covering a region basically corresponding to the scanning region, the first image representing a reference image the pose of which is known with reference to the surveying coordinate system due to the position and orientation of the surveying instrument for acquiring data the first image is base on.

Providing a point cloud using a surveying instrument and a camera device

A device may include a laser configured to rotate about an azimuth axis of the device and an elevation axis of the device. The device may also include a plurality of cameras configured to rotate about the azimuth axis and fixed with respect to the elevation axis. The cameras may be configured to have a collective field-of-view that includes a collective elevation field-of-view and a collective azimuth field-of-view. The device may also include a control system configured to direct the laser to rotate about the azimuth axis and the elevation axis such that the laser is configured to scan a setting. The control system may also be configured to direct the cameras to each begin capturing an image of the setting at substantially the same time.

Alexander Velizhev

Papers

Non-associative markov networks for 3d point cloud classification

Implicit shape models for object detection in 3d point clouds

Cutting-Plane Training of Non-associative Markov Network for 3D Point Cloud Segmentation

Providing a point cloud using a surveying instrument and a camera device

Multi-camera laser scanner