We propose a novel deep learning-based framework to tackle the challenge of semantic segmentation of large-scale point clouds of millions of points. We argue that the organization of 3D point clouds can be efficiently captured by a structure called superpoint graph (SPG), derived from a partition of the scanned scene into geometrically homogeneous elements. SPGs offer a compact yet rich representation of contextual relationships between object parts, which is then exploited by a graph convolutional network. Our framework sets a new state of the art for segmenting outdoor LiDAR scans (+11.9 and +8.8 mIoU points for both Semantic3D test sets), as well as indoor scans (+12.4 mIoU points for the S3DIS dataset).

/pdf/large-scale-point-cloud-semantic-segmentation-with-34l1w5clj2.pdf

Large-Scale Point Cloud Semantic Segmentation with Superpoint Graphs

We study the problem of perceiving forest or mountain trails from a single monocular image acquired from the viewpoint of a robot traveling on the trail itself. Previous literature focused on trail segmentation, and used low-level features such as image saliency or appearance contrast; we propose a different approach based on a deep neural network used as a supervised image classifier. By operating on the whole image at once, our system outputs the main direction of the trail compared to the viewing direction. Qualitative and quantitative results computed on a large real-world dataset (which we provide for download) show that our approach outperforms alternatives, and yields an accuracy comparable to the accuracy of humans that are tested on the same image classification task. Preliminary results on using this information for quadrotor control in unseen trails are reported. To the best of our knowledge, this is the first letter that describes an approach to perceive forest trials, which is demonstrated on a quadrotor micro aerial vehicle.

https://www.zora.uzh.ch/id/eprint/125475/1/RAL16_Giusti.pdf

A Machine Learning Approach to Visual Perception of Forest Trails for Mobile Robots

3D semantic scene labeling is fundamental to agents operating in the real world. In particular, labeling raw 3D point sets from sensors provides fine-grained semantics. Recent works leverage the capabilities of Neural Networks(NNs), but are limited to coarse voxel predictions and do not explicitly enforce global consistency. We present SEGCloud, an end-to-end framework to obtain 3D point-level segmentation that combines the advantages of NNs, trilinear interpolation(TI) and fully connected Conditional Random Fields (FC-CRF). Coarse voxel predictions from a 3D Fully Convolutional NN are transferred back to the raw 3D points via trilinear interpolation. Then the FC-CRF enforces global consistency and provides fine-grained semantics on the points. We implement the latter as a differentiable Recurrent NN to allow joint optimization. We evaluate the framework on two indoor and two outdoor 3D datasets (NYU V2, S3DIS, KITTI, Semantic3D.net), and show performance comparable or superior to the state-of-the-art on all datasets.

SEGCloud: Semantic Segmentation of 3D Point Clouds

We present a micro aerial vehicle (MAV) system, built with inexpensive off-the-shelf hardware, for autonomously following trails in unstructured, outdoor environments such as forests. The system introduces a deep neural network (DNN) called TrailNet for estimating the view orientation and lateral offset of the MAV with respect to the trail center. The DNN-based controller achieves stable flight without oscillations by avoiding overconfident behavior through a loss function that includes both label smoothing and entropy reward. In addition to the TrailNet DNN, the system also utilizes vision modules for environmental awareness, including another DNN for object detection and a visual odometry component for estimating depth for the purpose of low-level obstacle detection. All vision systems run in real time on board the MAV via a Jetson TX1. We provide details on the hardware and software used, as well as implementation details. We present experiments showing the ability of our system to navigate forest trails more robustly than previous techniques, including autonomous flights of 1 km.

Toward low-flying autonomous MAV trail navigation using deep neural networks for environmental awareness

Ripe with possibilities offered by deep-learning techniques and useful in applications related to remote sensing, computer vision, and robotics, 3D point cloud semantic segmentation (PCSS) and point cloud segmentation (PCS) are attracting increasing interest. This article summarizes available data sets and relevant studies on recent developments in PCSS and PCS.

/pdf/linking-points-with-labels-in-3d-a-review-of-point-cloud-1venutor5l.pdf

Linking Points With Labels in 3D: A Review of Point Cloud Semantic Segmentation

In this paper, we focus on 3D point cloud classification by assigning semantic labels to each point in the scene. We propose to use simplified Markov networks to model the contextual relations between points, where the node potentials are calculated from point-wise classification results using off-the-shelf classifiers, such as Random Forest and Support Vector Machines, and the edge potentials are set by physical distance between points. Our experimental results show that this approach yields comparable if not better results with improved speed compared with state-of-the-art methods. We also propose a novel robust neighborhood filtering method to exclude outliers in the neighborhood of points, in order to reduce noise in local geometric statistics when extracting features and also to reduce number of false edges when constructing Markov networks. We show that applying robust neighborhood filtering improves the results when classifying point clouds with more object categories.

Simplified markov random fields for efficient semantic labeling of 3D point clouds

We describe a framework for finding and tracking “trails” for autonomous outdoor robot navigation. Through a combination of visual cues and ladar-derived structural information, the algorithm is able to follow paths which pass through multiple zones of terrain smoothness, border vegetation, tread material, and illumination conditions. Our shape-based visual trail tracker assumes that the approaching trail region is approximately triangular under perspective. It generates region hypotheses from a learned distribution of expected trail width and curvature variation, and scores them using a robust measure of color and brightness contrast with flanking regions. The structural component analogously rewards hypotheses which correspond to empty or low-density regions in a groundstrike-filtered ladar obstacle map. Our system's performance is analyzed on several long sequences with diverse appearance and structural characteristics. Ground-truth segmentations are used to quantify performance where available, and several alternative algorithms are compared on the same data.

Appearance contrast for fast, robust trail-following

We describe a wheeled robotic system which navigates along outdoor “trails” intended for hikers and bikers. Through a combination of appearance and structural cues derived from stereo omnidirectional color cameras and a tiltable laser range-finder, the system is able to detect and track rough paths despite widely varying tread material, border vegetation, and illumination conditions. The approaching trail region is efficiently segmented in a top-down fashion based on color, brightness, and/or height contrast with flanking areas, and a differential motion planner searches for maximally-safe paths within that region according to several criteria. When the trail tracker’s confidence drops the robot slows down to allow a more detailed search, and when it senses a dangerous situation due to excessive slope, dense trailside obstacles, or visual trail segmentation failure, it stops entirely to acquire and analyze a ladar-derived point cloud in order to reset the tracker. Our system’s ability to negotiate a variety of challenging trail types over long distances is demonstrated through a number of live runs through different terrain and in different weather conditions.

A Trail-Following Robot Which Uses Appearance and Structural Cues

We propose a simple contrast-based method for tree detection and shape estimation from ground-plane perspective images for purposes of counting, classification, modeling, or robotic obstacle avoidance. Under the assumption that tree trunks are relatively narrow and vertical shapes which strongly differ in appearance from the scene background and have boundaries of opposite contrast, we apply a bank of bar filters parametrized from camera intrinsics combined with trunk location and diameter limits, and integrate results vertically. Non-maximum suppression is applied to candidates in the resulting trunk likelihood image. We present results demonstrating the effectiveness of our tree detection algorithm on a variety of forested images obtained directly from our robot as well as sampled from the web, and quantify performance using ground truth on a subset of those images. We also compare the results of our method with several related published approaches.

Tree trunk detection using contrast templates

We describe a system which follows “trails” for autonomous outdoor robot navigation. Through a combination of visual cues provided by stereo omnidirectional color cameras and ladar-based structural information, the algorithm is able to detect and track rough paths despite widely varying tread material, border vegetation, and illumination conditions. The approaching trail region is simply modeled as a circular arc of constant width. Using an adaptive measure of color and brightness contrast between a hypothetical region and flanking areas, the tracker performs a robust randomized search for the most likely trail region and robot pose relative to it with no a priori appearance model. Stereo visual odometry improves tracker dynamics on uneven terrain and permits local obstacle map maintenance. A motion planner is also described which takes the trail shape estimate and local map to plan smooth trajectories around in-trail and near-trail hazards. Our system's performance is analyzed on several long sequences with diverse appearance and structural characteristics using ground-truth segmentations.

/pdf/trail-following-with-omnidirectional-vision-1uc36h6caw.pdf

Yan Lu

Papers

Simplified markov random fields for efficient semantic labeling of 3D point clouds

Appearance contrast for fast, robust trail-following

A Trail-Following Robot Which Uses Appearance and Structural Cues

Tree trunk detection using contrast templates

Trail following with omnidirectional vision