Semantic segmentation of urban scenes is an essential component in various applications of autonomous driving. It makes great progress with the rise of deep learning technologies. Most of the current semantic segmentation networks use single-modal sensory data, which are usually the RGB images produced by visible cameras. However, the segmentation performance of these networks is prone to be degraded when lighting conditions are not satisfied, such as dim light or darkness. We find that thermal images produced by thermal imaging cameras are robust to challenging lighting conditions. Therefore, in this article, we propose a novel RGB and thermal data fusion network named FuseSeg to achieve superior performance of semantic segmentation in urban scenes. The experimental results demonstrate that our network outperforms the state-of-the-art networks. Note to Practitioners —This article investigates the problem of semantic segmentation of urban scenes when lighting conditions are not satisfied. We provide a solution to this problem via information fusion with RGB and thermal data. We build an end-to-end deep neural network, which takes as input a pair of RGB and thermal images and outputs pixel-wise semantic labels. Our network could be used for urban scene understanding, which serves as a fundamental component of many autonomous driving tasks, such as environment modeling, obstacle avoidance, motion prediction, and planning. Moreover, the simple design of our network allows it to be easily implemented using various deep learning frameworks, which facilitates the applications on different hardware or software platforms.

FuseSeg: Semantic Segmentation of Urban Scenes Based on RGB and Thermal Data Fusion

Crack is one of the most common road distresses which may pose road safety hazards. Generally, crack detection is performed by either certified inspectors or structural engineers. This task is, however, time-consuming, subjective and labor-intensive. In this paper, a novel road crack detection algorithm which is based on deep learning and adaptive image segmentation is proposed. Firstly, a deep convolutional neural network is trained to determine whether an image contains cracks or not. The images containing cracks are then smoothed using bilateral filtering, which greatly minimizes the number of noisy pixels. Finally, cracks are extracted from the road surface using an adaptive thresholding method. The experimental results illustrate that our network can classify images with an accuracy of 99.92%, and the cracks can be successfully extracted from the images using our proposed thresholding algorithm.

Road Crack Detection Using Deep Convolutional Neural Network and Adaptive Thresholding

Road Crack Detection Using Deep Convolutional Neural Network and Adaptive Thresholding.

Many modern robotics applications rely on 3D maps of the environment. Due to the large memory requirements of dense 3D maps, compression techniques are often necessary to store or transmit 3D maps efficiently. In this work, we investigate the problem of compressing dense 3D point cloud maps such as those obtained from an autonomous vehicle in large outdoor environments. We tackle the problem by learning a set of local feature descriptors from which the point cloud can be reconstructed efficiently and effectively. We propose a novel deep convolutional autoencoder architecture that directly operates on the points themselves so that we avoid voxelization. Additionally, we propose a deconvolution operator to upsample point clouds, which allows us to decompress to an arbitrary density. Our experiments show that our learned compression achieves better reconstructions at the same bit rate compared to other state-of-the-art compression algorithms. We furthermore demonstrate that our approach generalizes well to different LiDAR sensors. For example, networks learned on maps generated from KITTI point clouds still achieve state-of-the-art compression results for maps generated from nuScences point clouds.

Deep Compression for Dense Point Cloud Maps

Recently, the development and application of lane line departure warning systems have been in the market. For any of the systems, the key part of lane line tracking, lane line identification, or lane line departure warning is whether it can accurately and quickly detect lane lines. Since 1990s, they have been studied and implemented for the situations defined by the good viewing conditions and the clear lane markings on road. After then, the accuracy for particular situations, the robustness for a wide range of scenarios, time efficiency and integration into higher-order tasks define visual lane line detection and tracking as a continuing research subject. At present, these kinds of lane marking line detection methods based on machine vision and image processing can be divided into two categories: the traditional image processing and semantic segmentation (includes deep learning) methods. The former mainly involves feature-based and model-based steps, and which can be classified into similarity- and discontinuity-based ones; and the model-based step includes different parametric straight line, curve or pattern models. The semantic segmentation includes different machine learning, neural network and deep learning methods, which is the new trend for the research and application of lane line departure warning systems. This paper describes and analyzes the lane line departure warning systems, image processing algorithms and semantic segmentation methods for lane line detection.

Lane departure warning systems and lane line detection methods based on image processing and semantic segmentation: A review

Due to the enormous volume of point cloud data, transmitting and storing the data requires large bandwidth and storage space. It could be a critical bottleneck, especially in tasks such as autonomous driving. In this letter, we propose a novel point cloud compression algorithm based on clustering. The proposed scheme starts with a range image-based segmentation step, which segments the three-dimensional (3-D) range data into ground and main objects. Then, it introduces a novel prediction method according to the segmented regions’ shape. This prediction method is inspired by the depth modeling modes used in 3-D high-efficiency video coding for depth map coding. Finally, the few prediction residual is efficiently compressed with several lossless or lossy data compression techniques. Experimental results show that the proposed algorithm can largely eliminate the spatial redundant information of the point cloud data. The lossless compression scheme reaches a compression ratio of nearly 5%, which means that the point cloud is compressed to 5% of its original size without any distance distortion. Compared with other methods, the proposed compression algorithm also shows better performance.

A Novel Point Cloud Compression Algorithm Based on Clustering

Autonomous 3-D exploration with unmanned aerial vehicles (UAVs) is increasingly prevalent for environment monitoring without human intervention. In this article, we present a systematic solution toward efficient UAV exploration in 3-D environments. Innovatively, a road map is incrementally built and maintained along with the exploration process, which explicitly exhibits the topological structure of the 3-D environment. By simplifying the environment, the road map can efficiently provide the information gain and the cost-to-go for a candidate region to be explored, which are two quantities for next-best-view (NBV) evaluation, thus prompting the efficiency for NBV determination. In addition, with reference to the global plan queried on the road map, we propose a local planner based on the potential field method that drives the robot to the information-rich area during the navigation process, which further improves the exploration efficiency. The proposed framework and its composed modules are verified in various 3-D environments, which exhibit their distinctive features in NBV selection and better performance in improving the exploration efficiency than other methods.

Efficient Autonomous Exploration With Incrementally Built Topological Map in 3-D Environments

A survey of learning-based robot motion planning

Lane detection is very important for self-driving vehicles. In recent years, computer stereo vision has been prevalently used to enhance the accuracy of the lane detection systems. This paper mainly presents a multiple lane detection algorithm developed based on optimised dense disparity map estimation, where the disparity information obtained at time $t_{n}$ is utilised to optimise the process of disparity estimation at time $t_{n+1}(n \ge 0)$. This is achieved by estimating the road model at time $t_{n}$ and then controlling the search range for the disparity estimation at time $t_{n+1}$. The lanes are then detected using our previously published algorithm, where the vanishing point information is used to model the lanes. The experimental results illustrate that the runtime of the disparity estimation is reduced by around ${37{\%} }$and the accuracy of the lane detection is about ${99{\%}}$.

Multiple Lane Detection Algorithm Based on Optimised Dense Disparity Map Estimation

In this paper, we propose a novel perceptual-based intra coding optimization algorithm for the High Efficiency Video Coding (HEVC) using deep convolution networks (DCNs). According to the saliency map, the algorithm can intelligently adjust bit rate allocation between the salient and non-salient regions of the video. The proposed strategy mainly consists of two techniques, saliency map extraction, and intelligent bit rate allocation. First, we train a DCN model to generate the saliency map that highlights semantically salient regions. Compared with the texture-based region of interest (ROI) extraction techniques, our model is more consistent with the human visual system (HVS). Second, based on the saliency map, a modified rate-distortion optimization (RDO) method is designed to adaptively adjust bit rate allocation. As a result, the quality of the salient regions will be improved by allocating more bits while allocating fewer bit rates for the non-salient regions. The experimental results demonstrate that our approach can deal with multiple types of video to enhance the visual experience. For conventional videos, the proposed method achieves 0.64-dB PSNR improvement for the salient regions and saves 3.02% bit rate on average compared with HM16.7. Moreover, for conversational videos, the proposed method can significantly reduce the bit rate by 8.65% without dropping the quality of important regions.

/pdf/perceptual-based-hevc-intra-coding-optimization-using-deep-1le7ooysah.pdf

Han Ma

Papers

A Novel Point Cloud Compression Algorithm Based on Clustering

Efficient Autonomous Exploration With Incrementally Built Topological Map in 3-D Environments

A survey of learning-based robot motion planning

Multiple Lane Detection Algorithm Based on Optimised Dense Disparity Map Estimation

Perceptual-Based HEVC Intra Coding Optimization Using Deep Convolution Networks