Robust detection and tracking of objects is crucial for the deployment of autonomous vehicle technology. Image based benchmark datasets have driven development in computer vision tasks such as object detection, tracking and segmentation of agents in the environment. Most autonomous vehicles, however, carry a combination of cameras and range sensors such as lidar and radar. As machine learning based methods for detection and tracking become more prevalent, there is a need to train and evaluate such methods on datasets containing range sensor data along with images. In this work we present nuTonomy scenes (nuScenes), the first dataset to carry the full autonomous vehicle sensor suite: 6 cameras, 5 radars and 1 lidar, all with full 360 degree field of view. nuScenes comprises 1000 scenes, each 20s long and fully annotated with 3D bounding boxes for 23 classes and 8 attributes. It has 7x as many annotations and 100x as many images as the pioneering KITTI dataset. We define novel 3D detection and tracking metrics. We also provide careful dataset analysis as well as baselines for lidar and image based detection and tracking. Data, development kit and more information are available online.

nuScenes: A multimodal dataset for autonomous driving

/pdf/nuscenes-a-multimodal-dataset-for-autonomous-driving-3znmpkqyuu.pdf

nuScenes: A Multimodal Dataset for Autonomous Driving

Perception system design is a vital step in the development of an autonomous vehicle (AV). With the vast selection of available off-the-shelf schemes and seemingly endless options of sensor systems implemented in research and commercial vehicles, it can be difficult to identify the optimal system for one’s AV application. This article presents a comprehensive review of the state-of-the-art AV perception technology available today. It provides up-to-date information about the advantages, disadvantages, limits, and ideal applications of specific AV sensors; the most prevalent sensors in current research and commercial AVs; autonomous features currently on the market; and localization and mapping methods currently implemented in AV research. This information is useful for newcomers to the AV field to gain a greater understanding of the current AV solution landscape and to guide experienced researchers towards research areas requiring further development. Furthermore, this paper highlights future research areas and draws conclusions about the most effective methods for AV perception and its effect on localization and mapping. Topics discussed in the Perception and Automotive Sensors section focus on the sensors themselves, whereas topics discussed in the Localization and Mapping section focus on how the vehicle perceives where it is on the road, providing context for the use of the automotive sensors. By improving on current state-of-the-art perception systems, AVs will become more robust, reliable, safe, and accessible, ultimately providing greater efficiency, mobility, and safety benefits to the public.

Autonomous vehicle perception: The technology of today and tomorrow

A mobile laser scanning (MLS) system allows direct collection of accurate 3D point information in unprecedented detail at highway speeds and at less than traditional survey costs, which serves the fast growing demands of transportation-related road surveying including road surface geometry and road environment. As one type of road feature in traffic management systems, road markings on paved roadways have important functions in providing guidance and information to drivers and pedestrians. This paper presents a stepwise procedure to recognize road markings from MLS point clouds. To improve computational efficiency, we first propose a curb-based method for road surface extraction. This method first partitions the raw MLS data into a set of profiles according to vehicle trajectory data, and then extracts small height jumps caused by curbs in the profiles via slope and elevation-difference thresholds. Next, points belonging to the extracted road surface are interpolated into a geo-referenced intensity image using an extended inverse-distance-weighted (IDW) approach. Finally, we dynamically segment the geo-referenced intensity image into road-marking candidates with multiple thresholds that correspond to different ranges determined by point-density appropriate normality. A morphological closing operation with a linear structuring element is finally used to refine the road-marking candidates by removing noise and improving completeness. This road-marking extraction algorithm is comprehensively discussed in the analysis of parameter sensitivity and overall performance. An experimental study performed on a set of road markings with ground-truth shows that the proposed algorithm provides a promising solution to the road-marking extraction from MLS data.

Using mobile laser scanning data for automated extraction of road markings

Synthetic data is an increasingly popular tool for training deep learning models, especially in computer vision but also in other areas. In this work, we attempt to provide a comprehensive survey of the various directions in the development and application of synthetic data. First, we discuss synthetic datasets for basic computer vision problems, both low-level (e.g., optical flow estimation) and high-level (e.g., semantic segmentation), synthetic environments and datasets for outdoor and urban scenes (autonomous driving), indoor scenes (indoor navigation), aerial navigation, simulation environments for robotics, applications of synthetic data outside computer vision (in neural programming, bioinformatics, NLP, and more); we also survey the work on improving synthetic data development and alternative ways to produce it such as GANs. Second, we discuss in detail the synthetic-to-real domain adaptation problem that inevitably arises in applications of synthetic data, including synthetic-to-real refinement with GAN-based models and domain adaptation at the feature/model level without explicit data transformations. Third, we turn to privacy-related applications of synthetic data and review the work on generating synthetic datasets with differential privacy guarantees. We conclude by highlighting the most promising directions for further work in synthetic data studies.

Synthetic Data for Deep Learning

High quality perception is essential for autonomous driving (AD) systems. To reach the accuracy and robustness that are required by such systems, several types of sensors must be combined. Currently, mostly cameras and laser scanners (lidar) are deployed to build a representation of the world around the vehicle. While radar sensors have been used for a long time in the automotive industry, they are still under-used for AD despite their appealing characteristics (notably, their ability to measure the relative speed of obstacles and to operate even in adverse weather conditions). To a large extent, this situation is due to the relative lack of automotive datasets with real radar signals that are both raw and annotated. In this work, we introduce CARRADA, a dataset of synchronized camera and radar recordings with range-angle-Doppler annotations. We also present a semi-automatic annotation approach, which was used to annotate the dataset, and a radar semantic segmentation baseline, which we evaluate on several metrics. Both our code and dataset are available online.

CARRADA Dataset: Camera and Automotive Radar with Range-Angle-Doppler Annotations

In recent years, deep learning models have resulted in a huge amount of progress in various areas, including computer vision. By nature, the supervised training of deep models requires a large amount of data to be available. This ideal case is usually not tractable as the data annotation is a tremendously exhausting and costly task to perform. An alternative is to use synthetic data. In this paper, we take a comprehensive look into the effects of replacing real data with synthetic data. We further analyze the effects of having a limited amount of real data. We use multiple synthetic and real datasets along with a simulation tool to create large amounts of cheaply annotated synthetic data. We analyze the domain similarity of each of these datasets. We provide insights about designing a methodological procedure for training deep networks using these datasets.

How much real data do we actually need: Analyzing object detection performance using synthetic and real data

This paper describes a method of road segmentation and pattern recognition for road marking analysis. Efficient segmentation and pattern recognition are difficult due to other vehicles, outdoor lighting or shadows. Rectilinear marking and arrow extraction is carried out by mathematical morphology in order Io take into account geometrical characteristics (size for example). Thus, rectilinear objects such as white lines are extracted by Hough transform. We use Fourier descriptors to describe the boundaries of an object. Thus, a KNN classifier recognizes the object from a training base built with simulated data. Finally, the road is reconstructed with the elements detected in each image. Some results of the road segmentation and pattern recognition in difficult situations are shown. We introduce an interpolation method to deal with the absence of marking due to. for instance, dirty marking, occlusion by other vehicles or poor visibility.

Image segmentation and pattern recognition for road marking analysis

A method for detecting raindrops on a windscreen, which uses a camera, which is fitted inside the vehicle opposite the windscreen and which is focused at infinite. The method comprises steps of acquiring images by a camera through the windscreen, camera preferably focused at infinite, retrieving the contours of the spots present on the acquired images, and realizing the histogram of the widths of contours of the spots.

Method of detecting rain on a windscreen

A method for adjusting an orientation of a camera installed on a vehicle, comprising the following operations: installing the vehicle at a location provided with markings on the ground, detecting these markings, estimating a vanishing point based on these markings, and matching the vanishing point with a positioning point by angular displacement of the camera.

Julien Rebut

Papers

CARRADA Dataset: Camera and Automotive Radar with Range-Angle-Doppler Annotations

How much real data do we actually need: Analyzing object detection performance using synthetic and real data

Image segmentation and pattern recognition for road marking analysis

Method of detecting rain on a windscreen

Method for adjusting the orientation of a camera installed in a vehicle and system for carrying out this method