There is, I think, something ethereal about i —the square root of minus one. I remember first hearing about it at school. It seemed an odd beast at that time—an intruder hovering on the edge of reality.

Usually familiarity dulls this sense of the bizarre, but in the case of i it was the reverse: over the years the sense of its surreal nature intensified. It seemed that it was impossible to write mathematics that described the real world in …

I and i

Today, visual recognition systems are still rarely employed in robotics applications. Perhaps one of the main reasons for this is the lack of demanding benchmarks that mimic such scenarios. In this paper, we take advantage of our autonomous driving platform to develop novel challenging benchmarks for the tasks of stereo, optical flow, visual odometry/SLAM and 3D object detection. Our recording platform is equipped with four high resolution video cameras, a Velodyne laser scanner and a state-of-the-art localization system. Our benchmarks comprise 389 stereo and optical flow image pairs, stereo visual odometry sequences of 39.2 km length, and more than 200k 3D object annotations captured in cluttered scenarios (up to 15 cars and 30 pedestrians are visible per image). Results from state-of-the-art algorithms reveal that methods ranking high on established datasets such as Middlebury perform below average when being moved outside the laboratory to the real world. Our goal is to reduce this bias by providing challenging benchmarks with novel difficulties to the computer vision community. Our benchmarks are available online at: www.cvlibs.net/datasets/kitti

/pdf/are-we-ready-for-autonomous-driving-the-kitti-vision-zygefho352.pdf

Are we ready for autonomous driving? The KITTI vision benchmark suite

Probability Distributions.- Linear Models for Regression.- Linear Models for Classification.- Neural Networks.- Kernel Methods.- Sparse Kernel Machines.- Graphical Models.- Mixture Models and EM.- Approximate Inference.- Sampling Methods.- Continuous Latent Variables.- Sequential Data.- Combining Models.

Pattern Recognition and Machine Learning

Visual understanding of complex urban street scenes is an enabling factor for a wide range of applications. Object detection has benefited enormously from large-scale datasets, especially in the context of deep learning. For semantic urban scene understanding, however, no current dataset adequately captures the complexity of real-world urban scenes. To address this, we introduce Cityscapes, a benchmark suite and large-scale dataset to train and test approaches for pixel-level and instance-level semantic labeling. Cityscapes is comprised of a large, diverse set of stereo video sequences recorded in streets from 50 different cities. 5000 of these images have high quality pixel-level annotations, 20 000 additional images have coarse annotations to enable methods that leverage large volumes of weakly-labeled data. Crucially, our effort exceeds previous attempts in terms of dataset size, annotation richness, scene variability, and complexity. Our accompanying empirical study provides an in-depth analysis of the dataset characteristics, as well as a performance evaluation of several state-of-the-art approaches based on our benchmark.

https://www.cityscapes-dataset.com/wordpress/wp-content/papercite-data/pdf/cordts2016cityscapes.pdf

The Cityscapes Dataset for Semantic Urban Scene Understanding

We present a novel dataset captured from a VW station wagon for use in mobile robotics and autonomous driving research. In total, we recorded 6 hours of traffic scenarios at 10-100 Hz using a variety of sensor modalities such as high-resolution color and grayscale stereo cameras, a Velodyne 3D laser scanner and a high-precision GPS/IMU inertial navigation system. The scenarios are diverse, capturing real-world traffic situations, and range from freeways over rural areas to inner-city scenes with many static and dynamic objects. Our data is calibrated, synchronized and timestamped, and we provide the rectified and raw image sequences. Our dataset also contains object labels in the form of 3D tracklets, and we provide online benchmarks for stereo, optical flow, object detection and other tasks. This paper describes our recording platform, the data format and the utilities that we provide.

/pdf/vision-meets-robotics-the-kitti-dataset-2h3kd0lpgn.pdf

Vision meets robotics: The KITTI dataset

We present a method for improving segmentation tasks on images affected by adherent rain drops and streaks. We introduce a novel stereo dataset recorded using a system that allows one lens to be affected by real water droplets while keeping the other lens clear. We train a denoising generator using this dataset and show that it is effective at removing the effect of real water droplets, in the context of image reconstruction and road marking segmentation. To further test our de-noising approach, we describe a method of adding computer-generated adherent water droplets and streaks to any images, and use this technique as a proxy to demonstrate the effectiveness of our model in the context of general semantic segmentation. We benchmark our results using the CamVid road marking segmentation dataset, Cityscapes semantic segmentation datasets and our own real-rain dataset, and show significant improvement on all tasks.

I Can See Clearly Now : Image Restoration via De-Raining.

In this paper we propose the hybrid use of illuminant invariant and RGB images to perform image classification of urban scenes despite challenging variation in lighting conditions. Coping with lighting change (and the shadows thereby invoked) is a non-negotiable requirement for long term autonomy using vision. One aspect of this is the ability to reliably classify scene components in the presence of marked and often sudden changes in lighting. This is the focus of this paper. Posed with the task of classifying all parts in a scene from a full colour image, we propose that lighting invariant transforms can reduce the variability of the scene, resulting in a more reliable classification. We leverage the ideas of “data transfer” for classification, beginning with full colour images for obtaining candidate scene-level matches using global image descriptors. This is commonly followed by superpixellevel matching with local features. However, we show that if the RGB images are subjected to an illuminant invariant transform before computing the superpixel-level features, classification is significantly more robust to scene illumination effects. The approach is evaluated using three datasets. The first being our own dataset and the second being the KITTI dataset using manually generated ground truth for quantitative analysis. We qualitatively evaluate the method on a third custom dataset over a 750m trajectory.

https://www.robots.ox.ac.uk/~mobile/Papers/2014ICRA_upcroft.pdf

Lighting invariant urban street classification

This paper describes the design, build, automatic self-calibration and evaluation of a 3D Laser sensor using conventional parts. Our goal is to design a system which is an order of magnitude cheaper than commercial systems, with commensurate performance. In this paper we adopt point cloud ‘crispness’ as the measure of system performance that we wish to optimise. Concretely, we apply the information theoretic measure known as Renyi Quadratic Entropy to capture the degree of organisation of a point cloud. By expressing this quantity as a function of key unknown system parameters, we are able to deduce a full calibration of the sensor via an online optimisation. Beyond details on the sensor design itself, we fully describe the end-to-end extrinsic parameter calibration process, the estimation of the clock skews between the four constituent microprocessors and analyse the effect our spatial and temporal calibrations have on point cloud quality.

/pdf/automatic-self-calibration-of-a-full-field-of-view-3d-n-2p1bwthmtg.pdf

Automatic Self-calibration of a Full Field-of-View 3D n-Laser Scanner

Robust and reliable visual localisation at any time of day is an essential component towards low-cost autonomy for road vehicles. We present a method to perform online 6-DoF visual localisation across a wide range of outdoor illumination conditions throughout the day and night using a 3D scene prior collected by a survey vehicle. We propose the use of a one-dimensional illumination invariant colour space which stems from modelling the spectral properties of the camera and scene illumination in conjunction. We combine our previous work on Localisation with Appearance of Prior Structure (LAPS) with this illumination invariant colour space to demonstrate a marked improvement in our ability to localise throughout the day compared to using a conventional RGB colour space. Our ultimate goal is robust and reliable any-time localisation - an attractive proposition for low-cost autonomy for road vehicles. Accordingly, we demonstrate our technique using 32km of data collected over a full 24-hour period from a road vehicle.

LAPS-II: 6-DoF day and night visual localisation with prior 3D structure for autonomous road vehicles

A method and system for determining extrinsic calibration parameters for at least one pair of sensing devices mounted on transportable apparatus obtains (202) image data (110)representing images captured by an image generating sensing device (102) of the pair at a series of poses during motion through an environment (120) by transportable apparatus (100). The method obtains (202) data (112) representing a 3D point cloud based on data produced by a 2D LIDAR sensing device (104) of the pair. The method selects (204) an image captured by the image at a particular pose. The method generates (210) a laser reflectance image based on a portion of the point cloud corresponding to the pose. The method computes (212) a metric measuring alignment between the selected image and the corresponding laser reflectance image and uses (214) the metric to determine extrinsic calibration parameters for at least one of the sensing devices.

Paul Newman

Papers

I Can See Clearly Now : Image Restoration via De-Raining.

Lighting invariant urban street classification

Automatic Self-calibration of a Full Field-of-View 3D n-Laser Scanner

LAPS-II: 6-DoF day and night visual localisation with prior 3D structure for autonomous road vehicles

Extrinsic calibration of imaging sensing devices and 2d lidars mounted on transportable apparatus