There is, I think, something ethereal about i —the square root of minus one. I remember first hearing about it at school. It seemed an odd beast at that time—an intruder hovering on the edge of reality.

Usually familiarity dulls this sense of the bizarre, but in the case of i it was the reverse: over the years the sense of its surreal nature intensified. It seemed that it was impossible to write mathematics that described the real world in …

I and i

Today, visual recognition systems are still rarely employed in robotics applications. Perhaps one of the main reasons for this is the lack of demanding benchmarks that mimic such scenarios. In this paper, we take advantage of our autonomous driving platform to develop novel challenging benchmarks for the tasks of stereo, optical flow, visual odometry/SLAM and 3D object detection. Our recording platform is equipped with four high resolution video cameras, a Velodyne laser scanner and a state-of-the-art localization system. Our benchmarks comprise 389 stereo and optical flow image pairs, stereo visual odometry sequences of 39.2 km length, and more than 200k 3D object annotations captured in cluttered scenarios (up to 15 cars and 30 pedestrians are visible per image). Results from state-of-the-art algorithms reveal that methods ranking high on established datasets such as Middlebury perform below average when being moved outside the laboratory to the real world. Our goal is to reduce this bias by providing challenging benchmarks with novel difficulties to the computer vision community. Our benchmarks are available online at: www.cvlibs.net/datasets/kitti

/pdf/are-we-ready-for-autonomous-driving-the-kitti-vision-zygefho352.pdf

Are we ready for autonomous driving? The KITTI vision benchmark suite

Probability Distributions.- Linear Models for Regression.- Linear Models for Classification.- Neural Networks.- Kernel Methods.- Sparse Kernel Machines.- Graphical Models.- Mixture Models and EM.- Approximate Inference.- Sampling Methods.- Continuous Latent Variables.- Sequential Data.- Combining Models.

Pattern Recognition and Machine Learning

Visual understanding of complex urban street scenes is an enabling factor for a wide range of applications. Object detection has benefited enormously from large-scale datasets, especially in the context of deep learning. For semantic urban scene understanding, however, no current dataset adequately captures the complexity of real-world urban scenes. To address this, we introduce Cityscapes, a benchmark suite and large-scale dataset to train and test approaches for pixel-level and instance-level semantic labeling. Cityscapes is comprised of a large, diverse set of stereo video sequences recorded in streets from 50 different cities. 5000 of these images have high quality pixel-level annotations, 20 000 additional images have coarse annotations to enable methods that leverage large volumes of weakly-labeled data. Crucially, our effort exceeds previous attempts in terms of dataset size, annotation richness, scene variability, and complexity. Our accompanying empirical study provides an in-depth analysis of the dataset characteristics, as well as a performance evaluation of several state-of-the-art approaches based on our benchmark.

https://www.cityscapes-dataset.com/wordpress/wp-content/papercite-data/pdf/cordts2016cityscapes.pdf

The Cityscapes Dataset for Semantic Urban Scene Understanding

We present a novel dataset captured from a VW station wagon for use in mobile robotics and autonomous driving research. In total, we recorded 6 hours of traffic scenarios at 10-100 Hz using a variety of sensor modalities such as high-resolution color and grayscale stereo cameras, a Velodyne 3D laser scanner and a high-precision GPS/IMU inertial navigation system. The scenarios are diverse, capturing real-world traffic situations, and range from freeways over rural areas to inner-city scenes with many static and dynamic objects. Our data is calibrated, synchronized and timestamped, and we provide the rectified and raw image sequences. Our dataset also contains object labels in the form of 3D tracklets, and we provide online benchmarks for stereo, optical flow, object detection and other tasks. This paper describes our recording platform, the data format and the utilities that we provide.

/pdf/vision-meets-robotics-the-kitti-dataset-2h3kd0lpgn.pdf

Vision meets robotics: The KITTI dataset

This paper is concerned with "loop closing" for mobile robots. Loop closing is the problem of correctly asserting that a robot has returned to a previously visited area. It is a particularly hard but important component of the Simultaneous Localization and Mapping (SLAM) problem. Here a mobile robot explores an a-priori unknown environment performing on-the-fly mapping while the map is used to localize the vehicle. Many SLAM implementations look to internal map and vehicle estimates (p.d.fs) to make decisions about whether a vehicle is revisiting a previously mapped area or is exploring a new region of workspace. We suggest that one of the reasons loop closing is hard in SLAM is precisely because these internal estimates can, despite best efforts, be in gross error. The "loop closer" we propose, analyze and demonstrate makes no recourse to the metric estimates of the SLAM system it supports and aids---it is entirely independent. At regular intervals the vehicle captures the appearance of the local scene (with camera and laser). We encode the similarity between all possible pairings of scenes in a "similarity matrix". We then pose the loop closing problem as the task of extracting statistically significant sequences of similar scenes from this matrix. We show how suitable analysis (introspection) and decomposition (remediation) of the similarity matrix allows for the reliable detection of loops despite the presence of repetitive and visually ambiguous scenes. We demonstrate the technique supporting a SLAM system driven by scan-matching laser data in a variety of settings. Some of the outdoor settings are beyond the capability of the SLAM system itself in which case GPS was used to provide a ground truth. We further show how the techniques can equally be applied to detect loop closure using spatial images taken with a scanning laser. We conclude with an extension of the loop closing technique to a multi-robot mapping problem in which the outputs of several, uncoordinated and SLAM-enabled robots are fused without requiring inter-vehicle observations or a-priori frame alignment.

/pdf/detecting-loop-closure-with-scene-sequences-394ppkixls.pdf

Detecting Loop Closure with Scene Sequences

Large scale exploration of the environment requires a constant time estimation engine. Bundle adjustment or pose relaxation do not fulfil these requirements as the number of parameters to solve grows with the size of the environment. We describe a relative simultaneous localisation and mapping system (RSLAM) for the constant-time estimation of structure and motion using a binocular stereo camera system as the sole sensor. Achieving robustness in the presence of difficult and changing lighting conditions and rapid motion requires careful engineering of the visual processing, and we describe a number of innovations which we show lead to high accuracy and robustness. In order to achieve real-time performance without placing severe limits on the size of the map that can be built, we use a topo-metric representation in terms of a sequence of relative locations. When combined with fast and reliable loop-closing, we mitigate the drift to obtain highly accurate global position estimates without any global minimisation. We discuss some of the issues that arise from using a relative representation, and evaluate our system on long sequences processed at a constant 30---45 Hz, obtaining precisions down to a few meters over distances of a few kilometres.

/pdf/rslam-a-system-for-large-scale-mapping-in-constant-time-431yxixj82.pdf

RSLAM: A System for Large-Scale Mapping in Constant-Time Using Stereo

This paper is about long-term navigation in environments whose appearance changes over time, suddenly or gradually. We describe, implement and validate an approach which allows us to incrementally learn a model whose complexity varies naturally in accordance with variation of scene appearance. It allows us to leverage the state of the art in pose estimation to build over many runs, a world model of sufficient richness to allow simple localisation despite a large variation in conditions. As our robot repeatedly traverses its workspace, it accumulates distinct visual experiences that in concert, implicitly represent the scene variation: each experience captures a visual mode. When operating in a previously visited area, we continually try to localise in these previous experiences while simultaneously running an independent vision-based pose estimation system. Failure to localise in a sufficient number of prior experiences indicates an insufficient model of the workspace and instigates the laying down of the live image sequence as a new distinct experience. In this way, over time we can capture the typical time-varying appearance of an environment and the number of experiences required tends to a constant. Although we focus on vision as a primary sensor throughout, the ideas we present here are equally applicable to other sensor modalities. We demonstrate our approach working on a road vehicle operating over a 3-month period at different times of day, in different weather and lighting conditions. We present extensive results analysing different aspects of the system and approach, in total processing over 136,000 frames captured from 37 km of driving.

Experience-based navigation for long-term localisation

Paul Michael Newman Doctor of Philosophy The University of Sydney March 1999 On the Structure and Solution of the Simultaneous Localisation and Map Building Problem This thesis is concerned with the simultaneous localisation and map building (SLAM) problem. The SLAM problem asks if it is possible for an autonomous vehicle to start in an unknown location in an unknown environment and then to incrementally build a map of this environment while simultaneously using this map to compute absolute vehicle location. The map and robot location estimates obtained from a successful SLAM system provide essential information upon which high level tasks such as path planning are predicated. A practicable solution to the SLAM problem is of inestimable value in the quest to create a truly autonomous mobile robot. The thesis has three principal theoretical contributions. The first is the elucidation of the structure of the SLAM problem. This is achieved by the analysis of a conventional and well known SLAM algorithm using global coordinates called, in this thesis, the Absolute Map Filter or AMF. Using this algorithm, three convergence theorems central to the SLAM problem are proved for the first time. They prove that the uncertainty in the estimated map decreases monotonically and achieves a defined lower bound. Futhermore, in the limit as the number of landmark observations increases, the relationship between landmarks becomes perfectly known. These proofs constitute the second theoretical contribution of the thesis. The third principal theoretical contribution of this thesis is the development of a novel SLAM solution capable of solving the SLAM problem in real time. This algorithm is called the Geometric Projection Filter or GPF. Rather than estimate the location of landmarks in global coordinates it estimates the relationships between individual landmarks. The convergence properties of this algorithm are derived and compared with those of the conventional AMF algorithm. An implementation of the GPF and the AMF is provided on a custom built subsea vehicle. The performance of the two filters are compared and shown to have the properties predicted by the preceding theoretical analysis. This implementation constitutes the fourth principal contribution of the thesis. It shows that the GPF can be used as the basis of a substantive real time deployment of a mobile robot in an initially unknown environment.

/pdf/on-the-structure-and-solution-of-the-simultaneous-toahon77wl.pdf

On the Structure and Solution of the Simultaneous Localisation and Map Building Problem

This document describes the MOOS-IvP autonomy software for unmanned marine vehicles and its use in large-scale ocean sensing systems. MOOS-IvP is composed of two open-source software projects funded by the Office of Naval Research. MOOS provides a core autonomy middleware capability, and the MOOS project additionally provides a set of ubiquitous infrastructure utilities. The IvP Helm is the primary component of an additional set of capabilities implemented to form a full marine autonomy suite known as MOOS-IvP. This software and architecture are platform and mission agnostic and allow for a scalable nesting of unmanned vehicle nodes to form large-scale, long-endurance ocean sensing systems composed of heterogeneous platform types with varying degrees of communications connectivity, bandwidth, and latency. © 2010 Wiley Periodicals, Inc.

/pdf/nested-autonomy-for-unmanned-marine-vehicles-with-moos-ivp-1flofynzlw.pdf

Paul Newman

Papers

Detecting Loop Closure with Scene Sequences

RSLAM: A System for Large-Scale Mapping in Constant-Time Using Stereo

Experience-based navigation for long-term localisation

On the Structure and Solution of the Simultaneous Localisation and Map Building Problem

Nested Autonomy for Unmanned Marine Vehicles with MOOS-IvP