Today, visual recognition systems are still rarely employed in robotics applications. Perhaps one of the main reasons for this is the lack of demanding benchmarks that mimic such scenarios. In this paper, we take advantage of our autonomous driving platform to develop novel challenging benchmarks for the tasks of stereo, optical flow, visual odometry/SLAM and 3D object detection. Our recording platform is equipped with four high resolution video cameras, a Velodyne laser scanner and a state-of-the-art localization system. Our benchmarks comprise 389 stereo and optical flow image pairs, stereo visual odometry sequences of 39.2 km length, and more than 200k 3D object annotations captured in cluttered scenarios (up to 15 cars and 30 pedestrians are visible per image). Results from state-of-the-art algorithms reveal that methods ranking high on established datasets such as Middlebury perform below average when being moved outside the laboratory to the real world. Our goal is to reduce this bias by providing challenging benchmarks with novel difficulties to the computer vision community. Our benchmarks are available online at: www.cvlibs.net/datasets/kitti

/pdf/are-we-ready-for-autonomous-driving-the-kitti-vision-zygefho352.pdf

Are we ready for autonomous driving? The KITTI vision benchmark suite

https://surfer.nmr.mgh.harvard.edu/ftp/articles/dale99-recon1.pdf

Cortical surface-based analysis. I. Segmentation and surface reconstruction

Presents a review of 200 references in content-based image retrieval. The paper starts with discussing the working conditions of content-based retrieval: patterns of use, types of pictures, the role of semantics, and the sensory gap. Subsequent sections discuss computational steps for image retrieval systems. Step one of the review is image processing for retrieval sorted by color, texture, and local geometry. Features for retrieval are discussed next, sorted by: accumulative and global features, salient points, object and shape features, signs, and structural combinations thereof. Similarity of pictures and objects in pictures is reviewed for each of the feature types, in close connection to the types and means of feedback the user of the systems is capable of giving by interaction. We briefly discuss aspects of system engineering: databases, system architecture, and evaluation. In the concluding section, we present our view on: the driving force of the field, the heritage from computer vision, the influence on computer vision, the role of similarity and of interaction, the need for databases, the problem of evaluation, and the role of the semantic gap.

/pdf/content-based-image-retrieval-at-the-end-of-the-early-years-q45iy9ejai.pdf

Content-based image retrieval at the end of the early years

We investigate architectures of discriminatively trained deep Convolutional Networks (ConvNets) for action recognition in video. The challenge is to capture the complementary information on appearance from still frames and motion between frames. We also aim to generalise the best performing hand-crafted features within a data-driven learning framework.

Our contribution is three-fold. First, we propose a two-stream ConvNet architecture which incorporates spatial and temporal networks. Second, we demonstrate that a ConvNet trained on multi-frame dense optical flow is able to achieve very good performance in spite of limited training data. Finally, we show that multitask learning, applied to two different action classification datasets, can be used to increase the amount of training data and improve the performance on both. Our architecture is trained and evaluated on the standard video actions benchmarks of UCF-101 and HMDB-51, where it is competitive with the state of the art. It also exceeds by a large margin previous attempts to use deep nets for video classification.

https://proceedings.neurips.cc/paper/2014/file/00ec53c4682d36f5c4359f4ae7bd7ba1-Paper.pdf

Two-Stream Convolutional Networks for Action Recognition in Videos

/pdf/convex-analysisnoer-san-nojin-zhan-nituite-5dl2rbmoyr.pdf

Convex Analysisの二,三の進展について

We study an energy functional for computing optical flow that combines three assumptions: a brightness constancy assumption, a gradient constancy assumption, and a discontinuity-preserving spatio-temporal smoothness constraint. In order to allow for large displacements, linearisations in the two data terms are strictly avoided. We present a consistent numerical scheme based on two nested fixed point iterations. By proving that this scheme implements a coarse-to-fine warping strategy, we give a theoretical foundation for warping which has been used on a mainly experimental basis so far. Our evaluation demonstrates that the novel method gives significantly smaller angular errors than previous techniques for optical flow estimation. We show that it is fairly insensitive to parameter variations, and we demonstrate its excellent robustness under noise.

/pdf/high-accuracy-optical-flow-estimation-based-on-a-theory-for-2o1buld5hm.pdf

High Accuracy Optical Flow Estimation Based on a Theory for Warping

Preface Through many centuries physics has been one of the most fruitful sources of inspiration for mathematics. As a consequence, mathematics has become an economic language providing a few basic principles which allow to explain a large variety of physical phenomena. Many of them are described in terms of partial diierential equations (PDEs). In recent years, however, mathematics also has been stimulated by other novel elds such as image processing. Goals like image segmentation, multiscale image representation, or image restoration cause a lot of challenging mathematical questions. Nevertheless, these problems frequently have been tackled with a pool of heuristical recipes. Since the treatment of digital images requires very much computing power, these methods had to be fairly simple. With the tremendous advances in computer technology in the last decade, it has become possible to apply more sophisticated techniques such as PDE-based methods which have been inspired by physical processes. Among these techniques, parabolic PDEs have found a lot of attention for smoothing and restoration purposes, see e.g. 113]. To restore images these equations frequently arise from gradient descent methods applied to variational problems. Image smoothing by parabolic PDEs is closely related to the scale-space concept where one embeds the original image into a family of subsequently simpler , more global representations of it. This idea plays a fundamental role for extracting semantically important information. The pioneering work of Alvarez, Guichard, Lions and Morel 11] has demonstrated that all scale-spaces fulllling a few fairly natural axioms are governed by parabolic PDEs with the original image as initial condition. Within this framework, two classes can be justiied in a rigorous way as scale-spaces: the linear diiusion equation with constant dif-fusivity and nonlinear so-called morphological PDEs. All these methods satisfy a monotony axiom as smoothing requirement which states that, if one image is brighter than another, then this order is preserved during the entire scale-space evolution. An interesting class of parabolic equations which pursue both scale-space and restoration intentions is given by nonlinear diiusion lters. Methods of this type have been proposed for the rst time by Perona and Malik in 1987 190]. In v vi PREFACE order to smooth the image and to simultaneously enhance semantically important features such as edges, they apply a diiusion process whose diiusivity is steered by local image properties. These lters are diicult to analyse mathematically , as they may act locally like a backward diiusion process. …

/pdf/anisotropic-diffusion-in-image-processing-z13u09qvsm.pdf

Anisotropic diffusion in image processing

Differential methods belong to the most widely used techniques for optic flow computation in image sequences. They can be classified into local methods such as the Lucas-Kanade technique or Bigun's structure tensor method, and into global methods such as the Horn/Schunck approach and its extensions. Often local methods are more robust under noise, while global techniques yield dense flow fields. The goal of this paper is to contribute to a better understanding and the design of novel differential methods in four ways: (i) We juxtapose the role of smoothing/regularisation processes that are required in local and global differential methods for optic flow computation. (ii) This discussion motivates us to describe and evaluate a novel method that combines important advantages of local and global approaches: It yields dense flow fields that are robust against noise. (iii) Spatiotemporal and nonlinear extensions as well as multiresolution frameworks are presented for this hybrid method. (iv) We propose a simple confidence measure for optic flow methods that minimise energy functionals. It allows to sparsify a dense flow field gradually, depending on the reliability required for the resulting flow. Comparisons with experiments from the literature demonstrate the favourable performance of the proposed methods and the confidence measure.

/pdf/lucas-kanade-meets-horn-schunck-combining-local-and-global-3gjj0uw1x7.pdf

Lucas/Kanade meets Horn/Schunck: combining local and global optic flow methods

Nonlinear diffusion filtering in image processing is usually performed with explicit schemes. They are only stable for very small time steps, which leads to poor efficiency and limits their practical use. Based on a discrete nonlinear diffusion scale-space framework we present semi-implicit schemes which are stable for all time steps. These novel schemes use an additive operator splitting (AOS), which guarantees equal treatment of all coordinate axes. They can be implemented easily in arbitrary dimensions, have good rotational invariance and reveal a computational complexity and memory requirement which is linear in the number of pixels. Examples demonstrate that, under typical accuracy requirements, AOS schemes are at least ten times more efficient than the widely used explicit schemes.

/pdf/efficient-and-reliable-schemes-for-nonlinear-diffusion-4kfamil4rw.pdf

Efficient and reliable schemes for nonlinear diffusion filtering

The completion of interrupted lines or the enhancement of flow-like structures is a challenging task in computer vision, human vision, and image processing. We address this problem by presenting a multiscale method in which a nonlinear diffusion filter is steered by the so-called interest operator (second-moment matrix, structure tensor). An m-dimensional formulation of this method is analysed with respect to its well-posedness and scale-space properties. An efficient scheme is presented which uses a stabilization by a semi-implicit additive operator splitting (AOS), and the scale-space behaviour of this method is illustrated by applying it to both 2-D and 3-D images.

Joachim Weickert

Papers

High Accuracy Optical Flow Estimation Based on a Theory for Warping

Anisotropic diffusion in image processing

Lucas/Kanade meets Horn/Schunck: combining local and global optic flow methods

Efficient and reliable schemes for nonlinear diffusion filtering

Coherence-Enhancing Diffusion Filtering