Today, visual recognition systems are still rarely employed in robotics applications. Perhaps one of the main reasons for this is the lack of demanding benchmarks that mimic such scenarios. In this paper, we take advantage of our autonomous driving platform to develop novel challenging benchmarks for the tasks of stereo, optical flow, visual odometry/SLAM and 3D object detection. Our recording platform is equipped with four high resolution video cameras, a Velodyne laser scanner and a state-of-the-art localization system. Our benchmarks comprise 389 stereo and optical flow image pairs, stereo visual odometry sequences of 39.2 km length, and more than 200k 3D object annotations captured in cluttered scenarios (up to 15 cars and 30 pedestrians are visible per image). Results from state-of-the-art algorithms reveal that methods ranking high on established datasets such as Middlebury perform below average when being moved outside the laboratory to the real world. Our goal is to reduce this bias by providing challenging benchmarks with novel difficulties to the computer vision community. Our benchmarks are available online at: www.cvlibs.net/datasets/kitti

/pdf/are-we-ready-for-autonomous-driving-the-kitti-vision-zygefho352.pdf

Are we ready for autonomous driving? The KITTI vision benchmark suite

We investigate architectures of discriminatively trained deep Convolutional Networks (ConvNets) for action recognition in video. The challenge is to capture the complementary information on appearance from still frames and motion between frames. We also aim to generalise the best performing hand-crafted features within a data-driven learning framework.

Our contribution is three-fold. First, we propose a two-stream ConvNet architecture which incorporates spatial and temporal networks. Second, we demonstrate that a ConvNet trained on multi-frame dense optical flow is able to achieve very good performance in spite of limited training data. Finally, we show that multitask learning, applied to two different action classification datasets, can be used to increase the amount of training data and improve the performance on both. Our architecture is trained and evaluated on the standard video actions benchmarks of UCF-101 and HMDB-51, where it is competitive with the state of the art. It also exceeds by a large margin previous attempts to use deep nets for video classification.

https://proceedings.neurips.cc/paper/2014/file/00ec53c4682d36f5c4359f4ae7bd7ba1-Paper.pdf

Two-Stream Convolutional Networks for Action Recognition in Videos

The goal of precipitation nowcasting is to predict the future rainfall intensity in a local region over a relatively short period of time. Very few previous studies have examined this crucial and challenging weather forecasting problem from the machine learning perspective. In this paper, we formulate precipitation nowcasting as a spatiotemporal sequence forecasting problem in which both the input and the prediction target are spatiotemporal sequences. By extending the fully connected LSTM (FC-LSTM) to have convolutional structures in both the input-to-state and state-to-state transitions, we propose the convolutional LSTM (ConvLSTM) and use it to build an end-to-end trainable model for the precipitation nowcasting problem. Experiments show that our ConvLSTM network captures spatiotemporal correlations better and consistently outperforms FC-LSTM and the state-of-the-art operational ROVER algorithm for precipitation nowcasting.

Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting

Models based on deep convolutional networks have dominated recent image interpretation tasks; we investigate whether models which are also recurrent, or “temporally deep”, are effective for tasks involving sequences, visual and otherwise. We develop a novel recurrent convolutional architecture suitable for large-scale visual learning which is end-to-end trainable, and demonstrate the value of these models on benchmark video recognition tasks, image description and retrieval problems, and video narration challenges. In contrast to current models which assume a fixed spatio-temporal receptive field or simple temporal averaging for sequential processing, recurrent convolutional models are “doubly deep” in that they can be compositional in spatial and temporal “layers”. Such models may have advantages when target concepts are complex and/or training data are limited. Learning long-term dependencies is possible when nonlinearities are incorporated into the network state updates. Long-term RNN models are appealing in that they directly can map variable-length inputs (e.g., video frames) to variable length outputs (e.g., natural language text) and can model complex temporal dynamics; yet they can be optimized with backpropagation. Our recurrent long-term models are directly connected to modern visual convnet models and can be jointly trained to simultaneously learn temporal dynamics and convolutional perceptual representations. Our results show such models have distinct advantages over state-of-the-art models for recognition or generation which are separately defined and/or optimized.

/pdf/long-term-recurrent-convolutional-networks-for-visual-3znyib4gwu.pdf

Long-term recurrent convolutional networks for visual recognition and description

Humans perceive the three-dimensional structure of the world with apparent ease. However, despite all of the recent advances in computer vision research, the dream of having a computer interpret an image at the same level as a two-year old remains elusive. Why is computer vision such a challenging problem and what is the current state of the art? Computer Vision: Algorithms and Applications explores the variety of techniques commonly used to analyze and interpret images. It also describes challenging real-world applications where vision is being successfully used, both for specialized applications such as medical imaging, and for fun, consumer-level tasks such as image editing and stitching, which students can apply to their own personal photos and videos. More than just a source of recipes, this exceptionally authoritative and comprehensive textbook/reference also takes a scientific approach to basic vision problems, formulating physical models of the imaging process before inverting them to produce descriptions of a scene. These problems are also analyzed using statistical models and solved using rigorous engineering techniques Topics and features: structured to support active curricula and project-oriented courses, with tips in the Introduction for using the book in a variety of customized courses; presents exercises at the end of each chapter with a heavy emphasis on testing algorithms and containing numerous suggestions for small mid-term projects; provides additional material and more detailed mathematical topics in the Appendices, which cover linear algebra, numerical techniques, and Bayesian estimation theory; suggests additional reading at the end of each chapter, including the latest research in each sub-field, in addition to a full Bibliography at the end of the book; supplies supplementary course material for students at the associated website, http://szeliski.org/Book/. Suitable for an upper-level undergraduate or graduate-level course in computer science or engineering, this textbook focuses on basic techniques that work under real-world conditions and encourages students to push their creative boundaries. Its design and exposition also make it eminently suitable as a unique reference to the fundamental techniques and current research literature in computer vision.

/pdf/computer-vision-algorithms-and-applications-25dn6wu83j.pdf

Computer Vision: Algorithms and Applications

Die Scḧatzung von Bewegungsinformation aus Bildfolgen ist eines der zentralen Probleme des Maschinensehens. In diesem Zusammenhang spielen Variationsans̈atze eine große Rolle, die eine vollst ändige und genaue Bestimmung des Verschiebungsfelds zwischen aufeinander folgenden Bildern einer Bildfolge erm öglichen. In der hier zusammengefassten Dissertation [Bru06] werden in zweierlei Hinsicht wichtige Beitr̈age auf dem Gebiet der Bewegungsanalyse geleistet: (i) Zum einen wird ein systematischer Rahmen zur Entwicklung hochgenauer Variationsans ätze vorgestellt. Dieses erlaubt den Entwurf der zur Zeit pr äzisesten Scḧ atzverfahren in der gesamten Literatur. (ii) Zum anderen wird ein allgemeiner Ansatz zur Konstruktion hochperformanter Mehrgitteralgorithmen pr äsentiert. Dieser erm̈ oglicht die Berechnung der Ergebnisse in Echtzeit und macht damit erstmals diese äußerst pr̈ azisen Variationsans̈atze praktisch einsetzbar. Experimente zur Evaluation der Qualit ät und der Effizienz belegen diëUberlegenheit der entwickelten Sch ätzverfahren.

Variationsansätze zur Bewegungsschätzung: Präzise Modellierung und Effiziente Numerik.

This report documents the program and the results of Dagstuhl Seminar 11471 
Efficient Algorithms for Global Optimisation Methods in Computer Vision,
taking place November 20-25 in 2011. The focus of the seminar was to discuss
the design of efficient computer vision algorithms based on global optimisation 
methods in the context of the entire design pipeline. Since there is no
such conference that deals with all aspects of the design process -- modelling, 
mathematical analysis, numerical solvers, and parallelisation -- the seminar aimed at bringing together researchers from computer science and mathematics covering all four fields.

Efficient Algorithms for Global Optimisation Methods in Computer Vision (Dagstuhl Seminar 11471).

The depiction of motion in static representations has a long tradition in art and science alike. Often, motion is depicted by spatio-temporal summarizations that try to preserve as much information of the original dynamic content as possible. In our approach to depicting motion, we remove the spatial constraints and generate new content steered by the temporal changes in motion. Applying particle steering in combination with the dynamic color palette of the video content, we can create a wide range of different image styles. With recorded videos, or by live interaction with a webcam, one can influence the resulting image. We provide a set of intuitive parameters to affect the style of the result, the final image content depends on the video input. Based on a collection of results gathered from test users, we discuss example styles that can be achieved with FlowBrush. In general, our approach provides an open sandbox for creative people to generate aesthetic images from any video content they apply.

FlowBrush: optical flow art

This paper describes different low level parallelization strategies of a nonlinear diffusion filtering algorithm for digital image denoising. The nonlinear diffusion method uses a so-called additive operator splitting (AOS) scheme. This algorithm is very efficient, but requires frequent data exchanges. Our focus was to provide different data decomposition techniques which allow for achieving high efficiency for different hardware platforms. Depending on the available communication performance, our parallelization schemes allow for high scalability when using fast System Area Networks (SAN), but also provide significant performance enhancements on slower interconnects by optimizing data structures and communication patterns. Performance results are presented for a variety of commodity hardware platforms. Our most important result is a speedup factor of 210 using 256 processors of a high end cluster equipped with Myrinet.

/pdf/low-level-parallelization-of-nonlinear-diffusion-filtering-4xc8y6nlny.pdf

Low Level Parallelization of Nonlinear Diffusion Filtering Algorithms for Cluster Computing Environments

The integration of feature matches for handling large displacements is one of the key concepts of recent variational optical flow methods. In this context, many existing approaches rely on confidence measures to identify locations where a poor initial match can potentially be improved by adaptively integrating flow proposals. One very intuitive confidence measure to identify such locations is the matching cost of the data term. Problems arise, however, in the presence of illumination changes, since brightness constancy does not hold and invariant constancy assumptions typically discard too much information for an identification of poor matches. In this paper, we suggest a pipeline approach that addresses the aforementioned problem in two ways. First, we propose a novel confidence measure based on the illumination-compensated brightness constancy assumption. By estimating illumination changes from a pre-computed flow this measure allows us to reliably identify poor matches even in the presence of varying illumination. Secondly, in contrast to many existing pipeline approaches, we propose to integrate only feature matches that have been obtained from dense variational methods. This in turn not only provides robust matches due to the inherent regularization, it also demonstrates that in many cases sparse descriptor matches are not needed for large displacement optical flow. Experiments on the Sintel benchmark and on common large displacement sequences demonstrate the benefits of our strategy. They show a clear improvement over the baseline method and a comparable performance as similar methods from the literature based on sparse feature matches.

Andrés Bruhn

Papers

Variationsansätze zur Bewegungsschätzung: Präzise Modellierung und Effiziente Numerik.

Efficient Algorithms for Global Optimisation Methods in Computer Vision (Dagstuhl Seminar 11471).

FlowBrush: optical flow art

Low Level Parallelization of Nonlinear Diffusion Filtering Algorithms for Cluster Computing Environments

Illumination-Aware Large Displacement Optical Flow