We propose a deep convolutional neural network architecture codenamed Inception that achieves the new state of the art for classification and detection in the ImageNet Large-Scale Visual Recognition Challenge 2014 (ILSVRC14). The main hallmark of this architecture is the improved utilization of the computing resources inside the network. By a carefully crafted design, we increased the depth and width of the network while keeping the computational budget constant. To optimize quality, the architectural decisions were based on the Hebbian principle and the intuition of multi-scale processing. One particular incarnation used in our submission for ILSVRC14 is called GoogLeNet, a 22 layers deep network, the quality of which is assessed in the context of classification and detection.

/pdf/going-deeper-with-convolutions-1yobw2o2ds.pdf

Going deeper with convolutions

Today, visual recognition systems are still rarely employed in robotics applications. Perhaps one of the main reasons for this is the lack of demanding benchmarks that mimic such scenarios. In this paper, we take advantage of our autonomous driving platform to develop novel challenging benchmarks for the tasks of stereo, optical flow, visual odometry/SLAM and 3D object detection. Our recording platform is equipped with four high resolution video cameras, a Velodyne laser scanner and a state-of-the-art localization system. Our benchmarks comprise 389 stereo and optical flow image pairs, stereo visual odometry sequences of 39.2 km length, and more than 200k 3D object annotations captured in cluttered scenarios (up to 15 cars and 30 pedestrians are visible per image). Results from state-of-the-art algorithms reveal that methods ranking high on established datasets such as Middlebury perform below average when being moved outside the laboratory to the real world. Our goal is to reduce this bias by providing challenging benchmarks with novel difficulties to the computer vision community. Our benchmarks are available online at: www.cvlibs.net/datasets/kitti

/pdf/are-we-ready-for-autonomous-driving-the-kitti-vision-zygefho352.pdf

Are we ready for autonomous driving? The KITTI vision benchmark suite

Convolutional networks are powerful visual models that yield hierarchies of features. We show that convolutional networks by themselves, trained end-to-end, pixels-to-pixels, exceed the state-of-the-art in semantic segmentation. Our key insight is to build "fully convolutional" networks that take input of arbitrary size and produce correspondingly-sized output with efficient inference and learning. We define and detail the space of fully convolutional networks, explain their application to spatially dense prediction tasks, and draw connections to prior models. We adapt contemporary classification networks (AlexNet, the VGG net, and GoogLeNet) into fully convolutional networks and transfer their learned representations by fine-tuning to the segmentation task. We then define a novel architecture that combines semantic information from a deep, coarse layer with appearance information from a shallow, fine layer to produce accurate and detailed segmentations. Our fully convolutional network achieves state-of-the-art segmentation of PASCAL VOC (20% relative improvement to 62.2% mean IU on 2012), NYUDv2, and SIFT Flow, while inference takes one third of a second for a typical image.

Fully Convolutional Networks for Semantic Segmentation

Point cloud is an important type of geometric data structure. Due to its irregular format, most researchers transform such data to regular 3D voxel grids or collections of images. This, however, renders data unnecessarily voluminous and causes issues. In this paper, we design a novel type of neural network that directly consumes point clouds, which well respects the permutation invariance of points in the input. Our network, named PointNet, provides a unified architecture for applications ranging from object classification, part segmentation, to scene semantic parsing. Though simple, PointNet is highly efficient and effective. Empirically, it shows strong performance on par or even better than state of the art. Theoretically, we provide analysis towards understanding of what the network has learnt and why the network is robust with respect to input perturbation and corruption.

/pdf/pointnet-deep-learning-on-point-sets-for-3d-classification-1zge9sjodi.pdf

PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

A survey on deep learning in medical image analysis

Diffusion MRI provides unique information on the microarchitecture of biological tissues. One of the major challenges is finding a balance between image resolution, acquisition duration, noise level and image artifacts. Recent methods tackle this challenge by performing super-resolution reconstruction in image space or in diffusion space, regularization of the image data or of postprocessed data (such as the orientation distribution function, ODF) along different dimensions, and/or impose data-consistency in the original acquisition space. Each of these techniques has its own advantages; however, it is rare that even a few of them are combined. Here we present a holistic framework for diffusion MRI reconstruction that allows combining the advantages of all these techniques in a single reconstruction step. In proof-of-concept experiments, we demonstrate super-resolution on HARDI shells and in image space, regularization of the ODF and of the images in spatial and angular dimensions, and data consistency in the original acquisition space. Reconstruction quality is superior to standard reconstruction, demonstrating the feasibility of combining advanced techniques into one step.

/pdf/holistic-image-reconstruction-for-diffusion-mri-2bks38ay0n.pdf

Holistic Image Reconstruction for Diffusion MRI

The goal of image segmentation is to partition the image plane into a set of meaningful regions. While generic low-level segmentation algorithms often impose a prior which favors shorter boundaries, for segmenting familiar structures in images it may be advantageous to impose a more object-specific shape prior. Over the years, researchers have proposed different algorithms to impose prior shape knowledge based on either explicit or implicit representations of shape. In the following, I will provide a brief review of several approaches.

Shape Priors for Image Segmentation

In this work we show how sublabel-accurate multilabeling approaches can be derived by approximating a classical label-continuous convex relaxation of nonconvex free-discontinuity problems. This insight allows to extend these sublabel-accurate approaches from total variation to general convex and nonconvex regularizations. Furthermore, it leads to a systematic approach to the discretization of continuous convex relaxations. We study the relationship to existing discretizations and to discrete-continuous MRFs. Finally, we apply the proposed approach to obtain a sublabel-accurate and convex solution to the vectorial Mumford-Shah functional and show in several experiments that it leads to more precise solutions using fewer labels.

Sublabel-Accurate Discretization of Nonconvex Free-Discontinuity Problems

We propose a method to impose homogeneous linear inequality constraints of the form $Ax\leq 0$ on neural network activations. The proposed method allows a data-driven training approach to be combined with modeling prior knowledge about the task. One way to achieve this task is by means of a projection step at test time after unconstrained training. However, this is an expensive operation. By directly incorporating the constraints into the architecture, we can significantly speed-up inference at test time; for instance, our experiments show a speed-up of up to two orders of magnitude over a projection method. Our algorithm computes a suitable parameterization of the feasible set at initialization and uses standard variants of stochastic gradient descent to find solutions to the constrained network. Thus, the modeling constraints are always satisfied during training. Crucially, our approach avoids to solve an optimization problem at each training step or to manually trade-off data and constraint fidelity with additional hyperparameters. We consider constrained generative modeling as an important application domain and experimentally demonstrate the proposed method by constraining a variational autoencoder.

Linear Inequality Constraints for Neural Network Activations.

A feature descriptor or shape signature based on a solution to a Schrodinger differential equation allows to determine correspondences between graph structures and to analyse the properties of individual nodes in a graph with a high level of discriminativeness. In particular, the feature descriptor according to the present invention allows to clearly separate influences of different spatial scales and allows to treat all scales equally. An appropriate parameterization of the feature descriptor may be achieved through a theoretical stability analysis. The feature descriptor according the present invention may be employed for a shape analysis, shape segmentation, and shape retrieval, or whenever the properties of individual nodes in a graph need to be analysed to characterize the relation to the remainder of the graph, such as in the analysis of computer networks.

Daniel Cremers

Papers

Holistic Image Reconstruction for Diffusion MRI

Shape Priors for Image Segmentation

Sublabel-Accurate Discretization of Nonconvex Free-Discontinuity Problems

Linear Inequality Constraints for Neural Network Activations.

Wave kernel signature for feature description