Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers—8× deeper than VGG nets [40] but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions1, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.

/pdf/deep-residual-learning-for-image-recognition-b75jg61qrq.pdf

Deep Residual Learning for Image Recognition

In this work we investigate the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting. Our main contribution is a thorough evaluation of networks of increasing depth using an architecture with very small (3x3) convolution filters, which shows that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 weight layers. These findings were the basis of our ImageNet Challenge 2014 submission, where our team secured the first and the second places in the localisation and classification tracks respectively. We also show that our representations generalise well to other datasets, where they achieve state-of-the-art results. We have made our two best-performing ConvNet models publicly available to facilitate further research on the use of deep visual representations in computer vision.

Very Deep Convolutional Networks for Large-Scale Image Recognition

Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers---8x deeper than VGG nets but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. 
The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.

We propose a deep convolutional neural network architecture codenamed Inception that achieves the new state of the art for classification and detection in the ImageNet Large-Scale Visual Recognition Challenge 2014 (ILSVRC14). The main hallmark of this architecture is the improved utilization of the computing resources inside the network. By a carefully crafted design, we increased the depth and width of the network while keeping the computational budget constant. To optimize quality, the architectural decisions were based on the Hebbian principle and the intuition of multi-scale processing. One particular incarnation used in our submission for ILSVRC14 is called GoogLeNet, a 22 layers deep network, the quality of which is assessed in the context of classification and detection.

/pdf/going-deeper-with-convolutions-1yobw2o2ds.pdf

Going deeper with convolutions

In this paper, we address the problem of multi-person tracking in busy pedestrian
 zones, using a stereo rig mounted on a mobile platform. The
 complexity of the problem calls for an integrated solution, which
 extracts as much visual information as possible and combines it
 through cognitive feedback. We propose such an approach, which
 jointly estimates camera position, stereo depth, object detection,
 and tracking. We model the interplay between these components using a graphical model. Since the model has to
 incorporate object-object interactions, and temporal links to past
 frames, direct inference is intractable. We therefore propose a two-stage
 procedure: for each frame we first solve a simplified version of the
 model (disregarding interactions and temporal continuity) to
 estimate the scene geometry and an overcomplete set of object
 detections. Conditioned on these results, we then address object
 interactions, tracking, and prediction in a second step. The
 approach is experimentally evaluated on several long and difficult
 video sequences from busy inner-city locations. Our results show
 that the proposed integration makes it possible to deliver stable
 tracking performance in scenes of realistic complexity.

/pdf/robust-multi-person-tracking-from-moving-platforms-qjfn2rmsc7.pdf

Robust Multi-Person Tracking from Moving Platforms

In this paper we propose a new vision based method for robot localization using an omnidirectional camera. The method has three steps efficiently combined to deal with big reference image sets, each step evaluates less images than the previous but is more complex and accurate. Given the current uncalibrated image seen by the robot, the hierarchical algorithm gives the possibility of obtaining appearance-based (topological) and metric localization. Compared to other similar vision-based localization methods, the one proposed here has the advantage that it gets accurate metric localization based on a minimal reference image set, using the 1D three view geometry. Moreover, thanks to the linear wide baseline features used, the method is insensitive to illumination changes and occlusions, while keeping the computational load small. The simplicity of the radial line feature used speeds up the process while it keeps acceptable accuracy. We show experiments with two omnidirectional image data-sets to evaluate performance of the method.

https://webdiis.unizar.es/~csagues/publicaciones/06IROSworkshop_cp.pdf

Hierarchical localization by matching vertical lines in omnidirectional images

In this paper, we are interested in self-supervised learning the motion cues in videos using dynamic motion filters for a better motion representation to finally boost human action recognition in particular. Thus far, the vision community has focused on spatio-temporal approaches using standard filters, rather we here propose dynamic filters that adaptively learn the video-specific internal motion representation by predicting the short-term future frames. We name this new motion representation, as dynamic motion representation (DMR) and is embedded inside of 3D convolutional network as a new layer, which captures the visual appearance and motion dynamics throughout entire video clip via end-to-end network learning. Simultaneously, we utilize these motion representation to enrich video classification. We have designed the frame prediction task as an auxiliary task to empower the classification problem. With these overall objectives, to this end, we introduce a novel unified spatio-temporal 3D-CNN architecture (DynamoNet) that jointly optimizes the video classification and learning motion representation by predicting future frames as a multi-task learning problem. We conduct experiments on challenging human action datasets: Kinetics 400, UCF101, HMDB51. The experiments using the proposed DynamoNet show promising results on all the datasets.

DynamoNet: Dynamic Action and Motion Network

The motivation for this special issue is twofold. Firstly we wished to provide an issue that combines papers on texture analysis with papers on texture synthesis. By analysis we mean the processing of images of texture to produce a set of measurements. These measurements could concern: statistics such as average surface roughness or directionality; the class of texture to which the surface belongs; or the mapping of the surface into different regions using segmentation algorithms. By synthesis we mean the generation of descriptions or images of surface texture for computer graphics applications. Typical examples concern the generation of large areas of surface texture that are perceptually identical to the small samples from which they have been generated. Secondly we wanted to provide a showcase for recent research on three-dimensional surface texture. Critically the images of such surfaces are not purely a function of the colour albedo pattern. Rather, spatial variation of relief and reflectance mean that shadows, highlights and occlusion all affect the resulting imagery. Thus images are a function of viewpoint and illumination condition as well as surface texture—and so texture synthesis and analysis must take these effects into account. These issues are timely also when considered in the wider context of image-based rendering, of which texture mapping in a way was a precursor. This active field is a good example of the confluence of computer vision and graphics. This issue is organised into three sections. The first set of papers does not specifically address either analysis or synthesis as defined above but may be used for either. The first two of these papers propose and experimentally validate two different models of texture, while the third paper exploits shading and shadowing in surface reconstruction. The next three papers are all concerned with the classification of three-dimensional surface textures while the last four address texture synthesis issues.

Editorial: Special Issue on Texture Analysis and Synthesi

Object referring has important applications, especially for human-machine interaction. While having received great attention, the task is mainly attacked with written language (text) as input rather than spoken language (speech), which is more natural. This paper investigates Object Referring with Spoken Language (ORSpoken) by presenting two datasets and one novel approach. Objects are annotated with their locations in images, text descriptions and speech descriptions. This makes the datasets ideal for multi-modality learning. The approach is developed by carefully taking down ORSpoken problem into three sub-problems and introducing task-specific vision-language interactions at the corresponding levels. Experiments show that our method outperforms competing methods consistently and significantly. The approach is also evaluated in the presence of audio noise, showing the efficacy of the proposed vision-language interaction methods in counteracting background noise.

Luc Van Gool

Papers

Robust Multi-Person Tracking from Moving Platforms

Hierarchical localization by matching vertical lines in omnidirectional images

DynamoNet: Dynamic Action and Motion Network

Editorial: Special Issue on Texture Analysis and Synthesi

Object Referring in Visual Scene with Spoken Language