We propose a deep convolutional neural network architecture codenamed Inception that achieves the new state of the art for classification and detection in the ImageNet Large-Scale Visual Recognition Challenge 2014 (ILSVRC14). The main hallmark of this architecture is the improved utilization of the computing resources inside the network. By a carefully crafted design, we increased the depth and width of the network while keeping the computational budget constant. To optimize quality, the architectural decisions were based on the Hebbian principle and the intuition of multi-scale processing. One particular incarnation used in our submission for ILSVRC14 is called GoogLeNet, a 22 layers deep network, the quality of which is assessed in the context of classification and detection.

/pdf/going-deeper-with-convolutions-1yobw2o2ds.pdf

Going deeper with convolutions

Machine Learning is the study of methods for programming computers to learn. Computers are applied to a wide range of tasks, and for most of these it is relatively easy for programmers to design and implement the necessary software. However, there are many tasks for which this is difficult or impossible. These can be divided into four general categories. First, there are problems for which there exist no human experts. For example, in modern automated manufacturing facilities, there is a need to predict machine failures before they occur by analyzing sensor readings. Because the machines are new, there are no human experts who can be interviewed by a programmer to provide the knowledge necessary to build a computer system. A machine learning system can study recorded data and subsequent machine failures and learn prediction rules. Second, there are problems where human experts exist, but where they are unable to explain their expertise. This is the case in many perceptual tasks, such as speech recognition, hand-writing recognition, and natural language understanding. Virtually all humans exhibit expert-level abilities on these tasks, but none of them can describe the detailed steps that they follow as they perform them. Fortunately, humans can provide machines with examples of the inputs and correct outputs for these tasks, so machine learning algorithms can learn to map the inputs to the outputs. Third, there are problems where phenomena are changing rapidly. In finance, for example, people would like to predict the future behavior of the stock market, of consumer purchases, or of exchange rates. These behaviors change frequently, so that even if a programmer could construct a good predictive computer program, it would need to be rewritten frequently. A learning program can relieve the programmer of this burden by constantly modifying and tuning a set of learned prediction rules. Fourth, there are applications that need to be customized for each computer user separately. Consider, for example, a program to filter unwanted electronic mail messages. Different users will need different filters. It is unreasonable to expect each user to program his or her own rules, and it is infeasible to provide every user with a software engineer to keep the rules up-to-date. A machine learning system can learn which mail messages the user rejects and maintain the filtering rules automatically. Machine learning addresses many of the same research questions as the fields of statistics, data mining, and psychology, but with differences of emphasis. Statistics focuses on understanding the phenomena that have generated the data, often with the goal of testing different hypotheses about those phenomena. Data mining seeks to find patterns in the data that are understandable by people. Psychological studies of human learning aspire to understand the mechanisms underlying the various learning behaviors exhibited by people (concept learning, skill acquisition, strategy change, etc.).

Machine learning

We present a system for accurate real-time mapping of complex and arbitrary indoor scenes in variable lighting conditions, using only a moving low-cost depth camera and commodity graphics hardware. We fuse all of the depth data streamed from a Kinect sensor into a single global implicit surface model of the observed scene in real-time. The current sensor pose is simultaneously obtained by tracking the live depth frame relative to the global model using a coarse-to-fine iterative closest point (ICP) algorithm, which uses all of the observed depth data available. We demonstrate the advantages of tracking against the growing full surface model compared with frame-to-frame tracking, obtaining tracking and mapping results in constant time within room sized scenes with limited drift and high accuracy. We also show both qualitative and quantitative results relating to various aspects of our tracking and mapping system. Modelling of natural scenes, in real-time with only commodity sensor and GPU hardware, promises an exciting step forward in augmented reality (AR), in particular, it allows dense surfaces to be reconstructed in real-time, with a level of detail and robustness beyond any solution yet presented using passive computer vision.

/pdf/kinectfusion-real-time-dense-surface-mapping-and-tracking-pl8e8pbmre.pdf

KinectFusion: Real-time dense surface mapping and tracking

Humans perceive the three-dimensional structure of the world with apparent ease. However, despite all of the recent advances in computer vision research, the dream of having a computer interpret an image at the same level as a two-year old remains elusive. Why is computer vision such a challenging problem and what is the current state of the art? Computer Vision: Algorithms and Applications explores the variety of techniques commonly used to analyze and interpret images. It also describes challenging real-world applications where vision is being successfully used, both for specialized applications such as medical imaging, and for fun, consumer-level tasks such as image editing and stitching, which students can apply to their own personal photos and videos. More than just a source of recipes, this exceptionally authoritative and comprehensive textbook/reference also takes a scientific approach to basic vision problems, formulating physical models of the imaging process before inverting them to produce descriptions of a scene. These problems are also analyzed using statistical models and solved using rigorous engineering techniques Topics and features: structured to support active curricula and project-oriented courses, with tips in the Introduction for using the book in a variety of customized courses; presents exercises at the end of each chapter with a heavy emphasis on testing algorithms and containing numerous suggestions for small mid-term projects; provides additional material and more detailed mathematical topics in the Appendices, which cover linear algebra, numerical techniques, and Bayesian estimation theory; suggests additional reading at the end of each chapter, including the latest research in each sub-field, in addition to a full Bibliography at the end of the book; supplies supplementary course material for students at the associated website, http://szeliski.org/Book/. Suitable for an upper-level undergraduate or graduate-level course in computer science or engineering, this textbook focuses on basic techniques that work under real-world conditions and encourages students to push their creative boundaries. Its design and exposition also make it eminently suitable as a unique reference to the fundamental techniques and current research literature in computer vision.

/pdf/computer-vision-algorithms-and-applications-25dn6wu83j.pdf

Computer Vision: Algorithms and Applications

“Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告

We present a novel method for single image depth estimation using surface normal constraints. Existing depth estimation methods either suffer from the lack of geometric constraints, or are limited to the difficulty of reliably capturing geometric context, which leads to a bottleneck of depth estimation quality. We therefore introduce a simple yet effective method, named Adaptive Surface Normal (ASN) constraint, to effectively correlate the depth estimation with geometric consistency. Our key idea is to adaptively determine the reliable local geometry from a set of randomly sampled candidates to derive surface normal constraint, for which we measure the consistency of the geometric contextual features. As a result, our method can faithfully reconstruct the 3D geometry and is robust to local shape variations, such as boundaries, sharp corners and noises. We conduct extensive evaluations and comparisons using public datasets. The experimental results demonstrate our method outperforms the state-of-the-art methods and has superior efficiency and robustness.

Adaptive Surface Normal Constraint for Depth Estimation

. Implicit neural representations of 3D shapes form strong priors that are useful for various applications, such as single and multiple view 3D reconstruction. A downside of existing neural representations is that they require multiple network evaluations for rendering, which leads to high computational costs. This limitation forms a bottleneck particularly in the context of inverse problems, such as image-based 3D reconstruction. To address this issue, in this paper (i) we propose a novel hybrid 3D object representation based on a signed distance function (SDF) that we augment with a directional distance function (DDF), so that we can predict distances to the object surface from any point on a sphere enclosing the object. Moreover, (ii) using the proposed hybrid representation we address the multi-view consistency problem common in existing DDF representations. We evaluate our novel hybrid representation on the task of single-view depth reconstruction and show that our method is several times faster compared to competing methods, while at the same time achieving better reconstruction accuracy.

HDSDF: Hybrid Directional and Signed Distance Functions for Fast Inverse Rendering

INTRODUCTION This document contains additional information on video alignment, the display applications exploration, and on the user study conducted to evaluate our system. We give details on homography validation, estimating missing homographies, and homography filtering; we report on methods to extended Vidicontexts to work on portable devices and spherical displays; the iMovie and iMovie+pano interfaces used in our experiment are described in more detail; and we include a additional descriptions of the results obtained from our user study presented in the main paper.

/pdf/video-collections-in-panoramic-contexts-supplemental-3sfewu7pn2.pdf

Video Collections in Panoramic Contexts Supplemental Material

In our paper on Eikonal Rendering [1], presented in the SIGGRAPH 2007 main paper program, we propose a novel algorithm to render a variety of sophisticated lighting effects in and around refractive objects in real-time on a single PC. Our method enables us to realistically display objects with spatially-varying refractive indices, inhomogeneous attenuation characteristics, as well as well as spatially-varying reflectance and anisotropic scattering properties. We can reproduce arbitrarily curved light paths, volume and surface caustics, anisotropic scattering as well as total reflection by means of the same efficient theoretical framework. In our approach, scenes are represented volumetrically. One core component of our method is a fast GPU particle tracer to compute viewing ray trajectories. It uses ordinary differential equations derived from the eikonal equation. The second important component is a light simulator, which utilizes a similar ODE-based framework to adaptively trace light wavefronts through the scene in a few seconds. While the conference paper [1] focuses on the development of the theory and the validation of our results, this sketch describes in detail the new concepts and data structures that we developed to implement view rendering and light wavefront tracing on the GPU (see figure 1 for a stage overview).

GPU-based light wavefront simulation for real-time refractive object rendering

The advancement of generative radiance fields has pushed the boundary of
3D-aware image synthesis. Motivated by the observation that a 3D object should
look realistic from multiple viewpoints, these methods introduce a multi-view
constraint as regularization to learn valid 3D radiance fields from 2D images.
Despite the progress, they often fall short of capturing accurate 3D shapes due
to the shape-color ambiguity, limiting their applicability in downstream tasks.
In this work, we address this ambiguity by proposing a novel shading-guided
generative implicit model that is able to learn a starkly improved shape
representation. Our key insight is that an accurate 3D shape should also yield
a realistic rendering under different lighting conditions. This multi-lighting
constraint is realized by modeling illumination explicitly and performing
shading with various lighting conditions. Gradients are derived by feeding the
synthesized images to a discriminator. To compensate for the additional
computational burden of calculating surface normals, we further devise an
efficient volume rendering strategy via surface tracking, reducing the training
and inference time by 24% and 48%, respectively. Our experiments on multiple
datasets show that the proposed approach achieves photorealistic 3D-aware image
synthesis while capturing accurate underlying 3D shapes. We demonstrate
improved performance of our approach on 3D shape reconstruction against
existing methods, and show its applicability on image relighting. Our code will
be released at https://github.com/XingangPan/ShadeGAN.

Christian Theobalt

Papers

Adaptive Surface Normal Constraint for Depth Estimation

HDSDF: Hybrid Directional and Signed Distance Functions for Fast Inverse Rendering

Video Collections in Panoramic Contexts Supplemental Material

GPU-based light wavefront simulation for real-time refractive object rendering

A Shading-Guided Generative Implicit Model for Shape-Accurate 3D-Aware Image Synthesis