We propose a deep convolutional neural network architecture codenamed Inception that achieves the new state of the art for classification and detection in the ImageNet Large-Scale Visual Recognition Challenge 2014 (ILSVRC14). The main hallmark of this architecture is the improved utilization of the computing resources inside the network. By a carefully crafted design, we increased the depth and width of the network while keeping the computational budget constant. To optimize quality, the architectural decisions were based on the Hebbian principle and the intuition of multi-scale processing. One particular incarnation used in our submission for ILSVRC14 is called GoogLeNet, a 22 layers deep network, the quality of which is assessed in the context of classification and detection.

/pdf/going-deeper-with-convolutions-1yobw2o2ds.pdf

Going deeper with convolutions

We present a new dataset with the goal of advancing the state-of-the-art in object recognition by placing the question of object recognition in the context of the broader question of scene understanding. This is achieved by gathering images of complex everyday scenes containing common objects in their natural context. Objects are labeled using per-instance segmentations to aid in precise object localization. Our dataset contains photos of 91 objects types that would be easily recognizable by a 4 year old. With a total of 2.5 million labeled instances in 328k images, the creation of our dataset drew upon extensive crowd worker involvement via novel user interfaces for category detection, instance spotting and instance segmentation. We present a detailed statistical analysis of the dataset in comparison to PASCAL, ImageNet, and SUN. Finally, we provide baseline performance analysis for bounding box and segmentation detection results using a Deformable Parts Model.

/pdf/microsoft-coco-common-objects-in-context-55ihxq1zt5.pdf

Microsoft COCO: Common Objects in Context

We investigate conditional adversarial networks as a general-purpose solution to image-to-image translation problems. These networks not only learn the mapping from input image to output image, but also learn a loss function to train this mapping. This makes it possible to apply the same generic approach to problems that traditionally would require very different loss formulations. We demonstrate that this approach is effective at synthesizing photos from label maps, reconstructing objects from edge maps, and colorizing images, among other tasks. Moreover, since the release of the pix2pix software associated with this paper, hundreds of twitter users have posted their own artistic experiments using our system. As a community, we no longer hand-engineer our mapping functions, and this work suggests we can achieve reasonable results without handengineering our loss functions either.

/pdf/image-to-image-translation-with-conditional-adversarial-5058t24et2.pdf

Image-to-Image Translation with Conditional Adversarial Networks

Image-to-image translation is a class of vision and graphics problems where the goal is to learn the mapping between an input image and an output image using a training set of aligned image pairs. However, for many tasks, paired training data will not be available. We present an approach for learning to translate an image from a source domain X to a target domain Y in the absence of paired examples. Our goal is to learn a mapping G : X → Y such that the distribution of images from G(X) is indistinguishable from the distribution Y using an adversarial loss. Because this mapping is highly under-constrained, we couple it with an inverse mapping F : Y → X and introduce a cycle consistency loss to push F(G(X)) ≈ X (and vice versa). Qualitative results are presented on several tasks where paired training data does not exist, including collection style transfer, object transfiguration, season transfer, photo enhancement, etc. Quantitative comparisons against several prior methods demonstrate the superiority of our approach.

Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks

We investigate conditional adversarial networks as a general-purpose solution to image-to-image translation problems. These networks not only learn the mapping from input image to output image, but also learn a loss function to train this mapping. This makes it possible to apply the same generic approach to problems that traditionally would require very different loss formulations. We demonstrate that this approach is effective at synthesizing photos from label maps, reconstructing objects from edge maps, and colorizing images, among other tasks. Indeed, since the release of the pix2pix software associated with this paper, a large number of internet users (many of them artists) have posted their own experiments with our system, further demonstrating its wide applicability and ease of adoption without the need for parameter tweaking. As a community, we no longer hand-engineer our mapping functions, and this work suggests we can achieve reasonable results without hand-engineering our loss functions either.

We present interactive techniques for visually annotating independently moving objects in a video stream. Features in the video are automatically tracked and grouped in an off-line preprocess that enables later interactive manipulation and annotation. Examples of such annotations include speech and thought balloons, video graffiti, hyperlinks, and path arrows. Our system also employs a directmanipulation interface for random frame access using spatial constraints. This annotation interface can be employed in a variety of applications including surveillance, film and video editing, visual tagging, and authoring rich media such as hyperlinked video.

http://grail.cs.washington.edu/projects/ivoa/tr07/UW-CSE-2007-04-01.pdf

Interactive Video Object Annotation

This paper presents an approach to render novel views from input photographs, a task which is commonly referred to as image based rendering. We first compute dense view dependent depthmaps using consistent segmentation. This method jointly computes multiview stereo and segments input photographs while accounting for mixed pixels (matting). We take the images with depth as our input and then propose two rendering algorithms to render novel views using the segmentation, both realtime and off-line. We demonstrate the results of our approach on a wide variety of scenes.

/pdf/a-consistent-segmentation-approach-to-image-based-rendering-5gg9jen0gj.pdf

A Consistent Segmentation Approach to Image-based Rendering

Matting and compositing are fundamental operations in graphics and visual effects. Despite having enjoyed wide usage for many years, traditional matting and compositing have limitations. Traditional matting methods either require special setups or cannot handle objects with complex silhouettes. Furthermore, the traditional compositing model is effective in modeling color blending effects but not reflection, refraction, and shadows. In this dissertation, we address these limitations and present a set of new compositing models and matting methods. To pull mattes of complex silhouettes from natural images, we introduce a principled statistical approach called Bayesian image matting. We also extend this algorithm to handle video sequences with the help of optical flow computation and background estimation. On the compositing side, previous work on environment matting has been shown to handle refraction and reflection, but the resulting mattes are not very accurate. We propose a more accurate environment matting model and method that requires using more images. For shadows, we develop a physically-motivated shadow compositing equation. Based on this equation, we introduce a shadow matting method for extracting shadow mattes from videos with natural backgrounds, and we demonstrate a novel process for acquiring the photometric and geometric properties of the background to enable creation of realistic shadow composites. Finally, we present a novel application of Bayesian image matting for animating still pictures.

New models and methods for matting and compositing

Existing approaches to interaction with digital video are complex, and some operations lack the immediacy of interactive feedback. In this thesis, I present a framework for video annotation, visualization, and interaction that harnesses computer vision to aid users in understanding and commu nicating with digital video. I first review the literature concerning visual representations, navigation, and manipulation of video data, and explore the literature of professional film editing, summarizing some of the techniques applied by and operations performed by film editors. I describe a new approach for computing the motion of points and objects in a video clip, and I present my interactive system that utilizes this data to visually annotate independently moving objects in the video, including speech and thought balloons, video graffiti, hyperlinks, and path arrows. I also demonstrate an application of this interface to construct visualizations of a short video clip in a single static image, using the visual language of storyboards. The principal advantage of the storyboard representation over standard representations of video is that it requires only a moment to observe and comprehend but at the same time retains much of the detail of the source video. The layout of the storyboard can be optimized to place the elements in a configuration that maximizes the clarity of presentation. Finally, I also demonstrate two novel interaction techniques for random video frame access using either the natural spatial dimensions of a storyboard representation or an individual video frame. 
Throughout the thesis, I discuss how these approaches simplify and streamline the understanding and manipulation of video materials.

A framework for video annotation, visualization, and interaction

In this paper, we demonstrate a fully automatic method for converting a still image into a realistic animated looping video. We target scenes with continuous fluid motion, such as flowing water and billowing smoke. Our method relies on the observation that this type of natural motion can be convincingly reproduced from a static Eulerian motion description, i.e. a single, temporally constant flow field that defines the immediate motion of a particle at a given 2D location. We use an image-to-image translation network to encode motion priors of natural scenes collected from online videos, so that for a new photo, we can synthesize a corresponding motion field. The image is then animated using the generated motion through a deep warping technique: pixels are encoded as deep features, those features are warped via Eulerian motion, and the resulting warped feature maps are decoded as images. In order to produce continuous, seamlessly looping video textures, we propose a novel video looping technique that flows features both forward and backward in time and then blends the results. We demonstrate the effectiveness and robustness of our method by applying it to a large collection of examples including beaches, waterfalls, and flowing rivers.

Brian Curless

Papers

Interactive Video Object Annotation

A Consistent Segmentation Approach to Image-based Rendering

New models and methods for matting and compositing

A framework for video annotation, visualization, and interaction

Animating Pictures with Eulerian Motion Fields