“Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告

From the Publisher: 
The accessible presentation of this book gives both a general view of the entire computer vision enterprise and also offers sufficient detail to be able to build useful applications. Users learn techniques that have proven to be useful by first-hand experience and a wide range of mathematical methods. A CD-ROM with every copy of the text contains source code for programming practice, color images, and illustrative movies. Comprehensive and up-to-date, this book includes essential topics that either reflect practical significance or are of theoretical importance. Topics are discussed in substantial and increasing depth. Application surveys describe numerous important application areas such as image based rendering and digital libraries. Many important algorithms broken down and illustrated in pseudo code. Appropriate for use by engineers as a comprehensive reference to the computer vision enterprise.

Computer vision : a modern approach = 计算机视觉 : 一种现代的方法

We present a method that achieves state-of-the-art results for synthesizing novel views of complex scenes by optimizing an underlying continuous volumetric scene function using a sparse set of input views. Our algorithm represents a scene using a fully-connected (non-convolutional) deep network, whose input is a single continuous 5D coordinate (spatial location $(x,y,z)$ and viewing direction $(\theta, \phi)$) and whose output is the volume density and view-dependent emitted radiance at that spatial location. We synthesize views by querying 5D coordinates along camera rays and use classic volume rendering techniques to project the output colors and densities into an image. Because volume rendering is naturally differentiable, the only input required to optimize our representation is a set of images with known camera poses. We describe how to effectively optimize neural radiance fields to render photorealistic novel views of scenes with complicated geometry and appearance, and demonstrate results that outperform prior work on neural rendering and view synthesis. View synthesis results are best viewed as videos, so we urge readers to view our supplementary video for convincing comparisons.

NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis

A key requirement for leveraging supervised deep learning methods is the availability of large, labeled datasets. Unfortunately, in the context of RGB-D scene understanding, very little data is available &#x2013; current datasets cover a small range of scene views and have limited semantic annotations. To address this issue, we introduce ScanNet, an RGB-D video dataset containing 2.5M views in 1513 scenes annotated with 3D camera poses, surface reconstructions, and semantic segmentations. To collect this data, we designed an easy-to-use and scalable RGB-D capture system that includes automated surface reconstruction and crowdsourced semantic annotation. We show that using this data helps achieve state-of-the-art performance on several 3D scene understanding tasks, including 3D object classification, semantic voxel labeling, and CAD model retrieval.

/pdf/scannet-richly-annotated-3d-reconstructions-of-indoor-scenes-37ihdazbor.pdf

ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes

IEEE Transactions on Pattern Analysis and Machine Intelligence

Synthesizing realistic videos of humans using neural networks has been a popular alternative to the conventional graphics-based rendering pipeline due to its high efficiency. Existing works typically formulate this as an image-to-image translation problem in 2D screen space, which leads to artifacts such as over-smoothing, missing body parts, and temporal instability of fine-scale detail, such as pose-dependent wrinkles in the clothing. In this article, we propose a novel human video synthesis method that approaches these limiting factors by explicitly disentangling the learning of time-coherent fine-scale details from the embedding of the human in 2D screen space. More specifically, our method relies on the combination of two convolutional neural networks (CNNs). Given the pose information, the first CNN predicts a dynamic texture map that contains time-coherent high-frequency details, and the second CNN conditions the generation of the final video on the temporally coherent output of the first CNN. We demonstrate several applications of our approach, such as human reenactment and novel view synthesis from monocular video, where we show significant improvement over the state of the art both qualitatively and quantitatively.

/pdf/learning-dynamic-textures-for-neural-rendering-of-human-21ktprln7z.pdf

Learning Dynamic Textures for Neural Rendering of Human Actors

In this paper, we present an end-to-end pipeline for the online reconstruction of large-scale outdoor environments and tightly confined indoor spaces using a low-cost consumer-level hand-held RGB-D sensor. While scanning, the user sees a live view of the current reconstruction, allowing him to intervene immediately and to adapt the sensor path to the current scanning result. After a raw reconstruction has been acquired, we interactively warp the digital model to fit a geo-referenced map using a handle based deformation paradigm. Even large sites can be scanned within a few minutes, and no costly postprocessing is required.

We developed our prototype in cooperation with researchers from the field of ancient history and geography and extensively tested the system under real world conditions on an archeological excavation in Metropolis, Ionia, Turkey. The quality of the acquired digitized raw 3D models is evaluated by comparing them to actual imagery and a geo-referenced map of the excavation site. Our reconstructions can be used to take virtual measurements that are often required in research and are the basis for a digital preservation of our cultural heritage. In addition, digital models are a helpful tool for teaching as well as for edutainment purposes making such information accessible to the general public.

Low-cost real-time 3D reconstruction of large-scale excavation sites using an RGB-D camera

We present a novel approach that enables photo-realistic re-animation of portrait videos using only an input video. In contrast to existing approaches that are restricted to manipulations of facial expressions only, we are the first to transfer the full 3D head position, head rotation, face expression, eye gaze, and eye blinking from a source actor to a portrait video of a target actor. The core of our approach is a generative neural network with a novel space-time architecture. The network takes as input synthetic renderings of a parametric face model, based on which it predicts photo-realistic video frames for a given target actor. The realism in this rendering-to-video transfer is achieved by careful adversarial training, and as a result, we can create modified target videos that mimic the behavior of the synthetically-created input. In order to enable source-to-target video re-animation, we render a synthetic target video with the reconstructed head animation parameters from a source video, and feed it into the trained network -- thus taking full control of the target. With the ability to freely recombine source and target parameters, we are able to demonstrate a large variety of video rewrite applications without explicitly modeling hair, body or background. For instance, we can reenact the full head using interactive user-controlled editing, and realize high-fidelity visual dubbing. To demonstrate the high quality of our output, we conduct an extensive series of experiments and evaluations, where for instance a user study shows that our video edits are hard to detect.

Deep Video Portraits

We present Non-Rigid Neural Radiance Fields (NR-NeRF), a reconstruction and novel view synthesis approach for general non-rigid dynamic scenes. Our approach takes RGB images of a dynamic scene as input (e.g., from a monocular video recording), and creates a high-quality space-time geometry and appearance representation. We show that a single handheld consumer-grade camera is sufficient to synthesize sophisticated renderings of a dynamic scene from novel virtual camera views, e.g. a `bullet-time' video effect. NR-NeRF disentangles the dynamic scene into a canonical volume and its deformation. Scene deformation is implemented as ray bending, where straight rays are deformed non-rigidly. We also propose a novel rigidity network to better constrain rigid regions of the scene, leading to more stable results. The ray bending and rigidity network are trained without explicit supervision. Our formulation enables dense correspondence estimation across views and time, and compelling video editing applications such as motion exaggeration. Our code will be open sourced.

/pdf/non-rigid-neural-radiance-fields-reconstruction-and-novel-5bph89veug.pdf

Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video

We present a novel method to animate a static geometry by cloning a captured animation sequence. More precisely, given a sequence of range scans of a deforming object which has been captured by a real-time 3D scanner, we describe a novel algorithm to clone the animation of the recorded geometry onto another triangle mesh. To achieve this, we reconstruct a coherent animated mesh of the input sequence using a template deformation approach. Then we employ a new algorithm for robust marker-less non-rigid registration to deform one frame of the generated animation such that it matches a different 3D model. The resulting registration is further used to find correspondences between the animation and the target object which are in turn used to transfer the animation of the recorded sequence onto the target shape. Transferring the entire geometry of the animations results in very convincing facial expressions since even the smallest expression wrinkles are preserved. We evaluate the robustness and the performance of the proposed algorithms using a variety of data sets, including facial animations and whole body animations. Copyright © 2010 John Wiley & Sons, Ltd.

We propose new method for animation reconstruction from real-time 3D scanner data and a simple non-rigid registration technique based purely on geometric features. We then use the deformable registration to clone the reconstructed animation sequences onto a static triangle mesh, yielding realistic results for facial and whole-body animations.

Michael Zollhöfer

Papers

Learning Dynamic Textures for Neural Rendering of Human Actors

Low-cost real-time 3D reconstruction of large-scale excavation sites using an RGB-D camera

Deep Video Portraits

Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video

Animation transplantation