Book ChapterDOI
Free View Synthesis
Gernot Riegler,Vladlen Koltun +1 more
- pp 623-640
Reads0
Chats0
TLDR
This work presents a method for novel view synthesis from input images that are freely distributed around a scene that can synthesize images for free camera movement through the scene, and works for general scenes with unconstrained geometric layouts.Abstract:
We present a method for novel view synthesis from input images that are freely distributed around a scene. Our method does not rely on a regular arrangement of input views, can synthesize images for free camera movement through the scene, and works for general scenes with unconstrained geometric layouts. We calibrate the input images via SfM and erect a coarse geometric scaffold via MVS. This scaffold is used to create a proxy depth map for a novel view of the scene. Based on this depth map, a recurrent encoder-decoder network processes reprojected features from nearby views and synthesizes the new view. Our network does not need to be optimized for a given scene. After training on a dataset, it works in previously unseen environments with no fine-tuning or per-scene optimization. We evaluate the presented approach on challenging real-world datasets, including Tanks and Temples, where we demonstrate successful view synthesis for the first time and substantially outperform prior and concurrent work.read more
Citations
More filters
Posted Content
pixelNeRF: Neural Radiance Fields from One or Few Images
TL;DR: For example, pixelNeRF as discussed by the authors predicts a continuous neural scene representation conditioned on one or few input images, which can be trained across multiple scenes to learn a scene prior, enabling it to perform novel view synthesis in a feed-forward manner from a sparse set of views.
Posted Content
NeRF++: Analyzing and Improving Neural Radiance Fields.
TL;DR: A parametrization issue involved in applying NeRF to 360 captures of objects within large-scale, unbounded 3D scenes is addressed, and the method improves view synthesis fidelity in this challenging scenario.
Proceedings ArticleDOI
IBRNet: Learning Multi-View Image-Based Rendering
Qianqian Wang,Zhicheng Wang,Kyle Genova,Pratul P. Srinivasan,Howard Zhou,Jonathan T. Barron,Ricardo Martin-Brualla,Noah Snavely,Thomas Funkhouser +8 more
TL;DR: A method that synthesizes novel views of complex scenes by interpolating a sparse set of nearby views using a network architecture that includes a multilayer perceptron and a ray transformer that estimates radiance and volume density at continuous 5D locations.
Posted Content
Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes
TL;DR: A method to perform novel view and time synthesis of dynamic scenes, requiring only a monocular video with known camera poses as input, is presented, and a new representation that models the dynamic scene as a time-variant continuous function of appearance, geometry, and 3D scene motion is introduced.
Proceedings ArticleDOI
pixelNeRF: Neural Radiance Fields from One or Few Images
TL;DR: PixelNeRF as mentioned in this paper is a learning framework that predicts a continuous neural scene representation conditioned on one or few input images, enabling it to perform novel view synthesis in a feed-forward manner from a sparse set of views.
References
More filters
Proceedings Article
Adam: A Method for Stochastic Optimization
Diederik P. Kingma,Jimmy Ba +1 more
TL;DR: This work introduces Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments, and provides a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework.
Proceedings Article
Very Deep Convolutional Networks for Large-Scale Image Recognition
Karen Simonyan,Andrew Zisserman +1 more
TL;DR: In this paper, the authors investigated the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting and showed that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 layers.
Book ChapterDOI
U-Net: Convolutional Networks for Biomedical Image Segmentation
TL;DR: Neber et al. as discussed by the authors proposed a network and training strategy that relies on the strong use of data augmentation to use the available annotated samples more efficiently, which can be trained end-to-end from very few images and outperforms the prior best method (a sliding-window convolutional network) on the ISBI challenge for segmentation of neuronal structures in electron microscopic stacks.
Proceedings ArticleDOI
Learning Phrase Representations using RNN Encoder--Decoder for Statistical Machine Translation
Kyunghyun Cho,Bart van Merriënboer,Caglar Gulcehre,Dzmitry Bahdanau,Fethi Bougares,Holger Schwenk,Yoshua Bengio,Yoshua Bengio,Yoshua Bengio +8 more
TL;DR: In this paper, the encoder and decoder of the RNN Encoder-Decoder model are jointly trained to maximize the conditional probability of a target sequence given a source sequence.
Posted Content
U-Net: Convolutional Networks for Biomedical Image Segmentation
TL;DR: It is shown that such a network can be trained end-to-end from very few images and outperforms the prior best method (a sliding-window convolutional network) on the ISBI challenge for segmentation of neuronal structures in electron microscopic stacks.