Casual 3D photography
Reads0
Chats0
TLDR
An algorithm that enables casual 3D photography and proposes a novel parallax-tolerant stitching algorithm that warps the depth maps into the central panorama and stitches two color-and-depth panoramas for the front and back scene surfaces.Abstract:
We present an algorithm that enables casual 3D photography. Given a set of input photos captured with a hand-held cell phone or DSLR camera, our algorithm reconstructs a 3D photo, a central panoramic, textured, normal mapped, multi-layered geometric mesh representation. 3D photos can be stored compactly and are optimized for being rendered from viewpoints that are near the capture viewpoints. They can be rendered using a standard rasterization pipeline to produce perspective views with motion parallax. When viewed in VR, 3D photos provide geometrically consistent views for both eyes. Our geometric representation also allows interacting with the scene using 3D geometry-aware effects, such as adding new objects to the scene and artistic lighting effects.Our 3D photo reconstruction algorithm starts with a standard structure from motion and multi-view stereo reconstruction of the scene. The dense stereo reconstruction is made robust to the imperfect capture conditions using a novel near envelope cost volume prior that discards erroneous near depth hypotheses. We propose a novel parallax-tolerant stitching algorithm that warps the depth maps into the central panorama and stitches two color-and-depth panoramas for the front and back scene surfaces. The two panoramas are fused into a single non-redundant, well-connected geometric mesh. We provide videos demonstrating users interactively viewing and manipulating our 3D photos.read more
Citations
More filters
Journal ArticleDOI
Stereo magnification: learning view synthesis using multiplane images
TL;DR: This paper explores an intriguing scenario for view synthesis: extrapolating views from imagery captured by narrow-baseline stereo cameras, including VR cameras and now-widespread dual-lens camera phones, and proposes a learning framework that leverages a new layered representation that is called multiplane images (MPIs).
Journal ArticleDOI
Local light field fusion: practical view synthesis with prescriptive sampling guidelines
Ben Mildenhall,Pratul P. Srinivasan,Rodrigo Ortiz-Cayon,Nima Khademi Kalantari,Ravi Ramamoorthi,Ren Ng,Abhishek Kar +6 more
TL;DR: An algorithm for view synthesis from an irregular grid of sampled views that first expands each sampled view into a local light field via a multiplane image (MPI) scene representation, then renders novel views by blending adjacent local light fields.
Posted Content
Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines
Ben Mildenhall,Pratul P. Srinivasan,Rodrigo Ortiz-Cayon,Nima Khademi Kalantari,Ravi Ramamoorthi,Ren Ng,Abhishek Kar +6 more
TL;DR: An algorithm for view synthesis from an irregular grid of sampled views that first expands each sampled view into a local light field via a multiplane image (MPI) scene representation, then renders novel views by blending adjacent local light fields.
Proceedings ArticleDOI
DeepView: View Synthesis With Learned Gradient Descent
John Flynn,Michael Broxton,Paul Debevec,Matthew DuVall,Graham Fyffe,Ryan Overbeck,Noah Snavely,Richard Tucker +7 more
TL;DR: This work presents a novel approach to view synthesis using multiplane images (MPIs) that incorporates occlusion reasoning, improving performance on challenging scene features such as object boundaries, lighting reflections, thin structures, and scenes with high depth complexity.
Journal ArticleDOI
Deep blending for free-viewpoint image-based rendering
TL;DR: This work presents a new deep learning approach to blending for IBR, in which held-out real image data is used to learn blending weights to combine input photo contributions, and designs the network architecture and the training loss to provide high quality novel view synthesis, while reducing temporal flickering artifacts.
References
More filters
Journal ArticleDOI
A taxonomy and evaluation of dense two-frame stereo correspondence algorithms
TL;DR: This paper has designed a stand-alone, flexible C++ implementation that enables the evaluation of individual components and that can easily be extended to include new algorithms.
Journal ArticleDOI
What energy functions can be minimized via graph cuts
Vladimir Kolmogorov,R. Zabin +1 more
TL;DR: This work gives a precise characterization of what energy functions can be minimized using graph cuts, among the energy functions that can be written as a sum of terms containing three or fewer binary variables.
Proceedings Article
Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
TL;DR: In this article, two deep network stacks are employed to make a coarse global prediction based on the entire image, and another to refine this prediction locally, which achieves state-of-the-art results on both NYU Depth and KITTI.
Journal ArticleDOI
Accurate, Dense, and Robust Multiview Stereopsis
Yasutaka Furukawa,Jean Ponce +1 more
TL;DR: A novel algorithm for multiview stereopsis that outputs a dense set of small rectangular patches covering the surfaces visible in the images, which outperforms all others submitted so far for four out of the six data sets.
Journal ArticleDOI
ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras
Raul Mur-Artal,Juan D. Tardós +1 more
TL;DR: ORB-SLAM2 as mentioned in this paper is a complete SLAM system for monocular, stereo and RGB-D cameras, including map reuse, loop closing and relocalization capabilities.