View synthesis

About: View synthesis is a research topic. Over the lifetime, 1701 publications have been published within this topic receiving 42333 citations.

Multiview synthesis from stereo views

Abstract: A simple and efficient technique for view synthesis from a pair of rectified stereo views is presented here. Depth and colour information from the stereo pair is used to generate an intermediate view and its corresponding disparity map at any given position along the horizontal baseline. Holes in the generated view and disparity map are filled by using the disparity values to separate the hole positions into those present in the foreground or background layers. The results obtained are evaluated using objective quality measures and are compared with other state of the art methods. The high quality intermediate images generated here can be used in applications such as multiview autostereoscopic displays and free-viewpoint television.

Multi-View Imaging: Capturing and Rendering Interactive Environments

Abstract: This paper presents a self-reconfigurable camera array system that captures and renders 3D virtual scenes interactively It is composed of an array of 48 cameras mounted on mobile platforms We propose an efficient algorithm that is capable of rendering high-quality novel views from the captured images The algorithm reconstructs a view-dependent multiresolution 2D mesh model of the scene geometry on the fly and uses it for rendering The algorithm combines region of interest (ROI) identification, JPEG image decompression, lens distortion correction, scene geometry reconstruction and novel view synthesis seamlessly on a single Intel Xeon 24 GHz processor, which is capable of generating novel views at 4 10 frames per second (fps) In addition, we present a view-dependent adaptive capturing scheme that moves the cameras in order to show even better rendering results Such camera reconfiguration naturally leads to a nonuniform arrangement of the cameras on the camera plane, which is both view-dependent and scene-dependent

Novel View Synthesis on Unpaired Data by Conditional Deformable Variational Auto-Encoder

Abstract: Novel view synthesis often needs the paired data from both the source and target views. This paper proposes a view translation model under cVAE-GAN framework without requiring the paired data. We design a conditional deformable module (CDM) which uses the view condition vectors as the filters to convolve the feature maps of the main branch in VAE. It generates several pairs of displacement maps to deform the features, like the 2D optical flows. The results are fed into the deformed feature based normalization module (DFNM), which scales and offsets the main branch feature, given its deformed one as the input from the side branch. Taking the advantage of the CDM and DFNM, the encoder outputs a view-irrelevant posterior, while the decoder takes the code drawn from it to synthesize the reconstructed and the view-translated images. To further ensure the disentanglement between the views and other factors, we add adversarial training on the code. The results and ablation studies on MultiPIE and 3D chair datasets validate the effectiveness of the framework in cVAE and the designed module.

NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction

Abstract: We present a novel neural surface reconstruction method, called NeuS, for reconstructing objects and scenes with high fidelity from 2D image inputs. Existing neural surface reconstruction approaches, such as DVR and IDR, require foreground mask as supervision, easily get trapped in local minima, and therefore struggle with the reconstruction of objects with severe self-occlusion or thin structures. Meanwhile, recent neural methods for novel view synthesis, such as NeRF and its variants, use volume rendering to produce a neural scene representation with robustness of optimization, even for highly complex objects. However, extracting high-quality surfaces from this learned implicit representation is difficult because there are not sufficient surface constraints in the representation. In NeuS, we propose to represent a surface as the zero-level set of a signed distance function (SDF) and develop a new volume rendering method to train a neural SDF representation. We observe that the conventional volume rendering method causes inherent geometric errors (i.e. bias) for surface reconstruction, and therefore propose a new formulation that is free of bias in the first order of approximation, thus leading to more accurate surface reconstruction even without the mask supervision. Experiments on the DTU dataset and the BlendedMVS dataset show that NeuS outperforms the state-of-the-arts in high-quality surface reconstruction, especially for objects and scenes with complex structures and self-occlusion.