View synthesis

About: View synthesis is a research topic. Over the lifetime, 1701 publications have been published within this topic receiving 42333 citations.

Hole Filling for View Synthesis Using Depth Guided Global Optimization

Abstract: View synthesis is an effective way to generate multi-view contents from a limited number of views, and can be utilized for 2-D-to-3-D video conversion, multi-view video compression, and virtual reality. In the view synthesis techniques, depth-image-based rendering (DIBR) is an important method to generate virtual view from video-plus-depth sequence. However, some holes might be produced in the DIBR process. Many hole filling methods have been proposed to tackle this issue, but most of them cannot achieve globally coherent or acquire trusted contents. In this paper, a hole filling method with depth-guided global optimization is proposed for view synthesis. The global optimization is achieved by iterating the spatio-temporal approximate nearest neighbor (ANN) search and video reconstruction step. Directly applying global optimization might introduce some foreground artifacts to the synthesized video. To prevent this problem, some strategies have been developed in this paper. The depth information is applied to guide the spatio-temporal ANN searching and the initialization step is specified in the global optimization procedure. Our experimental results have demonstrated that the proposed method has better performance compared with other methods in terms of visual quality, trusted textures, and temporal consistency in the synthesized video.

Stylizing 3D Scene via Implicit Representation and HyperNetwork.

Abstract: In this work, we aim to address the 3D scene stylization problem - generating stylized images of the scene at arbitrary novel view angles A straightforward solution is to combine existing novel view synthesis and image/video style transfer approaches, which often leads to blurry results or inconsistent appearance Inspired by the high quality results of the neural radiance fields (NeRF) method, we propose a joint framework to directly render novel views with the desired style Our framework consists of two components: an implicit representation of the 3D scene with the neural radiance field model, and a hypernetwork to transfer the style information into the scene representation In particular, our implicit representation model disentangles the scene into the geometry and appearance branches, and the hypernetwork learns to predict the parameters of the appearance branch from the reference style image To alleviate the training difficulties and memory burden, we propose a two-stage training procedure and a patch sub-sampling approach to optimize the style and content losses with the neural radiance field model After optimization, our model is able to render consistent novel views at arbitrary view angles with arbitrary style Both quantitative evaluation and human subject study have demonstrated that the proposed method generates faithful stylization results with consistent appearance across different views

View synthesis of KDEX imagery for 3D security X-ray imaging

Abstract: We demonstrate the synthesis of intermediary views within a sequence of X-ray images that exhibit depth from motion or kinetic depth effect in a visual display. Each synthetic image replaces the requirement for a linear X-ray detector array during the image acquisition process. Scale invariant feature transform, SIFT, in combination with epipolar morphing is employed to produce synthetic imagery. Comparison between synthetic and ground truth images is reported to quantify the performance of the approach. Our work is a key aspect in the development of a new 3D imaging modality for the screening of luggage at airport checkpoints. This programme of research is in collaboration with the UK Home Office and the US Dept. of Homeland Security. (6 pages)

Dynamic View Synthesis From Dynamic Monocular Video

Abstract: We present an algorithm for generating novel views at arbitrary viewpoints and any input time step given a monocular video of a dynamic scene. Our work builds upon recent advances in neural implicit representation and uses continuous and differentiable functions for modeling the time-varying structure and the appearance of the scene. We jointly train a time-invariant static NeRF and a time-varying dynamic NeRF, and learn how to blend the results in an unsupervised manner. However, learning this implicit function from a single video is highly ill-posed (with infinitely many solutions that match the input video). To resolve the ambiguity, we introduce regularization losses to encourage a more physically plausible solution. We show extensive quantitative and qualitative results of dynamic view synthesis from casually captured videos.

Parallax View Generation for Static Scenes Using Parallax-Interpolation Adaptive Separable Convolution

Abstract: Reconstructing a Densely-Sampled Light Field (DSLF) from a Sparsely-Sampled Light Field (SSLF) is a challenging problem, for which various kinds of algorithms have been proposed. However, very few of them treat the angular information in a light field as the temporal information of a video from a virtual camera, i.e. the parallax views of a SSLF for a static scene can be turned into the key frames of a video captured by a virtual camera moving along the parallax axis. To this end, in this paper, a novel parallax view generation method, Parallax-Interpolation Adaptive Separable Convolution (PIASC), is proposed. The presented PIASC method takes full advantage of the motion coherence of static objects captured by a SSLF device to enhance the motion-sensitive convolution kernels of a state-of-the-art video frame interpolation method, i.e. Adaptive Separable Convolution (AdaSep-Conv). Experimental results on three development datasets of the grand challenge demonstrate the superior performance of PIASC for DSLF reconstruction of static scenes.