View synthesis

About: View synthesis is a research topic. Over the lifetime, 1701 publications have been published within this topic receiving 42333 citations.

Spatio-temporal consistent depth-image-based rendering using layered depth image and inpainting

Abstract: Depth-image-based rendering (DIBR) is a commonly used method for synthesizing additional views using video-plus-depth (V+D) format. A critical issue with DIBR-based view synthesis is the lack of information behind foreground objects. This lack is manifested as disocclusions, holes, next to the foreground objects in rendered virtual views as a consequence of the virtual camera “seeing” behind the foreground object. The disocclusions are larger in the extrapolation case, i.e. the single camera case. Texture synthesis methods (inpainting methods) aim to fill these disocclusions by producing plausible texture content. However, virtual views inevitably exhibit both spatial and temporal inconsistencies at the filled disocclusion areas, depending on the scene content. In this paper, we propose a layered depth image (LDI) approach that improves the spatio-temporal consistency. In the process of LDI generation, depth information is used to classify the foreground and background in order to form a static scene sprite from a set of neighboring frames. Occlusions in the LDI are then identified and filled using inpainting, such that no disocclusions appear when the LDI data is rendered to a virtual view. In addition to the depth information, optical flow is computed to extract the stationary parts of the scene and to classify the occlusions in the inpainting process. Experimental results demonstrate that spatio-temporal inconsistencies are significantly reduced using the proposed method. Furthermore, subjective and objective qualities are improved compared to state-of-the-art reference methods.

A study on the relationship between depth map quality and the overall 3D video quality of experience

Abstract: The emergence of multiview displays has made the need for synthesizing virtual views more pronounced, since it is not practical to capture all of the possible views when filming multiview content. View synthesis is performed using the available views and depth maps. There is a correlation between the quality of the synthesized views and the quality of depth maps. In this paper we study the effect of depth map quality on perceptual quality of synthesized view through subjective and objective analysis. Our evaluation results show that: 1) 3D video quality depends highly on the depth map quality and 2) the Visual Information Fidelity index computed between the reference and distorted depth maps has Pearson correlation ratio of 0.75 and Spearman rank order correlation coefficient of 0.67 with the subjective 3D video quality.

Light Field View Synthesis via Aperture Disparity and Warping Confidence Map

Abstract: This paper presents a learning-based approach to synthesize the view from an arbitrary camera position given a sparse set of images. A key challenge for this novel view synthesis arises from the reconstruction process, when the views from different input images may not be consistent due to obstruction in the light path. We overcome this by jointly modeling the epipolar property and occlusion in designing a convolutional neural network. We start by defining and computing the aperture disparity map, which approximates the parallax and measures the pixel-wise shift between two views. While this relates to free-space rendering and can fail near the object boundaries, we further develop a warping confidence map to address pixel occlusion in these challenging regions. The proposed method is evaluated on diverse real-world and synthetic light field scenes, and it shows better performance over several state-of-the-art techniques.

R-D optimized auxiliary information for inpainting-based view synthesis

Abstract: Texture and depth maps of two neighboring camera viewpoints are typically required for synthesis of an intermediate virtual view via depth-image-based rendering (DIBR). However, the bitrate overhead required for reconstruction ofmultiple texture and depthmaps at decoder can be large. The performance of multiview video encoders such as MVC is limited by the simple fact that the chosen representation is inherently redundant: a texture or depth pixel visible from both camera viewpoints is represented twice. In this paper, we propose an alternative 3D scene representation without such redundancy, yet at decoder, one can still reconstruct texture and depth maps of two camera viewpoints for DIBR-based synthesis of intermediate views. In particular, we propose to first encode texture and depth videos of a single viewpoint, which are used to synthesize the uncoded viewpoint via DIBR at decoder. Then, we encode additional rate-distortion (RD) optimal auxiliary information (AI) to guide an inpainting-based hole-filling algorithm at decoder and complete the missing information due to disocclusion. For a missing pixel patch in the synthesized view, the AI can: i) be skipped and then let the decoder by itself retrieve the missing information, ii) identify a suitable spatial region in the reconstructed view for patch-matching, or iii) explicitly encode missing pixel patch if no satisfactory patch can be found in the reconstructed view. Experimental results show that our alternative representation can achieve up to 41% bit-savings compared to H.264/MVC implementation.

Gaze-corrected view generation using stereo camera system for immersive videoconferencing

Abstract: In this paper, we present an immersive videoconferencing system that enables gaze correction between users in the internet protocol TV (IPTV) environment. After we capture the object using stereo cameras, we perform preprocessing techniques, such as camera calibration, color correction, and image rectification. The preprocessed images are down-sampled and disparities are computed by using the downsampled images. The disparity sequence is then filtered to improve temporal consistency. After central view synthesis, occlusion areas are decided and holes are filled. The entire system is implemented with parallel programming that is executed on the GPU for realtime processing. Finally, the user can observe the gaze-corrected image through display. From experimental results, we have verified that the proposed stereo camera system is sufficient to generate the natural gaze-corrected virtual image and realize immersive videoconferencing.