Current 3-D video (3DV) technology is based on stereo systems. These systems use stereo video coding for pictures delivered by two input cameras. Typically, such stereo systems only reproduce these two camera views at the receiver and stereoscopic displays for multiple viewers require wearing special 3-D glasses. On the other hand, emerging autostereoscopic multiview displays emit a large numbers of views to enable 3-D viewing for multiple users without requiring 3-D glasses. For representing a large number of views, a multiview extension of stereo video coding is used, typically requiring a bit rate that is proportional to the number of views. However, since the quality improvement of multiview displays will be governed by an increase of emitted views, a format is needed that allows the generation of arbitrary numbers of views with the transmission bit rate being constant. Such a format is the combination of video signals and associated depth maps. The depth maps provide disparities associated with every sample of the video signal that can be used to render arbitrary numbers of additional views via view synthesis. This paper describes efficient coding methods for video and depth data. For the generation of views, synthesis methods are presented, which mitigate errors from depth estimation and coding.

3-D Video Representation Using Depth Maps

Free-viewpoint television (FTV) is an innovative visual media that enables us to view a three-dimensional (3-D) scene by freely changing our viewpoints. We proposed the concept of FTV and constructed the world???s first real-time system including the complete chain of operation from image capture to display. We also carried out the FTV on a single personal computer (PC) and a mobile player. FTV is based on the ray-space method that represents one ray in real space with one point in the ray-space. We have developed several types of ray capture systems and interfaces such as a 360° capture/ray-reproducing display. FTV is regarded as the ultimate 3DTV, since it can generate infinite number of views. Thus, FTV is the key to immersive communication. Regarding FTV as the most challenging 3-D media, the Motion Picture Experts Group (MPEG) has been conducting its international standardization activities. This article reviews FTV and its related technologies.

Free-Viewpoint TV

FTV (Free-viewpoint Television) enables us to view a 3D scene by freely changing our viewpoint. FTV is the ultimate 3DTV with infinite number of views and ranked as the top of visual media. We realized FTV by developing various types of ray capture, processing and display technologies. We have also realized all-around ray-reproducing 3DTV. The international standardization of FTV has been conducted in MPEG. The first phase of FTV was MVC (Multi-view Video Coding) and the second phase is 3DV (3D Video).

FTV: Free-viewpoint Television

Free-viewpoint TV (FTV) is cutting the frontier of audiovisual communications. FTV is an innovative media that enables us to view 3-D space by freely changing our viewpoints. It also allows us to listen at any listening point in the 3-D space. Since FTV transmits all audiovisual information of the 3-D space, it can reconstruct an audiovisual replica of the 3-D space anywhere and anytime over distance and time. For video, FTV captures a part of rays in 3-D space by using many cameras, and the other rays that are not captured are obtained by interpolating the captured rays. We constructed real-time FTV systems including the complete chain of operation from image capture to display. We also carried out FTV on a laptop computer and a mobile player. For audio, two kinds of free listening-point systems are demonstrated. MPEG regarded FTV as the most challenging 3-D media and has been conducting its international standardization activities. The first phase of FTV was multiview video coding (MVC) and the second phase of FTV is 3-D video (3DV). MVC enables the efficient coding of multiple camera views and was completed in 2009. MVC has been adopted by Blu-ray 3-D. 3DV is a standard that targets serving a variety of 3-D displays and its call for proposals was issued in March 2011.

FTV for 3-D Spatial Communication

Systems and methods for generating pixel based depth estimates are disclosed. An image processing system operating as depth analysis engine generates an estimated depth associated with a pixel based on a reference image and other related images. A current depth estimate is refined based on neighboring pixels and calculated consistency scores. Further, depth estimates can be levered in object or scene recognition to trigger or initiate an action taken by a computing device.

Depth estimate determination, systems and methods

Artifact reduction using reliability reasoning for image generation of FTV

Provided is a virtual viewpoint image synthesizing method in which a virtual viewpoint image viewed from a virtual viewpoint is synthesized based on image information obtained from a plurality of viewpoints. The virtual viewpoint image is synthesized through a reference images obtaining step, a depth maps generating step, an up-sampling step, a virtual viewpoint information obtaining step, and a virtual viewpoint image synthesizing step.

Virtual viewpoint image synthesizing method and virtual viewpoint image synthesizing system

View synthesis using depth maps is an important application in 3D image processing. In this paper, a novel method is proposed for the plausible view synthesis of Free-viewpoint TV (FTV), using two input images and their depth maps. The depth estimation based on stereo matching is known to be error-prone, leading to noticeable artifacts in the synthesized new views. To produce high-quality view synthesis, we introduce a probabilistic framework which constrains the reliability of each pixel of new view by Maximizing Likelihood (ML). The spatial adaptive reliability is provided by incorporating Gamma hyper-prior and the synthesis error approximation. Furthermore, we generate the virtual view by solving a Maximum a Posterior (MAP) problem using graph cuts. We compare the proposed method with other depth based view synthesis approaches on MPEG test sequences. The results show the outperformance of our method both at subjective artifacts reduction and objective PSNR improvement.

Probabilistic reliability based view synthesis for FTV

View synthesis using depth maps is a crucial application for Free-viewpoint TV (FTV). In this paper, we propose a novel reliability based view synthesis method using two references and their depth maps. The depth estimation with stereo matching is known to be error-prone, leading to noticeable artifacts in the synthesized new views. In order to provide plausible virtual views for FTV, our focus is on the error suppression for the synthesized view. We innovatively introduce the continuous reliability using error approximation by the reference cross-check. The new view interpolation algorithm is generated with the criterion of Least Sum of Squared Errors (LSSE). Furthermore, the proposed algorithm can be considered as a reliable version of the conventional linear view blending. We experimentally demonstrate the effectiveness of our framework with MPEG standard test sequences. The results show that our method outperforms state-of-the-art view interpolation methods both at eliminating artifacts and improving PSNR.

Error supression in view synthesis using reliability reasoning for FTV

View synthesis using depth maps is a well-known technique for exploiting the redundancy between multi-view videos. In this paper, we deal with the bitrates of view synthesis at the decoder side of FTV that would use compressed depth maps and views. Both inherent depth estimation error and coding distortion would degrade synthesis quality. The focus is to reduce bitrates required for generating the high-quality virtual view. We employ a reliable view synthesis method which is compared with standard MPEG view synthesis software. The experimental results show that the bitrates required for synthesizing high-quality virtual view could be reduced by utilizing our enhanced view synthesis technique to improve the PSNR at medium bitrates.

Lu Yang

Papers

Artifact reduction using reliability reasoning for image generation of FTV

Virtual viewpoint image synthesizing method and virtual viewpoint image synthesizing system

Probabilistic reliability based view synthesis for FTV

Error supression in view synthesis using reliability reasoning for FTV

Reducing bitrates of compressed video with enhanced view synthesis for FTV