We present an efficient and simple method for introducing temporal consistency to a large class of optimization driven image-based computer graphics problems. Our method extends recent work in edge-aware filtering, approximating costly global regularization with a fast iterative joint filtering operation. Using this representation, we can achieve tremendous efficiency gains both in terms of memory requirements and running time. This enables us to process entire shots at once, taking advantage of supporting information that exists across far away frames, something that is difficult with existing approaches due to the computational burden of video data. Our method is able to filter along motion paths using an iterative approach that simultaneously uses and estimates per-pixel optical flow vectors. We demonstrate its utility by creating temporally consistent results for a number of applications including optical flow, disparity estimation, colorization, scribble propagation, sparse data up-sampling, and visual saliency computation.

Practical temporal consistency for image-based graphics applications

FTV (Free-viewpoint Television) enables us to view a 3D scene by freely changing our viewpoint. FTV is the ultimate 3DTV with infinite number of views and ranked as the top of visual media. We realized FTV by developing various types of ray capture, processing and display technologies. We have also realized all-around ray-reproducing 3DTV. The international standardization of FTV has been conducted in MPEG. The first phase of FTV was MVC (Multi-view Video Coding) and the second phase is 3DV (3D Video).

FTV: Free-viewpoint Television

Systems and methods for generating pixel based depth estimates are disclosed. An image processing system operating as depth analysis engine generates an estimated depth associated with a pixel based on a reference image and other related images. A current depth estimate is refined based on neighboring pixels and calculated consistency scores. Further, depth estimates can be levered in object or scene recognition to trigger or initiate an action taken by a computing device.

Depth estimate determination, systems and methods

A “Point Cloud Smoother” provides various techniques for refining a 3D point cloud or other 3D input model to generate a smoothed and denoised 3D output model. Smoothing and denoising is achieved, in part, by robustly fitting planes to a neighborhood of points around each point of the input model and using those planes to estimate new points and corresponding normals of the 3D output model. These techniques are useful for a number of purposes, including, but not limited to, free viewpoint video (FVV), which, when combined with the smoothing techniques enabled by the Point Cloud Smoother, allows 3D data of videos or images to be denoised and then rendered and viewed from any desired viewpoint that is supported by the input data.

Smoothing and robust normal estimation for 3D point clouds

FTV (free-viewpoint television) is an innovative visual media that allows users to view a three-dimensional (3D) scene by freely changing their viewpoints. Thus, it enables realistic viewing and free navigation of 3D scenes. FTV is the ultimate 3DTV with infinite number of views and ranked at the top of visual media. FTV is not a conventional pixel-based system but a ray-based system. New types of ray capture, processing, and display technologies have been developed for FTV. These technologies were also used to realize an all-around ray-reproducing 3DTV. The international standardization of FTV has been promoted in MPEG. The first phase of FTV ismulti-view video coding and the second phase is 3D video. In this paper, the FTV system and its technologies are reviewed.

/pdf/ftv-free-viewpoint-television-5cyvj3vm0v.pdf

FTV (free-viewpoint television)

3D scene information can be represented in several ways. In applications based on a (N-)view plus (N-)depth representation, both view and depth data is compressed. In this paper we present a depth compression method containing an depth up-sample filter which uses the color view as prior. Our method of depth down-/up-sampling is able to maintain clear object boundaries in the reconstructed depth maps. Our experimental results show that the proposed depth re-sampling filter, used in combination with a standard state-of-the art video encoder, can increase both the coding efficiency and rendering quality.

Color based depth up-sampling for depth compression

Provided is a virtual viewpoint image synthesizing method in which a virtual viewpoint image viewed from a virtual viewpoint is synthesized based on image information obtained from a plurality of viewpoints. The virtual viewpoint image is synthesized through a reference images obtaining step, a depth maps generating step, an up-sampling step, a virtual viewpoint information obtaining step, and a virtual viewpoint image synthesizing step.

Virtual viewpoint image synthesizing method and virtual viewpoint image synthesizing system

In this paper, we propose a semi-automatic depth estimation algorithm whereby the user defines object depth boundaries
and disparity initialization. Automatic depth estimation methods generally have difficulty to obtain good depth results
around object edges and in areas with low texture. The goal of our method is to improve the depth in these areas and
reduce view synthesis artifacts in Depth Image Based Rendering. Good view synthesis quality is very important in
applications such as 3DTV and Free-viewpoint Television (FTV). In our proposed method, initial disparity values for
smooth areas can be input through a so-called manual disparity map, and depth boundaries are defined by a manually
created edge map which can be supplied for one or multiple frames. For evaluation we used MPEG multi-view videos
and we demonstrate our algorithm can significantly improve the depth maps and reduce view synthesis artifacts.

A semi-automatic multi-view depth estimation method

In 3DTV and free viewpoint imaging systems based on a view-plus-depth representation, depth compression is important for high-quality view synthesis. Several publications have proposed depth down-/up-sampling as part of the depth coding strategy. Recently, we proposed a depth up-sampling method which uses the high resolution view in the process of depth up-sampling. Actually, in 2007 Kopf et.al. already proposed a Joint Bilateral Upsampler(JBU), which uses a high resolution input image as prior in a joint bilateral filter for upscaling. In this paper we compare our previous method with the JBU approach, in the context of depth coding. Furthermore, we analyze the influence of view coding on the depth up-sampling result.

Depth up-sampling for depth coding using view information

View synthesis using depth maps is a well-known technique for exploiting the redundancy between multi-view videos. In this paper, we deal with the bitrates of view synthesis at the decoder side of FTV that would use compressed depth maps and views. Both inherent depth estimation error and coding distortion would degrade synthesis quality. The focus is to reduce bitrates required for generating the high-quality virtual view. We employ a reliable view synthesis method which is compared with standard MPEG view synthesis software. The experimental results show that the bitrates required for synthesizing high-quality virtual view could be reduced by utilizing our enhanced view synthesis technique to improve the PSNR at medium bitrates.

Meindert Onno Wildeboer

Papers

Color based depth up-sampling for depth compression

Virtual viewpoint image synthesizing method and virtual viewpoint image synthesizing system

A semi-automatic multi-view depth estimation method

Depth up-sampling for depth coding using view information

Reducing bitrates of compressed video with enhanced view synthesis for FTV