Depth Coding using Downsampling and View based Upsampling
01 Jan 2011-The Journal of The Institute of Image Information and Television Engineers-Vol. 65, Iss: 4, pp 497-504
TL;DR: The proposed depth resampling filter, used in combination with a standard state-of-the-art video encoder, can increase both the coding efficiency and rendering quality, particularly at lower bit rates.
Abstract: There are several ways to represent 3D scene information. One popular way is based on N-view plus N-depth representation. In applications based on this representation, efficient compression of both view and depth is important. In this paper, we present a depth coding method that uses depth downsampling and a novel depth upsample filter that uses the color view in the depth upsampling. Our method of depth down- and up-sampling is able to reconstruct clear object boundaries in the upsampled depth maps, and, therefore, we can obtain a better coding efficiency. Our experimental results show that the proposed depth resampling filter, used in combination with a standard state-of-the-art video encoder, can increase both the coding efficiency and rendering quality, particularly at lower bit rates.
Citations
More filters
TL;DR: This paper investigated several cases where one or both of the views are transmitted only in the low resolution during the delay time, and proposed adequate view synthesis method for multi resolution multi-view video plus depth.
Abstract: Streaming application of multi-view and free-viewpoint video is potentially attractive but due to the limitation of bandwidth, transmitting all multi-view video in high resolution may not be feasible. Our goal is to propose a new streaming data format that can be adapted to the limited bandwidth and capable of free-viewpoint video streaming using multi-view video plus depth (MVD). Given a requested free-viewpoint, we use the two closest views and corresponding depth maps to perform free-viewpoint video synthesis. We propose a new data format that consists of all views and corresponding depth maps in a lowered resolution, and the two closest views to the requested viewpoint in the high resolution. When the requested viewpoint changes, the two closest viewpoints will change, but one or both of views are transmitted only in the low resolution during the delay time. Therefore, the resolution compensation is required. In this paper, we investigated several cases where one or both of the views are transmitted only in the low resolution. We proposed adequate view synthesis method for multi resolution multi-view video plus depth. Experimental results show that our framework achieves view synthesis quality close to high resolution multi-view video plus depth.
4 citations
01 May 2015
TL;DR: Experimental results show that the proposed framework achieves view synthesis quality close to view synthesis using high resolution multi-view video plus depth using high Resolution Multi-view Video plus depth (MVD).
Abstract: Our goal is to propose and evaluate a new streaming data format that can be adapted to the limited bandwidth and capable of free-viewpoint video streaming using super multi-view video plus depth (MVD). Additionally, we proposed a view synthesis method for our data format. Given a requested free-viewpoint, we use the two closest views and corresponding depth maps to perform free-viewpoint video synthesis. The new data format consists of all views and corresponding depth maps in a lowered resolution, and the two closest views to the requested viewpoint in the high resolution. When the requested viewpoint changes, the two closest viewpoints will change, but one or both views are transmitted only in the low resolution during periods of large round-trip delay time. Therefore, the resolution compensation is required before view synthesis. Experimental results show that our proposed framework achieves view synthesis quality close to view synthesis using high resolution multi-view video plus depth.
References
More filters
01 Aug 2004
TL;DR: This paper shows how high-quality video-based rendering of dynamic scenes can be accomplished using multiple synchronized video streams combined with novel image-based modeling and rendering algorithms, and develops a novel temporal two-layer compressed representation that handles matting.
Abstract: The ability to interactively control viewpoint while watching a video is an exciting application of image-based rendering. The goal of our work is to render dynamic scenes with interactive viewpoint control using a relatively small number of video cameras. In this paper, we show how high-quality video-based rendering of dynamic scenes can be accomplished using multiple synchronized video streams combined with novel image-based modeling and rendering algorithms. Once these video streams have been processed, we can synthesize any intermediate view between cameras at any time, with the potential for space-time manipulation.In our approach, we first use a novel color segmentation-based stereo algorithm to generate high-quality photoconsistent correspondences across all camera views. Mattes for areas near depth discontinuities are then automatically extracted to reduce artifacts during view synthesis. Finally, a novel temporal two-layer compressed representation that handles matting is developed for rendering at interactive rates.
1,677 citations
TL;DR: Details of a system that allows for an evolutionary introduction of depth perception into the existing 2D digital TV framework are presented and a comparison with the classical approach of "stereoscopic" video is compared.
Abstract: This paper presents details of a system that allows for an evolutionary introduction of depth perception into the existing 2D digital TV framework. The work is part of the European Information Society Technologies (IST) project “Advanced Three-Dimensional Television System Technologies” (ATTEST), an activity, where industries, research centers and universities have joined forces to design a backwards-compatible, flexible and modular broadcast 3D-TV system. At the very heart of the described new concept is the generation and distribution of a novel data representation format, which consists of monoscopic color video and associated perpixel depth information. From these data, one or more “virtual” views of a real-world scene can be synthesized in real-time at the receiver side (i. e. a 3D-TV set-top box) by means of so-called depth-image-based rendering (DIBR) techniques. This publication will provide: (1) a detailed description of the fundamentals of this new approach on 3D-TV; (2) a comparison with the classical approach of “stereoscopic” video; (3) a short introduction to DIBR techniques in general; (4) the development of a specific DIBR algorithm that can be used for the efficient generation of high-quality “virtual” stereoscopic views; (5) a number of implementation details that are specific to the current state of the development; (6) research on the backwards-compatible compression and transmission of 3D imagery using state-of-the-art MPEG (Moving Pictures Expert Group) tools.
1,560 citations
TL;DR: A new window-based method for correspondence search using varying support-weights based on color similarity and geometric proximity to reduce the image ambiguity and outperforms other local methods on standard stereo benchmarks.
Abstract: We present a new window-based method for correspondence search using varying support-weights. We adjust the support-weights of the pixels in a given support window based on color similarity and geometric proximity to reduce the image ambiguity. Our method outperforms other local methods on standard stereo benchmarks.
1,267 citations
TL;DR: A real-time FTV is constructed including the complete chain from capturing to display, and a new algorithm was developed to generate free viewpoint images to create ray-based image engineering through the development of FTV.
Abstract: We have been developing ray-based 3D information systems that consist of ray acquisition, ray processing, and ray display. Free viewpoint television (FTV) based on the ray-space method is a typical example. FTV will bring an epoch-making change in the history of television because it enables us to view a distant 3D world freely by changing our viewpoints as if we were there. We constructed a real-time FTV including the complete chain from capturing to display. A new algorithm was developed to generate free viewpoint images. In addition, a new user interface is presented for FTV to make full use of 3D information. FTV is not a pixel-based system but a ray-based system. We are creating ray-based image engineering through the development of FTV.
261 citations
01 Nov 2002
TL;DR: In this paper, a free-viewpoint TV system based on Ray-Space representation is described. But the work is limited to a single camera and a PC cluster with 16 PCs for data processing.
Abstract: This paper describes a novel Free-Viewpoint TV system based on Ray-Space representation. This system consists of a multi-view camera system for 3-D data capturing, a PC cluster with 16 PCs for data processing such as data compression and view interpolation, input device to specify a viewpoint, and a conventional 2-D display to show an arbitrary viewpoint image. To generate an arbitrary viewpoint image, the Ray-Space method is used. First, the multi-view image is converted to the Ray-Space data. Then, interpolation of the Ray-Space using adaptive filter is applied. Finally, an arbitrary view image is generated from the interpolated dense Ray-Space. This paper also describes various compression methods, such as model-based compression, arbitrary-shaped DCT(Discrete Cosine Transform), VQ(Vector Quantization), and subband coding. Finally, a demonstration of a full real-time system from capturing to display is explained.© (2002) COPYRIGHT SPIE--The International Society for Optical Engineering. Downloading of the abstract is permitted for personal use only.
138 citations