Keywords: LTS3 Reference LTS-ARTICLE-2004-019 Record created on 2006-06-14, modified on 2016-08-08

Introduction to the special issue on multimedia implementation », IEEE Trans. On Circuits and Systems for Video Technology

In this paper we will present the concept of a system that allows for an evolutionary introduction of depth perception into the existing 2D digital TV framework. The work is part of the European Information Society Technologies (IST) project “Advanced Three-Dimensional Television System Technologies” (ATTEST), an activity where industries, research centers and universities have joined forces to design a backwardscompatible, flexible and modular broadcast 3D-TV system, where all parts of the 3D processing chain are optimised to one another. This includes content creation, coding and transmission, display and research in human 3D perception, which be will used to guide the development process. The goals of the project comprise the development of a novel broadcast 3D camera, algorithms to convert existing 2D-video material into 3D, a 2Dcompatible coding and transmission scheme for 3D-video using MPEG2/4/7 technologies and the design of two new autostereoscopic displays.

IST project ATTEST - an evolutionary and optimised approach on 3D-TV

Depth Image Based Rendering (DIBR) technique has been recognized as a promising tool for supporting advanced 3D video services required in MultiView Video (MVV) systems. However, an inherent problem with DIBR is to fill newly exposed areas (holes) caused by disocclusions. This paper addresses the disocclusion problem. To deal with small disocclusions, hole-filling strategies have been designed by the state-of-the-art through pre-processing techniques of the depth video. For larger disocclusions, where depth pre-processing has some limitations, we propose an inpainting approach to retrieve missing pixels. Specifically, we propose in the texture and structure propagation process to take into account the depth information by distinguishing foreground and background parts of the scene. Experimental results illustrate the efficiency of the proposed method.

Depth-aided image inpainting for novel view synthesis

In three-dimensional (3D) video coding, distortion in texture video and depth maps can all affect the quality of the synthesized virtual views. Therefore, under the total bitrate constraint, effective bit allocation between texture and depth information is very important for 3D video coding. In this paper, the major technical contribution is to formulate view synthesis quality for optimal resource allocation in 3D video coding, since such quality is what that matters most to the ultimate user (i.e., the viewer) of the system; to be more specific, a new joint bit allocation and rate control method for multi-view video plus depth (MVD) based 3D video coding is proposed accordingly. We firstly derive a view synthesis distortion model to characterize the effect of coding distortion of texture video and depth maps on the synthesized virtual views. Based on this model, we derive a rate-distortion model to characterize the relationship between the bitrate and the view synthesis distortion, and the optimal bitrate ratio between texture and depth is established adaptively by solving the associated optimization problem. Finally, the rate control algorithm is performed on view level, texture/depth level and frame level. Experimental results show that compared with other methods, the proposed bit allocation method obtains higher performance of view synthesis. Moreover, the proposed rate control method can accurately control the bitrate to satisfy the total bitrate constraint.

Joint Bit Allocation and Rate Control for Coding Multi-View Video Plus Depth Based 3D Video

Free viewpoint video (FVV) has received considerable attention owing to its widespread applications in several areas such as immersive entertainment, remote surveillance and distanced education. Since FVV images are synthesized via a depth image-based rendering (DIBR) procedure in the “blind” environment (without reference images), a real-time and reliable blind quality assessment metric is urgently required. However, the existing image quality assessment metrics are insensitive to the geometric distortions engendered by DIBR. In this research, a novel blind method of DIBR-synthesized images is proposed based on measuring geometric distortion, global sharpness and image complexity. First, a DIBR-synthesized image is decomposed into wavelet subbands by using discrete wavelet transform. Then, the Canny operator is employed to detect the edges of the binarized low-frequency subband and high-frequency subbands. The edge similarities between the binarized low-frequency subband and high-frequency subbands are further computed to quantify geometric distortions in DIBR-synthesized images. Second, the log-energies of wavelet subbands are calculated to evaluate global sharpness in DIBR-synthesized images. Third, a hybrid filter combining the autoregressive and bilateral filters is adopted to compute image complexity. Finally, the overall quality score is derived to normalize geometric distortion and global sharpness by the image complexity. Experiments show that our proposed quality method is superior to the competing reference-free state-of-the-art DIBR-synthesized image quality models.

Blind Quality Metric of DIBR-Synthesized Images in the Discrete Wavelet Transform Domain

The efficient compression of multi-view-video-plus-depth (MVD) data raises the bit-rate allocation issue for the compression of texture and depth data. This question has not been solved yet because not all surveys reckon on a shared framework. This paper studies the impact of bit-rate allocation for texture and depth data relying on the quality of an intermediate synthesized view. The results show that depending on the acquisition configuration, the synthesized views require a different ratio between the depth and texture bit-rate: between 40% and 60% of the total bit-rate should be allocated to depth.

/pdf/bit-rate-allocation-for-multi-view-video-plus-depth-3rycryy89o.pdf

Bit-rate allocation for multi-view video plus depth

Layered Depth Image (LDI) representations are attractive compact representations for multi-view videos. Any virtual viewpoint can be rendered from LDI by using view synthesis technique. However, rendering from classical LDI leads to annoying visual artifacts, such as cracks and disocclusions. Visual quality gets even worse after a DCT-based compression of the LDI, because of blurring effects on depth discontinuities. In this paper, we propose a novel object-based LDI representation, improving synthesized virtual views quality, in a rate-constrained context. Pixels from each LDI layer are reorganised to enhance depth continuity.

/pdf/object-based-layered-depth-images-for-improved-virtual-view-k4p1k1ta22.pdf

Object-based Layered Depth Images for improved virtual view synthesis in rate-constrained context

This paper addresses the disocclusion problem which may occur when using Depth-Image-Based Rendering (DIBR) techniques in 3DTV and Free-Viewpoint TV applications. A new DIBR technique is proposed, which combines three methods: a Joint Projection Filling (JPF) method to handle disocclusions in synthesized depth maps; a backward projection to synthesize virtual views; and a full-Z depth-aided inpainting to fill in disoccluded areas in textures. The JPF method performs the pixels warping for virtual depth map synthesis while making use of an occlusion-compatible pixel ordering strategy, to detect cracks and disocclusions, and to select the pixels to be propagated in the occlusion areas filling process. The full-Z depth-aided inpainting method fills in disocclusions with textures at the correct depth, preserving the boundaries of the objects. Ghosting artifacts, which might otherwise result from pixel projections, are here avoided by introducing a confidence measure on background pixels to be used in the JPF process.

https://hal.archives-ouvertes.fr/hal-00628019/document

Joint projection filling method for occlusion handling in Depth-Image-Based Rendering

This paper describes an Incremental algorithm for Layer Depth Image construction (I-LDI) from multi-view plus depth data sets. A solution to sampling artifacts is proposed, based on pixel interpolation (inpainting) restricted to isolated unknown pixels. A solution to ghosting artifacts is also proposed, based on a depth discontinuity detection, followed by a local foreground / background classification. We propose a formulation of warping equations which reduces time consumption, specifically for LDI warping. Tests on Breakdancers and Ballet MVD data sets show that extra layers in I-LDI contain only 10% of first layer pixels, compared to 50% for LDI. I-LDI Layers are also more compact, with a less spread pixel distribution, and thus easier to compress than LDI Visual rendering is of similar quality with I-LDI and LDI.

/pdf/incremental-ldi-for-multi-view-coding-4971w7e110.pdf

Incremental-LDI for multi-view coding

Multi-view video plus depth (MVD) data offer a reliable representation of three-dimensional (3D) scenes for 3D video applications. This is a huge amount of data whose compression is an important challenge for researchers at the current time. Consisting of texture and depth video sequences, the question of the relationship between these two types of data regarding bit-rate allocation often raises. This paper questions the required ratio between texture and depth when encoding MVD data. In particular, the paper investigates the elements impacting on the best bit-rate ratio between depth and color: total bit-rate budget, input data features, encoding strategy, and assessed view.

/pdf/a-study-of-depth-texture-bit-rate-allocation-in-multi-view-591mjwv2nf.pdf

Vincent Jantet

Papers

Bit-rate allocation for multi-view video plus depth

Object-based Layered Depth Images for improved virtual view synthesis in rate-constrained context

Joint projection filling method for occlusion handling in Depth-Image-Based Rendering

Incremental-LDI for multi-view coding

A study of depth/texture bit-rate allocation in multi-view video plus depth compression