There is disclosed a method, apparatus and computer program product in which at least one view component of a first type and at least one view component of a second type are obtained. The order of the texture view component and the depth view component in an access unit is determined and at least one indication of the order is encoded. The coding of the view components is adapted on the basis of the order. There is also disclosed a method, apparatus and computer program product in which at least one encoded view component of a first type and at least one encoded view component of a second type are received. Also at least one encoded indication of the order of the view components is received. The at least one encoded indication is decoded and the decoding of the view component sis adapted on the basis of the order.

Method and apparatus for video coding

The purpose of the present invention is to enable correct stream reception to take place at the receiving end, in a manner accurately adapted to dynamic changes in delivered content. In a first transmission mode, a predetermined number of extended video streams constituting scalable-coded image data, and respectively including a basic video stream that includes lowest-layer image data, and image data of layers other than the lowest layer. In a second transmission mode, only the basic video stream that includes basic image data is transmitted. Identifying information for identifying the first mode and the second mode is inserted into the video stream. At the receiving end, the transmission mode can be easily ascertained, making it possible for correct stream reception to take place in a manner accurately adapted to changes in the stream composition, that is, dynamic changes in delivered content.

Transmission device, transmission method, reception device, and reception method

This paper describes a new algorithm for depth image super resolution and denoising using a single depth image as input. A robust coupled dictionary learning method with locality coordinate constraints is introduced to reconstruct the corresponding high resolution depth map. The local constraints effectively reduce the prediction uncertainty and prevent the dictionary from over-fitting. We also incorporate an adaptively regularized shock filter to simultaneously reduce the jagged noise and sharpen the edges. Furthermore, a joint reconstruction and smoothing framework is proposed with an L0 gradient smooth constraint, making the reconstruction more robust to noise. Experimental results demonstrate the effectiveness of our proposed algorithm compared to previously reported methods.

Joint Super Resolution and Denoising From a Single Depth Image

The 3D extension of High Efficiency Video Coding significantly improves the coding efficiency of 3D video at the expense of computational complexity. This paper presents a novel fast mode decision algorithm for depth map coding based on the grayscale similarity and inter-view correlation. First, depth map grayscale similarity is adopted to judge whether the reference frame could assist the coding of the current frame. When the difference in the average grayscale between the co-located coding unit (CU) and the current CU is smaller than the similarity threshold, the depth level of the current CU will be restricted by that of the coded reference CU. Second, the grayscale similarity and inter-view correlation are jointly used for dependent views to achieve early decision on the best prediction unit (PU) mode. The mode decision procedure will be determined early when the co-located CU, which has a grayscale similarity with the current CU, selects Merge or Inter $2N \times 2N$ as the best prediction mode. Moreover, when the corresponding CU in the independent view selects Merge or Inter $2N \times 2N$ as the best prediction mode, the current CU will skip other PU modes checking based on the strong inter-view correlation. Finally, different strategies are proposed for the P-frames and B-frames of dependent views in view of the characteristics of different prediction structures. For B frames, the PU mode information of the coded independent view is utilized as reference to skip the unnecessary mode decision processes. For P frames, the spatial–temporal correlation is considered in the process of early mode decision to determine whether to choose the Merge mode or Inter $2N \times 2N$ as the best mode. Experimental results show that our proposed scheme achieves considerable time saving with negligible degradation of coding performance.

Fast Mode Decision Based on Grayscale Similarity and Inter-View Correlation for Depth Map Coding in 3D-HEVC

An apparatus configured to code video information comprises a memory unit and a processor in communication with the memory unit. The memory unit is configured to store video information associated with a reference layer (RL) and an enhancement layer (EL). The EL comprises an EL video unit and the RL comprises an RL video unit corresponding to the EL video unit. The processor is configured to perform upsampling and bit-depth conversion on pixel information of the RL video unit in a single combined process to determine predicted pixel information of the EL video unit, and determine the EL video unit using the predicted pixel information.

Device and method for scalable coding of video information

Systems, methods, and instrumentaliiies are provided to implement video coding system (VCS). The VCS may be configured to receive a video signal, which may include one or more layers (e.g., a base layer (BL) and/or one or more enhancement layers (ELs)). The VCS may be configured to process a BL picture into an inter-layer reference (ILR) picture, e.g., using picture level inter-layer prediction process. The VCS may be configured to select one or both of the processed IL R picture or an enhancement layer (EL) reference picture. The selected reference picture(s) may comprise one of the EL reference picture, or the ILR picture. The VCS may be configured to predict a current EL picture using one or more of the selected ILR picture or the EL reference picture. The VCS may be configured to store the processed ILR picture in an EL decoded picture buffer (DPB).

Codec architecture for multiple layer video coding

In this paper we present a novel stereo view synthesis algorithm that is highly accurate with respect to inter-view consistency, thus to enabling stereo contents to be viewed on the autostereoscopic displays. The algorithm finds identical occluded regions within each virtual view and aligns them together to extract a surrounding background layer. The background layer for each occluded region is then used with an exemplar based inpainting method to synthesize all virtual views simultaneously.

Our algorithm requires the alignment and extraction of background layers for each occluded region; however, these two steps are done efficiently with lower computational complexity in comparison to previous approaches using the exemplar based inpainting algorithms. Thus, it is more efficient than existing algorithms that synthesize one virtual view at a time.

This paper also describes the implementation of a simplified GPU accelerated version of the approach and its implementation in CUDA. Our CUDA method has sublinear complexity in terms of the number of views that need to be generated, which makes it especially useful for generating content for autostereoscopic displays that require many views to operate.

An objective of our work is to allow the user to change depth and viewing perspective on the fly. Therefore, to further accelerate the CUDA variant of our approach, we present a modified version of our method to warp the background pixels from reference views to a middle view to recover background pixels. We then use an exemplar based inpainting method to fill in the occluded regions. We use warping of the foreground from the reference images and background from the filled regions to synthesize new virtual views on the fly. Our experimental results indicate that the simplified CUDA implementation decreases running time by orders of magnitude with negligible loss in quality.

On consistent inter-view synthesis for autostereoscopic displays

An embodiment of the invention is a method that provides common data and enforces multiple view consistency. In a preferred embodiment, a coordinate alignment matrix is generated after initial warped views are formed. The matrix is used efficiently to repair holes/occlusions when synthesizing final views. In another preferred embodiment, a common background layer is generated based upon initial warped views. Holes are repaired on the common background layer, which can then be used for synthesizing final views. The cost of the latter method is sub-linear, meaning that additional views can be generated with lesser cost. Preferred methods of the invention also use a placement matrix to generate initial warped views, which simplifies warping to a look-up operation.

Multi view synthesis method and display devices with spatial and inter-view consistency

Current trends in video technology indicate a significant increase in spatial and temporal resolution of video data. Recently, a linear-runtime feature diffusion algorithm was presented which aims for fast and accurate processing of such high resolution data. In this paper, we introduce this algorithm from the perspective of image-based depth estimation, expanding upon the algorithm by requiring interview consistency in the depth diffusion process. We also discuss different application scenarios and provide an in-depth analysis of the method in this context.

Depth estimation and depth enhancement by diffusion of depth features

In this paper, we propose a novel multiview video plus depth (MVD) codec that uses a depth-based prediction mode (DBPM) both for texture videos and the depth maps. We show that the proposed codec can provide up to 9.2%, 9.9%, and 6.7% bitrate savings over H.264/MVC (MVC) for coding MVD data, depth maps, and multiview videos, respectively. In addition, we present subjective test results that compare the perceptual video quality of stereo videos coded with the DBPM-enabled codec and MVC. We also show that a typical Lagrangian rate-distortion optimization is effective to successfully choose between the prediction modes available to the proposed encoder. We provide a complexity analysis for the proposed codec as well and show that the encoder complexity is comparable to MVC, while the decoder complexity is about twice of MVC. Finally, we analyze the effects of different encoder control parameters on the virtual view synthesis quality. Results show that video QP has the most influence on the synthesis quality, and there is a tradeoff between the depth map QP and resolution. Lower depth map resolutions yield more bitrate savings, whereas for similar bitrates higher resolution and higher QP depth maps achieve better view synthesis performance.

Can Bal

Papers

Codec architecture for multiple layer video coding

On consistent inter-view synthesis for autostereoscopic displays

Multi view synthesis method and display devices with spatial and inter-view consistency

Depth estimation and depth enhancement by diffusion of depth features

Multiview Video Plus Depth Coding With Depth-Based Prediction Mode