Current 3-D video (3DV) technology is based on stereo systems. These systems use stereo video coding for pictures delivered by two input cameras. Typically, such stereo systems only reproduce these two camera views at the receiver and stereoscopic displays for multiple viewers require wearing special 3-D glasses. On the other hand, emerging autostereoscopic multiview displays emit a large numbers of views to enable 3-D viewing for multiple users without requiring 3-D glasses. For representing a large number of views, a multiview extension of stereo video coding is used, typically requiring a bit rate that is proportional to the number of views. However, since the quality improvement of multiview displays will be governed by an increase of emitted views, a format is needed that allows the generation of arbitrary numbers of views with the transmission bit rate being constant. Such a format is the combination of video signals and associated depth maps. The depth maps provide disparities associated with every sample of the video signal that can be used to render arbitrary numbers of additional views via view synthesis. This paper describes efficient coding methods for video and depth data. For the generation of views, synthesis methods are presented, which mitigate errors from depth estimation and coding.

3-D Video Representation Using Depth Maps

In this paper, we propose a new method of depth-image-based rendering (DIBR) for free-viewpoint TV (FTV). In the conventional method, we estimated the depth of an object on the virtual image plane, which is called view-dependent depth estimation, and the virtual view images are rendered using the view-dependent depth map. In this method, virtual viewpoint images are rendered with 3D warping instead of estimating the view-dependent depth, since depth estimation is usually costly and it is desirable to eliminate it from the rendering process. However, 3D warping causes some problems that do not occur in the method with view-dependent depth estimation; for example, the appearance of holes on the rendered image, and the occurrence of depth discontinuity on the surface of the object at virtual image plane. Depth discontinuity causes artifacts on the rendered image. In this paper, these problems are solved by projecting depth map to the virtual image plane and performing post-filtering on the projected depth map. In the experiments, high-quality arbitrary viewpoint images were obtained by rendering images from relatively small number of cameras.

View generation with 3D warping using depth information for FTV

Free-viewpoint television (FTV) is an innovative visual media that enables us to view a three-dimensional (3-D) scene by freely changing our viewpoints. We proposed the concept of FTV and constructed the world???s first real-time system including the complete chain of operation from image capture to display. We also carried out the FTV on a single personal computer (PC) and a mobile player. FTV is based on the ray-space method that represents one ray in real space with one point in the ray-space. We have developed several types of ray capture systems and interfaces such as a 360° capture/ray-reproducing display. FTV is regarded as the ultimate 3DTV, since it can generate infinite number of views. Thus, FTV is the key to immersive communication. Regarding FTV as the most challenging 3-D media, the Motion Picture Experts Group (MPEG) has been conducting its international standardization activities. This article reviews FTV and its related technologies.

Free-Viewpoint TV

3DTV technology has brought out new challenges such as the question of synthesized views evaluation. Synthesized views are generated through a depth image-based rendering (DIBR) process. This process induces new types of artifacts whose impact on visual quality has to be identified considering various contexts of use. While visual quality assessment has been the subject of many studies in the last 20 years, there are still some unanswered questions regarding new technological improvement. DIBR is bringing new challenges mainly because it deals with geometric distortions. This paper considers the DIBR-based synthesized view evaluation problem. Different experiments have been carried out. They question the protocols of subjective assessment and the reliability of the objective quality metrics in the context of 3DTV, in these specific conditions (DIBR-based synthesized views), and they consist in assessing seven different view synthesis algorithms through subjective and objective measurements. Results show that usual metrics are not sufficient for assessing 3-D synthesized views, since they do not correctly render human judgment. Synthesized views contain specific artifacts located around the disoccluded areas, but usual metrics seem to be unable to express the degree of annoyance perceived in the whole image. This study provides hints for a new objective measure. Two approaches are proposed: the first one is based on the analysis of the shifts of the contours of the synthesized view; the second one is based on the computation of a mean SSIM score of the disoccluded areas.

/pdf/towards-a-new-quality-metric-for-3-d-synthesized-view-1csqdjmaru.pdf

Towards a New Quality Metric for 3-D Synthesized View Assessment

This paper presents a literature survey on existing disparity map algorithms. It focuses on four main stages of processing as proposed by Scharstein and Szeliski in a taxonomy and evaluation of dense two-frame stereo correspondence algorithms performed in 2002. To assist future researchers in developing their own stereo matching algorithms, a summary of the existing algorithms developed for every stage of processing is also provided. The survey also notes the implementation of previous software-based and hardware-based algorithms. Generally, the main processing module for a software-based implementation uses only a central processing unit. By contrast, a hardware-based implementation requires one or more additional processors for its processing module, such as graphical processing unit or a field programmable gate array. This literature survey also presents a method of qualitative measurement that is widely used by researchers in the area of stereo vision disparity mappings.

/pdf/literature-survey-on-stereo-vision-disparity-map-algorithms-3mtr9e2zyc.pdf

Literature Survey on Stereo Vision Disparity Map Algorithms

Free viewpoint images can be generated from multi-view images using Ray-Space method. Ray-Space data requires ray interpolation so as to satisfy the plenoptic function. Ray interpolation is realized by estimating view-dependent depth. Depth estimation is usually costly process, thus it is desirable that this process is skipped from rendering process to achieve real-time rendering. This paper proposes a method to render a novel view image using multi-view images and depth maps which are computed in advance. Virtual viewpoint image is generated by 3D warping, which causes some problems that have not occurred in the method with view dependent depth estimation. We handled these problems by projecting depth map to virtual image plane first and perform post-filtering on the projected depth map. We succeeded in obtaining high quality arbitrary viewpoint images from relatively small number of cameras.

View Generation with 3D Warping Using Depth Information for FTV

In this paper, we propose a new refinement filter for depth maps. The filter convolutes a depth map by a jointly computed kernel on a natural image with a weight map. We call the filter weighted joint bilateral filter. The filter fits an outline of an object in the depth map to the outline of the object in the natural image, and it reduces noises. An additional filter of slope depth compensation filter removes blur across object boundary. The filter set’s computational cost is low and is independent of depth ranges. Thus we can refine depth maps to generate accurate depth map with lower cost. In addition, we can apply the filters for various types of depth map, such as computed by simple block matching, Markov random field based optimization, and Depth sensors. Experimental results show that the proposed filter has the best performance of improvement of depth map accuracy, and the proposed filter can perform real-time refinement.

Weighted Joint Bilateral Filter with Slope Depth Compensation Filter for Depth Map Refinement

In this paper, we deal with harvesting fruit in a 3-D virtual space for an olfactory and haptic media display system. Making use of the system, a user can pick fruit from a tree in the 3-D virtual space and perceive the smell of picked fruit by using an olfactory display (SyP@D2) and the reaction force when picking fruit by using a haptic interface device (PHANToM Omni). In the case where there exists inter-stream synchronization error between olfactory and haptic media, we assess the inter-stream synchronization quality as the quality of experience (QoE).

QoE assessment in olfactory and haptic media transmission: Influence of inter-stream synchronization error

Ray-Space is categorized by Image-Based Rendering (IBR), thus generated views have photo-realistic quality.
While this method has the performance of high quality imaging, this needs a lot of images or cameras. The reason
why that is Ray-Space requires various direction's and position's views instead of 3D depth information. In this
paper, we reduce that flood of information using view-centered ray interpolation. View-centered interpolation
means estimating view dependent depth value (or disparity map) at generating view-point and interpolating
that of pixel values using multi-view images and depth information. The combination of depth estimation and
interpolation realizes the rendering photo-realistic images effectively. Unfortunately, however, if depth estimation
is week or mistake, a lot of artifacts appear in creating images. Thus powerful depth estimation method is
required. When we render the free viewpoint images video, we perform the depth estimation at every frame.
Thus we want to keep a lid on computing cost. Our depth estimation method is based on dynamic programming
(DP). This method optimizes and solves depth images at the weak matching area with high-speed performance.
But scan-line noises become appeared because of the limit of DP. So, we perform the DP multi-direction pass and
sum-up the result of multi-passed DPs. Our method fulfills the low computation cost and high depth estimation
performance.

Norishige Fukushima

Papers

View generation with 3D warping using depth information for FTV

View Generation with 3D Warping Using Depth Information for FTV

Weighted Joint Bilateral Filter with Slope Depth Compensation Filter for Depth Map Refinement

QoE assessment in olfactory and haptic media transmission: Influence of inter-stream synchronization error

Free Viewpoint Image Generation Using Multi-Pass Dynamic Programming