From the Publisher: 
The accessible presentation of this book gives both a general view of the entire computer vision enterprise and also offers sufficient detail to be able to build useful applications. Users learn techniques that have proven to be useful by first-hand experience and a wide range of mathematical methods. A CD-ROM with every copy of the text contains source code for programming practice, color images, and illustrative movies. Comprehensive and up-to-date, this book includes essential topics that either reflect practical significance or are of theoretical importance. Topics are discussed in substantial and increasing depth. Application surveys describe numerous important application areas such as image based rendering and digital libraries. Many important algorithms broken down and illustrated in pseudo code. Appropriate for use by engineers as a comprehensive reference to the computer vision enterprise.

Computer vision : a modern approach = 计算机视觉 : 一种现代的方法

Significant improvements in video compression capability have been demonstrated with the introduction of the H.264/MPEG-4 advanced video coding (AVC) standard. Since developing this standard, the Joint Video Team of the ITU-T Video Coding Experts Group (VCEG) and the ISO/IEC Moving Picture Experts Group (MPEG) has also standardized an extension of that technology that is referred to as multiview video coding (MVC). MVC provides a compact representation for multiple views of a video scene, such as multiple synchronized video cameras. Stereo-paired video for 3-D viewing is an important special case of MVC. The standard enables inter-view prediction to improve compression capability, as well as supporting ordinary temporal and spatial prediction. It also supports backward compatibility with existing legacy systems by structuring the MVC bitstream to include a compatible “base view.” Each other view is encoded at the same picture resolution as the base view. In recognition of its high-quality encoding capability and support for backward compatibility, the stereo high profile of the MVC extension was selected by the Blu-Ray Disc Association as the coding format for 3-D video with high-definition resolution. This paper provides an overview of the algorithmic design used for extending H.264/MPEG-4 AVC towards MVC. The basic approach of MVC for enabling inter-view prediction and view scalability in the context of H.264/MPEG-4 AVC is reviewed. Related supplemental enhancement information (SEI) metadata is also described. Various “frame compatible” approaches for support of stereo-view video as an alternative to MVC are also discussed. A summary of the coding performance achieved by MVC for both stereo- and multiview video is also provided. Future directions and challenges related to 3-D video are also briefly discussed.

/pdf/overview-of-the-stereo-and-multiview-video-coding-extensions-18yz9qlji4.pdf

Overview of the Stereo and Multiview Video Coding Extensions of the H.264/MPEG-4 AVC Standard

Current 3-D video (3DV) technology is based on stereo systems. These systems use stereo video coding for pictures delivered by two input cameras. Typically, such stereo systems only reproduce these two camera views at the receiver and stereoscopic displays for multiple viewers require wearing special 3-D glasses. On the other hand, emerging autostereoscopic multiview displays emit a large numbers of views to enable 3-D viewing for multiple users without requiring 3-D glasses. For representing a large number of views, a multiview extension of stereo video coding is used, typically requiring a bit rate that is proportional to the number of views. However, since the quality improvement of multiview displays will be governed by an increase of emitted views, a format is needed that allows the generation of arbitrary numbers of views with the transmission bit rate being constant. Such a format is the combination of video signals and associated depth maps. The depth maps provide disparities associated with every sample of the video signal that can be used to render arbitrary numbers of additional views via view synthesis. This paper describes efficient coding methods for video and depth data. For the generation of views, synthesis methods are presented, which mitigate errors from depth estimation and coding.

3-D Video Representation Using Depth Maps

This paper describes an extension of the high efficiency video coding (HEVC) standard for coding of multi-view video and depth data. In addition to the known concept of disparity-compensated prediction, inter-view motion parameter, and inter-view residual prediction for coding of the dependent video views are developed and integrated. Furthermore, for depth coding, new intra coding modes, a modified motion compensation and motion vector coding as well as the concept of motion parameter inheritance are part of the HEVC extension. A novel encoder control uses view synthesis optimization, which guarantees that high quality intermediate views can be generated based on the decoded data. The bitstream format supports the extraction of partial bitstreams, so that conventional 2D video, stereo video, and the full multi-view video plus depth format can be decoded from a single bitstream. Objective and subjective results are presented, demonstrating that the proposed approach provides 50% bit rate savings in comparison with HEVC simulcast and 20% in comparison with a straightforward multi-view extension of HEVC without the newly developed coding tools.

3D High-Efficiency Video Coding for Multi-View Video and Depth Data

The present disclosure pertains to an image-processing device and method that make it possible to suppress block noise. A βLUT_input calculator and a clip processor determine βLUT_input, which is a value inputted to an existing-β generator and an expanded-β generator. When the value of βLUT_input qp from the clip processor is 51 or less, the existing-β generator determines β using a LUT conforming to the HVEC standard, and supplies same to a filter process determination unit. When the value of βLUT_input qp from the clip processor is larger than 51, the expanded-β generator determines the expanded β and supplies same to the filter process determination part. This disclosure, for example, can be applied to an image processing device.

Image processing device and method

To facilitate new video applications such as three-dimensional video (3DV) and free-viewpoint video (FVV), multiple view plus depth format (MVD), which consists of both video views and the corresponding per-pixel depth images, is being investigated. Virtual views can be generated using depth image based rendering (DIBR), which takes video and the corresponding depth images as input. This paper discusses view synthesis techniques based on DIBR, which includes forward warping, blending and hole filling. Especially, we will emphasize on the techniques brought to the MPEG view synthesis reference software (VSRS). Unlike the case in the field of computer graphics, the ground truth depth images for nature content are very difficult to obtain. The estimated depth images used for view synthesis typically contain different types of noises. Some robust synthesis modes to combat against the depth errors are also presented in this paper. In addition, we briefly discuss how to use synthesis techniques with minor modifications to generate the occlusion layer information for layered depth video (LDV) data, which is another potential format for 3DV applications.

View synthesis techniques for 3D video

Video representations that support view synthesis based on depth maps, such as multiview plus depth (MVD), have been recently proposed, raising interest in efficient tools for depth map coding. In this paper, we derive a new distortion metric that takes into consideration camera parameters and global video characteristics in order to quantify the effect of lossy coding of depth maps on synthesized view quality. In addition, a new skip mode selection method is proposed based on local video characteristics. Experimental results with the proposed mode selection scheme show coding gains of up to 2 dB for the synthesized views, as well as better subjective quality.

Depth map distortion analysis for view rendering and depth coding

New data formats that include both video and the corresponding depth maps, such as multiview plus depth
(MVD), enable new video applications in which intermediate video views (virtual views) can be generated using
the transmitted/stored video views (reference views) and the corresponding depth maps as inputs. We propose a
depth map coding method based on a new distortion measurement by deriving relationships between distortions
in coded depth map and rendered view. In our experiments we use a codec based on H.264/AVC tools, where the
rate-distortion (RD) optimization for depth encoding makes use of the new distortion metric. Our experimental
results show the efficiency of the proposed method, with coding gains of up to 1.6 dB in interpolated frame
quality as compared to encoding the depth maps using the same coding tools but applying RD optimization
based on conventional distortion metrics.

Depth map coding with distortion estimation of rendered view

With the development of 3D display and interactive multimedia systems, new 3D video applications, such as 3DTV and Free Viewpoint Video, are attracting significant interests. In order to enable these new applications, new data formats including captured 2D video sequences and corresponding depth maps have been proposed. Compared to conventional video frames, depth maps have very different characteristics. First, they typically consist of homogeneous areas partitioned by sharp edges representing depth discontinuities, while the sharp discontinuities play very important roles in view rendering. Second, there exists structure similarity between depth map and corresponding video, in which the edges in depth exhibit quite similar behaviors as the edges in the corresponding video. In conventional video coding techniques with transforms followed by quantization, there usually exist large artifacts along sharp edges and it costs significant more bits to represent the edges with higher accuracy. In order to suppress the coding artifacts while preserving edges, and to better compress depth information, we propose in this paper two new depth compression techniques: Trilateral Filter and Sparse Dyadic Mode. Both techniques utilize the structure similarity between depth and corresponding video while focusing on different aspects in depth compression. As a new in-loop filter, Trilateral Filter is designed to filter depth with coding artifacts based on the proximity of pixel positions, the similarity among depth samples, as well as the similarity among the collocated pixels in the video frame. While Sparse Dyadic Mode is used as an intra mode to reconstruct depth map with sparse representations of depth blocks and effective reference of edge information from video frames. With these two new coding tools, we can achieve up to about 1.5 dB gain on rendering quality as compared to depth sequences coded using MVC under the same coding rate.

New Depth Coding Techniques With Utilization of Corresponding Video

There are provided methods and apparatus for adaptive reference filtering. An apparatus includes an encoder (100) for encoding at least one picture. The encoder (100) performs adaptive filtering of at least one reference picture to respectively obtain at least one filtered reference picture, and predictively codes the at least one picture using the at least one filtered reference picture. The at least one reference picture is a picture wherein at least one sample thereof is used for inter-prediction either of subsequent to the at least one sample being applied to an in-loop filter or in an absence of the at least one sample being applied to an in-loop filter.

PoLin Lai

Papers

View synthesis techniques for 3D video

Depth map distortion analysis for view rendering and depth coding

Depth map coding with distortion estimation of rendered view

New Depth Coding Techniques With Utilization of Corresponding Video

Methods and apparatus for adaptive reference filtering