In this paper, we propose a machine learning-based fast coding unit (CU) depth decision method for High Efficiency Video Coding (HEVC), which optimizes the complexity allocation at CU level with given rate-distortion (RD) cost constraints. First, we analyze quad-tree CU depth decision process in HEVC and model it as a three-level of hierarchical binary decision problem. Second, a flexible CU depth decision structure is presented, which allows the performances of each CU depth decision be smoothly transferred between the coding complexity and RD performance. Then, a three-output joint classifier consists of multiple binary classifiers with different parameters is designed to control the risk of false prediction. Finally, a sophisticated RD-complexity model is derived to determine the optimal parameters for the joint classifier, which is capable of minimizing the complexity in each CU depth at given RD degradation constraints. Comparative experiments over various sequences show that the proposed CU depth decision algorithm can reduce the computational complexity from 28.82% to 70.93%, and 51.45% on average when compared with the original HEVC test model. The Bjontegaard delta peak signal-to-noise ratio and Bjontegaard delta bit rate are −0.061 dB and 1.98% on average, which is negligible. The overall performance of the proposed algorithm outperforms those of the state-of-the-art schemes.

Machine Learning-Based Coding Unit Depth Decisions for Flexible Complexity Allocation in High Efficiency Video Coding

High efficiency video coding (HEVC) is the latest video coding standard that has been developed by JCT-VC. It employs plenty of efficient coding algorithms (e.g., highly flexible quad-tree coding block partitioning), and outperforms H.264/AVC by 35–43% bitrate reduction. However, it imposes enormous computational complexity on encoder due to the optimization processing in the efficient coding tools, especially the rate distortion optimization on coding unit (CU), prediction unit, and transform unit. In this article, we propose a CU splitting early termination algorithm to reduce the heavy computational burden on encoder. CU splitting is modeled as a binary classification problem, on which a support vector machine (SVM) is applied. In order to reduce the impact of outliers as well as to maintain the RD performance while a misclassification occurs, RD loss due to misclassification is introduced as weights in SVM training. Efficient and representative features are extracted and optimized by a wrapper approach to eliminate dependency on video content as well as on encoding configurations. Experimental results show that the proposed algorithm can achieve about 44.7% complexity reduction on average with only 1.35% BD-rate increase under the “random access” configuration, and 41.9% time saving with 1.66% BD-rate increase under the “low delay” setting, compared with the HEVC reference software.

/pdf/cu-splitting-early-termination-based-on-weighted-svm-7wfby2o2fg.pdf

CU splitting early termination based on weighted SVM

In this paper we will present the concept of a system that allows for an evolutionary introduction of depth perception into the existing 2D digital TV framework. The work is part of the European Information Society Technologies (IST) project “Advanced Three-Dimensional Television System Technologies” (ATTEST), an activity where industries, research centers and universities have joined forces to design a backwardscompatible, flexible and modular broadcast 3D-TV system, where all parts of the 3D processing chain are optimised to one another. This includes content creation, coding and transmission, display and research in human 3D perception, which be will used to guide the development process. The goals of the project comprise the development of a novel broadcast 3D camera, algorithms to convert existing 2D-video material into 3D, a 2Dcompatible coding and transmission scheme for 3D-video using MPEG2/4/7 technologies and the design of two new autostereoscopic displays.

IST project ATTEST - an evolutionary and optimised approach on 3D-TV

This application generally relates to capturing and aligning panoramic image and depth data. In one embodiment, a device is provided that comprises a housing and a plurality of cameras configured to capture two-dimensional images, wherein the cameras are arranged at different positions on the housing and have different azimuth orientations relative to a center point such that the cameras have a collective field-of-view spanning up to 360° horizontally. The device further comprises a plurality of depth detection components configured to capture depth data, wherein the depth detection components are arranged at different positions on the housing and have different azimuth orientations relative to the center point such that the depth detection components have the collective field-of-view spanning up to 360° horizontally.

Capturing and aligning panoramic image and depth data

Machine learning based video coding optimizations: A survey

In this article, we propose a 2D to 3D video conversion scheme for MPEG videos. The difficulty for 2D/3D conversion problem lies on depth estimation/ assignment with insufficient information. Our depth assignment is based on the analyses of multiple cues, e.g., motion parallax, atmospheric perspective, texture gradient, linear perspective, and relative height, separately for the foreground objects and the background area. To fit more kinds of videos, the proposed depth assignment scheme is content-adaptive by segmenting a video into shots and classifying each of them into three categories for different conversion schemes. Subjective experiments show that the 3D stereo video generated by using our depth assignment scheme and the Depth Image Based Rendering (DIBR) technique presents little difference to that created based on the depth ground truths.

A 2D to 3D conversion scheme based on depth cues analysis for MPEG videos

Stereo video, targeting at matching what the humans see in the real world, offers depth perception on observed scenes. The problem of compressing such a huge amount of video data has received considerable attention in the past few years. Its challenge comes from the much more complicated parameter selection (e.g., mode decision) process than single-channel video coding. To cope with this problem, the currently developed H.264/AVC-based JMVM platform is adopted here for implementation, where the encoding of the left-view channel is purely based on predictions from the temporal domain, while for the right-view channel, combined predictions from the temporal and the inter-view domains are exploited and a hierarchical two-stage neural classifier is designed for fast mode decision. The first-stage neural classifier determines candidates of block partition for each macroblock, while the second-stage classifier aims to choose the most probable prediction sources among the temporally forward/backward and inter-view directions. All input features for both stages of neural classifiers are calculated from simple inter-frame and inter-view analyses. In our scheme, the popular fast motion estimation schemes can be also cooperated for further speedup. Experiment results reveal that our proposed algorithm is capable of achieving up to 97% of time savings with nearly ignorable quality degradation and acceptable bit-rate increase (up to 5.67%).

A Fast H.264/AVC-Based Stereo Video Encoding Algorithm Based on Hierarchical Two-Stage Neural Classification

A 2D video to 3D video conversion system includes a video content analysis unit, a depth estimation unit, a post-processing unit, and a stereoscopic video generation unit. The video content analysis unit can analyze a 2D video datum and extract useful information including motion and color from the 2D video datum for depth estimation. The depth estimation unit is adapted for receiving the useful information, calculating motion cue and contrast cue for initial depth estimation, and generating an initial depth map. The post-processing unit is adapted for correcting the initial depth map in spatial domain and temporal domain to increase accuracy in spatial domain and depth continuity between adjacent time instances and for processing the caption in the video to generate a final depth map. The stereoscopic video generation unit is adapted for synthesizing 3D video datum from the final depth map and 2D video datum.

System for converting 2d video into 3d video

In this paper, a 2D to 3D stereo image conversion scheme is proposed for 3D content creation. The difficulty in this problem lies on depth estimation/assignment from a mono image, which actually does not have sufficient information. To estimate the depth map, we adopt a strategy of performing foreground/background separation first, then classifying a background depth profile by neural network, estimating foreground depth from image cues, and finally combining them. To enhance stereoscopic perception for the synthesized images viewed on 3D display, depth refinement based on bilateral filter and HVS-based contrast modification between the foreground and background are adopted. Subjective experiments show that the stereo images generated by using the proposed scheme can provide good 3D perception.

/pdf/2d-to-3d-image-conversion-based-on-classification-of-2kte98nb1s.pdf

2D to 3d image conversion based on classification of background depth profiles

In this article, an automatic stereoscopic video conversion scheme which accepts MPEG-encoded videos as input is proposed. Our scheme is depth-based, relying on spatio-temporal analysis of the decoded video data to yield depth perception cues, such as temporal motion and spatial contrast, which reflect the relative depths between the foreground and the background areas. Our scheme is shot-adaptive, demanding that shot change detection and shot classification be performed for tuning of algorithm or parameters that are used for depth cue combination. The above-mentioned depth estimation is initially block-based, followed by a locally adaptive joint trilateral upsampling algorithm to reduce the computing load significantly. A recursive temporal filter is used to reduce the possible depth fluctuations (and also artifacts in the synthesized images) resulting from wrong depth estimations. The traditional Depth-Image-Based-Rendering algorithm is used to synthesize the left- and right-view frames for 3D display. Subjective tests show that videos converted by our scheme provide comparable perceived depth and visual quality with those converted from the depth data calculated by stereo vision techniques. Also, our scheme is shown to outperform the well-known TriDef software in terms of human’s perceived 3D depth. Based on the implementation by using “OpenMP” parallel programming model, our scheme is capable of executing in real-time on a multi-core CPU platform.

/pdf/a-stereoscopic-video-conversion-scheme-based-on-spatio-4yxbhunddw.pdf

Wei-Chih Chen

Papers

A 2D to 3D conversion scheme based on depth cues analysis for MPEG videos

A Fast H.264/AVC-Based Stereo Video Encoding Algorithm Based on Hierarchical Two-Stage Neural Classification

System for converting 2d video into 3d video

2D to 3d image conversion based on classification of background depth profiles

A stereoscopic video conversion scheme based on spatio-temporal analysis of MPEG videos