Recently, consumer depth cameras have gained significant popularity due to their affordable cost. However, the limited resolution and the quality of the depth map generated by these cameras are still problematic for several applications. In this paper, a novel framework for the single depth image superresolution is proposed. In our framework, the upscaling of a single depth image is guided by a high-resolution edge map, which is constructed from the edges of the low-resolution depth image through a Markov random field optimization in a patch synthesis based manner. We also explore the self-similarity of patches during the edge construction stage, when limited training data are available. With the guidance of the high-resolution edge map, we propose upsampling the high-resolution depth image through a modified joint bilateral filter. The edge-based guidance not only helps avoiding artifacts introduced by direct texture prediction, but also reduces jagged artifacts and preserves the sharp edges. Experimental results demonstrate the effectiveness of our method both qualitatively and quantitatively compared with the state-of-the-art methods.

Edge-Guided Single Depth Image Super Resolution

This paper describes a new algorithm for depth image super resolution and denoising using a single depth image as input. A robust coupled dictionary learning method with locality coordinate constraints is introduced to reconstruct the corresponding high resolution depth map. The local constraints effectively reduce the prediction uncertainty and prevent the dictionary from over-fitting. We also incorporate an adaptively regularized shock filter to simultaneously reduce the jagged noise and sharpen the edges. Furthermore, a joint reconstruction and smoothing framework is proposed with an L0 gradient smooth constraint, making the reconstruction more robust to noise. Experimental results demonstrate the effectiveness of our proposed algorithm compared to previously reported methods.

Joint Super Resolution and Denoising From a Single Depth Image

Recently, consumer depth cameras have gained significant popularity due to their affordable cost. However, the limited resolution and quality of the depth map generated by these cameras are still problems for several applications. In this paper, we propose a novel framework for single depth image super resolution guided by a high resolution edge map constructed from the edges in the low resolution depth image via a Markov Random Field (MRF) optimization. With the guidance of the high resolution edge map, the high resolution depth image is up-sampled via a joint bilateral filter. The edge guidance not only helps avoid artifacts introduced by direct texture prediction, but also reduces the jagged artifacts and preserves the sharp edges. Experimental results demonstrate the effectiveness of our proposed algorithm compared to previously reported methods.

Edge guided single depth image super resolution

We present a novel method that can enhance the spatial resolution of stereo images using a parallax prior. While traditional stereo imaging has focused on estimating depth from stereo images, our method utilizes stereo images to enhance spatial resolution instead of estimating disparity. The critical challenge for enhancing spatial resolution from stereo images: how to register corresponding pixels with subpixel accuracy. Since disparity in traditional stereo imaging is calculated per pixel, it is directly inappropriate for enhancing spatial resolution. We, therefore, learn a parallax prior from stereo image datasets by jointly training two-stage networks. The first network learns how to enhance the spatial resolution of stereo images in luminance, and the second network learns how to reconstruct a high-resolution color image from high-resolution luminance and chrominance of the input image. Our two-stage joint network enhances the spatial resolution of stereo images significantly more than single-image super-resolution methods. The proposed method is directly applicable to any stereo depth imaging methods, enabling us to enhance the spatial resolution of stereo images.

Enhancing the Spatial Resolution of Stereo Images Using a Parallax Prior

We develop a new model for no-reference 3D stereopair quality assessment that considers the impact of binocular fusion, rivalry, suppression, and a reverse saliency effect on the perception of distortion. The resulting framework, dubbed the S3D INtegrated Quality (SINQ) Predictor, first fuses the left and right views of a stereopair into a single synthesized cyclopean image using a novel modification of an existing binocular perceptual model. Specifically, the left and right views of a stereopair are fused using a measure of “cyclopean” spatial activity. A simple product estimate is also calculated as the correlation between left and right disparity-corrected corresponding binocular pixels. Univariate and bivariate statistical features are extracted from the four available image sources: the left view, the right view, the synthesized “cyclopean” spatial activity image, and the binocular product image. Based on recent evidence regarding the placement of 3D fixation by subjects viewing stereoscopic 3D (S3D) content, we also deploy a reverse saliency weighting on the normalized “cyclopean” spatial activity image. Both one- and two-stage frameworks are then used to map the feature vectors to predicted quality scores. SINQ is thoroughly evaluated on the LIVE 3D image quality database (Phase I and Phase II). The experimental results show that SINQ delivers better performance than state of the art 2D and 3D quality assessment methods on six public databases, especially on asymmetric distortions.

Binocular spatial activity and reverse saliency driven no-reference stereopair quality assessment

From a rectified stereo image pair, the task of view synthesis is to generate images from any viewpoint along the baseline. The main difficulty of the problem is how to fill occluded regions. In this paper, we present a new method for view synthesis that is both fast and accurate. Occlusions are filled using color and disparity information to produce consistent pixel estimates. Results are comparable to current state-of-the-art methods in terms of objective measures while computation time is drastically reduced. This work has applications in free-viewpoint television, angular scalability for 3D video coding/decoding, and stereo-to-multiview conversion.

Efficient stereo-to-multiview synthesis

Mixed resolution stereoscopic coding exploits the perceptual phenomenon of binocular suppression by encoding each view of a stereo pair at a different resolution. Many investigations have shown that a large reduction in bandwidth can be gained for a relatively small compromise in image quality by this asymmetric representation. However, the viability of such an encoding method depends on the subjective response to viewing such videos at length. While methods for mixed resolution coding have been developed on the presumption of a certain visual fatigue response, none have actually examined it. In this paper, we address this shortcoming in three experiments comparing two methods of binocular suppression processing. The first two experiments reveal subjects' preferences in terms of overall quality between the two methods for short exposures, and the third experiment examines the fatigue resulting from 10-min exposures to mixed resolution encoded videos.

Comparing Perceived Quality and Fatigue for Two Methods of Mixed Resolution Stereoscopic Coding

Mixed resolution stereoscopic coding is a compression method that reduces the bandwidth of stereo video by transmitting one of the two views at a lower resolution While the visual system weights the sharper view more heavily resulting in a nearly fully sharp 3D percept, certain applications require, or would benefit from, having both views at full resolution In this paper, we present a super-resolution method that recovers the lost frequency information in mixed resolution stereo video by exploiting stereo correspondence while emphasizing spatial, temporal, and spectral consistency Our method is distinguished by operating on video modeled as a 3D Markov network, whereas previous approaches focused solely on images

Video super-resolution for mixed resolution stereo

Disparity estimation is a fundamental task in stereo imaging and is a well-studied problem. Recently, methods have been adapted to the video domain where motion is used as a matching criterion to help disambiguate spatially similar candidates. In this paper, we analyze the validity of the underlying assumptions of spatio-temporal disparity estimation, and determine the extent to which motion aids the matching process. By analyzing the error signal for spatio-temporal block matching under the sum of squared differences criterion and treating motion as a stochastic process, we determine the probability of a false match as a function of image features, motion distribution, image noise, and number of frames in the spatio-temporal patch. This performance quantification provides insight into when spatio-temporal matching is most beneficial in terms of the scene and motion, and can be used as a guide to select parameters for stereo matching algorithms. We validate our results through simulation and experiments on stereo video.

Discriminability Limits in Spatio-Temporal Stereo Block Matching

Binocular luster is an extremely salient effect seen in 3D when an object in each stereo image exhibits a different contrast polarity relative to the background. The object appears to shimmer, a phenomenon seen in nature and on 3D displays, which the Human Visual System rapidly detects. This binocular luster is also induced by compression of stereo imagery where corresponding blocks are quantized to different values. In this paper, we discuss the psychovisual background of binocular luster, introduce the “shine” artifact induced by compression, and present an algorithm for detection and removal of shine from JPEG compressed stereo images without introducing additional bitrate.

Ankit K. Jain

Papers

Efficient stereo-to-multiview synthesis

Comparing Perceived Quality and Fatigue for Two Methods of Mixed Resolution Stereoscopic Coding

Video super-resolution for mixed resolution stereo

Discriminability Limits in Spatio-Temporal Stereo Block Matching

Detection and removal of binocular luster in compressed 3D images