Accurate and high-quality depth maps are required in lots of 3D applications, such as multi-view rendering, 3D reconstruction and 3DTV. However, the resolution of captured depth image is much lower than that of its corresponding color image, which affects its application performance. In this paper, we propose a novel depth map super-resolution (SR) method by taking view synthesis quality into account. The proposed approach mainly includes two technical contributions. First, since the captured low-resolution (LR) depth map may be corrupted by noise and occlusion, we propose a credibility based multi-view depth maps fusion strategy, which considers the view synthesis quality and interview correlation, to refine the LR depth map. Second, we propose a view synthesis quality based trilateral depth-map up-sampling method, which considers depth smoothness, texture similarity and view synthesis quality in the up-sampling filter. Experimental results demonstrate that the proposed method outperforms state-of-the-art depth SR methods for both super-resolved depth maps and synthesized views. Furthermore, the proposed method is robust to noise and achieves promising results under noise-corruption conditions.

Depth Map Super-Resolution Considering View Synthesis Quality

Acquiring scenery depth is a fundamental task in computer vision, with many applications in manufacturing, surveillance, or robotics relying on accurate scenery information. Time-of-flight cameras can provide depth information in real-time and overcome short-comings of traditional stereo analysis. However, they provide limited spatial resolution and sophisticated upscaling algorithms are sought after. In this paper, we present a sensor fusion approach to time-of-flight super resolution, based on the combination of depth and texture sources. Unlike other texture guided approaches, we interpret the depth upscaling process as a weighted energy optimization problem. Three different weights are introduced, employing different available sensor data. The individual weights address object boundaries in depth, depth sensor noise, and temporal consistency. Applied in consecutive order, they form three weighting strategies for time-of-flight super resolution. Objective evaluations show advantages in depth accuracy and for depth image based rendering compared with state-of-the-art depth upscaling. Subjective view synthesis evaluation shows a significant increase in viewer preference by a factor of four in stereoscopic viewing conditions. To the best of our knowledge, this is the first extensive subjective test performed on time-of-flight depth upscaling. Objective and subjective results proof the suitability of our approach to time-of-flight super resolution approach for depth scenery capture.

/pdf/a-weighted-optimization-approach-to-time-of-flight-sensor-odnup23nzo.pdf

A Weighted Optimization Approach to Time-of-Flight Sensor Fusion

Recently, there has been remarkable growth of interest in the development and applications of time-of-flight (ToF) depth cameras. Despite the permanent improvement of their characteristics, the practical applicability of ToF cameras is still limited by low resolution and quality of depth measurements. This has motivated many researchers to combine ToF cameras with other sensors in order to enhance and upsample depth images. In this paper, we review the approaches that couple ToF depth images with high-resolution optical images. Other classes of upsampling methods are also briefly discussed. Finally, we provide an overview of performance evaluation tests presented in the related studies.

/pdf/image-guided-tof-depth-upsampling-a-survey-1n6w3xj431.pdf

Image-guided ToF depth upsampling: a survey

An apparatus for generating an output depth map for an image comprises a first depth processor (103) which generates a first depth map for the image from an input depth map. A second depth processor (105) generates a second depth map for the image by applying an image property dependent filtering to the input depth map. The image property dependent filtering may specifically be a cross-bilateral filtering of the input depth map. An edge processor (107) determines an edge map for the image and a combiner (109) generates the output depth map for the image by combining the first depth map and the second depth map in response to the edge map. Specifically, the second depth map may be weighted higher around edges than away from edges. The invention may in many embodiments provide a temporally and spatially more stable depth map while reducing degradations and artifacts introduced by the processing.

Generation of a depth map for an image

Highly accurate depth images at video frame rate are required in many areas of computer vision, such as 3D reconstruction, 3D video capturing or manufacturing. Nowadays low cost depth cameras, which deliver a high frame rate, are widely spread but suffer from a high level of noise and a low resolution. Thus, a sophisticated real time upsampling algorithm is strongly required. In this paper we propose a new sensor fusion approach called Combined Bilateral Filter (CBF) together with the new Depth Discontinuity Preservation (DDP) post processing, which combine the information of a depth and a color sensor. Thereby we especially focus on two drawbacks that are common in related algorithms namely texture copying and upsampling without depth discontinuity preservation. The output of our algorithm is a higher resolution depth image with essentially reduced noise, no aliasing effects, no texture copying and very sharply preserved edges. In a ground truth comparison our algorithm was able to reduce the mean error up to 73% within around 30ms. Furthermore, we compare our method against other state of the art algorithms and obtain superior results.

https://www.dfki.de/web/forschung/publikationen/renameFileForDownload?filename=CBF_VISAPP15_final.pdf&file_id=uploads_2391

Combined Bilateral Filter for Enhanced Real-time Upsampling of Depth Images

Depth maps are used in many applications, e.g. 3D television, stereo matching, segmentation, etc. Recently, a new generation of active 3D range sensors, such as time-of-flight (TOF) cameras, enables recording of full frame depth maps at video frame rate. Unfortunately, depth maps captured with the TOF cameras have limited resolution and poor image quality, being serverely influenced by the random and systematic noise, which makes them innaposite for generating high quality 3D images. In this paper, we proposed a method to enhance the quality and increase the spatial resolution of range data by upsampling the range information with the data from a high resolution video camera. Our algorithm is an adaptive upsampling filter that takes into account the inherent noisy nature of depth data. Thus, we can improve reconstruction quality, boost the resolution of the data to that of the video sensor, and prevent unwanted artifacts like texture copy into geometry.

CholSu Kim

Papers

Depth super resolution using bilateral filter