This paper reports on the 2018 PIRM challenge on perceptual super-resolution (SR), held in conjunction with the Perceptual Image Restoration and Manipulation (PIRM) workshop at ECCV 2018. In contrast to previous SR challenges, our evaluation methodology jointly quantifies accuracy and perceptual quality, therefore enabling perceptual-driven methods to compete alongside algorithms that target PSNR maximization. Twenty-one participating teams introduced algorithms which well-improved upon the existing state-of-the-art methods in perceptual SR, as confirmed by a human opinion study. We also analyze popular image quality measures and draw conclusions regarding which of them correlates best with human opinion scores. We conclude with an analysis of the current trends in perceptual SR, as reflected from the leading submissions.

/pdf/the-2018-pirm-challenge-on-perceptual-image-super-resolution-vzmp1nma1u.pdf

The 2018 PIRM Challenge on Perceptual Image Super-Resolution

Rapid development of affordable and portable consumer depth cameras facilitates the use of depth information in many computer vision tasks such as intelligent vehicles and 3D reconstruction. However, depth map captured by low-cost depth sensors (e.g., Kinect) usually suffers from low spatial resolution, which limits its potential applications. In this paper, we propose a novel deep network for depth map super-resolution (SR), called DepthSR-Net. The proposed DepthSR-Net automatically infers a high-resolution (HR) depth map from its low-resolution (LR) version by hierarchical features driven residual learning. Specifically, DepthSR-Net is built on residual U-Net deep network architecture. Given LR depth map, we first obtain the desired HR by bicubic interpolation upsampling and then construct an input pyramid to achieve multiple level receptive fields. Next, we extract hierarchical features from the input pyramid, intensity image, and encoder–decoder structure of U-Net. Finally, we learn the residual between the interpolated depth map and the corresponding HR one using the rich hierarchical features. The final HR depth map is achieved by adding the learned residual to the interpolated depth map. We conduct an ablation study to demonstrate the effectiveness of each component in the proposed network. Extensive experiments demonstrate that the proposed method outperforms the state-of-the-art methods. In addition, the potential usage of the proposed network in other low-level vision problems is discussed.

Hierarchical Features Driven Residual Learning for Depth Map Super-Resolution

This paper reviews the AIM 2019 challenge on real world super-resolution. It focuses on the participating methods and final results. The challenge addresses the real world setting, where paired true high and low-resolution images are unavailable. For training, only one set of source input images is therefore provided in the challenge. In Track 1: Source Domain the aim is to super-resolve such images while preserving the low level image characteristics of the source input domain. In Track 2: Target Domain a set of high-quality images is also provided for training, that defines the output domain and desired quality of the super-resolved images. To allow for quantitative evaluation, the source input images in both tracks are constructed using artificial, but realistic, image degradations. The challenge is the first of its kind, aiming to advance the state-of-the-art and provide a standard benchmark for this newly emerging task. In total 7 teams competed in the final testing phase, demonstrating new and innovative solutions to the problem.

AIM 2019 Challenge on Real-World Image Super-Resolution: Methods and Results

In this paper we propose a novel method for depth image superresolution which combines recent advances in example based upsampling with variational superresolution based on a known blur kernel. Most traditional depth superresolution approaches try to use additional high resolution intensity images as guidance for superresolution. In our method we learn a dictionary of edge priors from an external database of high and low resolution examples. In a novel variational sparse coding approach this dictionary is used to infer strong edge priors. Additionally to the traditional sparse coding constraints the difference in the overlap of neighboring edge patches is minimized in our optimization. These edge priors are used in a novel variational superresolution as anisotropic guidance of the higher order regularization. Both the sparse coding and the variational superresolution of the depth are solved based on a primal-dual formulation. In an exhaustive numerical and visual evaluation we show that our method clearly outperforms existing approaches on multiple real and synthetic datasets.

/pdf/variational-depth-superresolution-using-example-based-edge-4v97uqqy4n.pdf

Variational Depth Superresolution Using Example-Based Edge Representations

Deep learning for monocular depth estimation: A review

Noise adaptive super-resolution from single image via non-local mean and sparse representation

Depth map sensed by low-cost active sensor is often limited in resolution, whereas depth information achieved from structure from motion or sparse depth scanning techniques may result in a sparse point cloud. Achieving a high-resolution (HR) depth map from a low resolution (LR) depth map or densely reconstructing a sparse non-uniformly sampled depth map are fundamentally similar problems with different types of upsampling requirements. The first problem involves upsampling in a uniform grid, whereas the second type of problem requires an upsampling in a non-uniform grid. In this paper, we propose a new approach to address such issues in a unified framework, based on sparse representation. Unlike, most of the approaches of depth map restoration, our approach does not require an HR intensity image. Based on example depth maps, sub-dictionaries of exemplars are constructed, and are used to restore HR/dense depth map. In the case of uniform upsampling of LR depth map, an edge preserving constraint is used for preserving the discontinuity present in the depth map, and a pyramidal reconstruction strategy is applied in order to deal with higher upsampling factors. For upsampling of non-uniformly sampled sparse depth map, we compute the missing information in local patches from that from similar exemplars. Furthermore, we also suggest an alternative method of reconstructing dense depth map from very sparse non-uniformly sampled depth data by sequential cascading of uniform and non-uniform upsampling techniques. We provide a variety of qualitative and quantitative results to demonstrate the efficacy of our approach for depth map restoration.

Depth Map Restoration From Undersampled Data

Quality of an image is associated with edge of the image. It is important to preserve the edge of the image while deriving high resolution (HR) image from low resolution (LR) image, also known as superresolution (SR) problem. This paper proposes an edge preserving constraint, which preserve the edge information of image by minimizing the differences between edges of LR image and the edges of the reconstructed image (down-sampled version), in sparse coding based SR problem. Partial edge evidences, derived using 1-D processing of image, are used separately in the constraints. The experimental results show that proposed approach preserves the edges of image as well as outperforms objectively the existing SR approaches.

Edge preserving single image super resolution in sparse environment

Mixed-dense connection networks for image and video super-resolution

Atmospheric medium often constrains the visibility of outdoor scenes due to scattering of light rays. This causes attenuation in the irradiance reaching the imaging device along with an additive component to render a hazy effect in the image. The visibility is further reduced for poorly illuminated scenes. The attenuation becomes wavelength dependent in underwater scenario, causing undesired color cast along with hazy effect. In order to suppress the effect of different atmospheric/underwater conditions such as haze and to enhance the contrast of such images, we reformulate local haziness in a generalized manner. The parameters are estimated by harnessing the similarity of patches within a local neighborhood. Unlike existing methods, our approach is developed based on the assumption that for outdoor scenes, the depth of patches changes gradually in a local neighborhood surrounding the patch. This change in depth can be approximated by patch similarity in that neighborhood. As the attenuation in irradiance of an image in presence of atmospheric medium relies on the depth of the scene, the coefficients related to the attenuation are estimated from the weights of patch similarity. The additive haze effect is deduced using non-local mean of the patch. Our experimental results demonstrate the effectiveness of our approach in reducing the haze component as well as in enhancing the image under different conditions of haze (daytime, nighttime, and underwater).

Srimanta Mandal

Papers

Noise adaptive super-resolution from single image via non-local mean and sparse representation

Depth Map Restoration From Undersampled Data

Edge preserving single image super resolution in sparse environment

Mixed-dense connection networks for image and video super-resolution

Local Proximity for Enhanced Visibility in Haze