Free-form inpainting is the task of adding new content to an image in the regions specified by an arbitrary binary mask. Most existing approaches train for a certain distribution of masks, which limits their generalization capabilities to unseen mask types. Furthermore, training with pixel-wise and perceptual losses often leads to simple textural extensions towards the missing areas instead of semantically meaningful generation. In this work, we propose RePaint: A Denoising Diffusion Probabilistic Model (DDPM) based inpainting approach that is applicable to even extreme masks. We employ a pretrained unconditional DDPM as the generative prior. To condition the generation process, we only alter the reverse diffusion iterations by sampling the unmasked regions using the given image infor-mation. Since this technique does not modify or condition the original DDPM network itself, the model produces high-quality and diverse output images for any inpainting form. We validate our method for both faces and general-purpose image inpainting using standard and extreme masks. Re-Paint outperforms state-of-the-art Autoregressive, and GAN approaches for at least five out of six mask distributions. Github Repository: git.io/RePaint

RePaint: Inpainting using Denoising Diffusion Probabilistic Models

Motion blur is a common photography artifact in dynamic environments that typically comes jointly with the other types of degradation. This paper reviews the NTIRE 2021 Challenge on Image Deblurring. In this challenge report, we describe the challenge specifics and the evaluation results from the 2 competition tracks with the proposed solutions. While both the tracks aim to recover a high-quality clean image from a blurry image, different artifacts are jointly involved. In track 1, the blurry images are in a low resolution while track 2 images are compressed in JPEG format. In each competition, there were 338 and 238 registered participants and in the final testing phase, 18 and 17 teams competed. The winning methods demonstrate the state-of-the-art performance on the image deblurring task with the jointly combined artifacts.

/pdf/ntire-2021-challenge-on-image-deblurring-4xwplefo3i.pdf

NTIRE 2021 Challenge on Image Deblurring

This paper reviews the first challenge on high-dynamic range (HDR) imaging that was part of the New Trends in Image Restoration and Enhancement (NTIRE) workshop, held in conjunction with CVPR 2021. This manuscript focuses on the newly introduced dataset, the proposed methods and their results. The challenge aims at estimating a HDR image from one or multiple respective low-dynamic range (LDR) observations, which might suffer from under-or over-exposed regions and different sources of noise. The challenge is composed by two tracks: In Track 1 only a single LDR image is provided as input, whereas in Track 2 three differently-exposed LDR images with inter-frame motion are available. In both tracks, the ultimate goal is to achieve the best objective HDR reconstruction in terms of PSNR with respect to a ground-truth image, evaluated both directly and with a canonical tonemapping operation.

/pdf/ntire-2021-challenge-on-high-dynamic-range-imaging-dataset-dmp4lwf5u6.pdf

NTIRE 2021 Challenge on High Dynamic Range Imaging: Dataset, Methods and Results

This paper reports on the NTIRE 2021 challenge on perceptual image quality assessment (IQA), held in conjunction with the New Trends in Image Restoration and Enhancement workshop (NTIRE) workshop at CVPR 2021. As a new type of image processing technology, perceptual image processing algorithms based on Generative Adversarial Networks (GAN) have produced images with more realistic textures. These output images have completely different characteristics from traditional distortions, thus pose a new challenge for IQA methods to evaluate their visual quality. In comparison with previous IQA challenges, the training and testing datasets in this challenge include the outputs of perceptual image processing algorithms and the corresponding subjective scores. Thus they can be used to develop and evaluate IQA methods on GAN-based distortions. The challenge has 270 registered participants in total. In the final testing stage, 13 participating teams submitted their models and fact sheets. Almost all of them have achieved much better results than existing IQA methods, while the winning method can demonstrate state-of-the-art performance.

/pdf/ntire-2021-challenge-on-perceptual-image-quality-assessment-18h691tz8u.pdf

NTIRE 2021 Challenge on Perceptual Image Quality Assessment

In this paper, we introduce the first Challenge on Multi-modal Aerial View Object Classification (MAVOC) in con-junction with the NTIRE 2021 workshop at CVPR. This challenge is composed of two different tracks using EO and SAR imagery. Both EO and SAR sensors possess different advantages and drawbacks. The purpose of this competition is to analyze how to use both sets of sensory information in complementary ways. We discuss the top methods submitted for this competition and evaluate their results on our blind test set. Our challenge results show significant improvement of more than 15% accuracy from our current baselines for each track of the competition.

/pdf/ntire-2021-multi-modal-aerial-view-object-classification-2q4v58tfmu.pdf

NTIRE 2021 Multi-modal Aerial View Object Classification Challenge

This paper reviews the NTIRE 2021 challenge on learning the super-Resolution space. It focuses on the participating methods and final results. The challenge addresses the problem of learning a model capable of predicting the space of plausible super-resolution (SR) images, from a single low-resolution image. The model must thus be capable of sampling diverse outputs, rather than just generating a single SR image. The goal of the challenge is to spur research into developing learning formulations and models better suited for the highly ill-posed SR problem. And thereby advance the state-of-the-art in the broader SR field. In order to evaluate the quality of the predicted SR space, we propose a new evaluation metric and perform a comprehensive analysis of the participating methods. The challenge contains two tracks: 4× and 8 scale factor. In total, 11 teams competed in the final testing× phase.

/pdf/ntire-2021-learning-the-super-resolution-space-challenge-2rbcxvbz27.pdf

NTIRE 2021 Learning the Super-Resolution Space Challenge

Depth-guided asymmetric CycleGAN for rain synthesis and image deraining

In the Face Super-Resolution (FSR) task, it is important to precisely recover facial textures while maintaining facial contours for realistic high resolution faces. Although several CNN-based FSR methods have achieved great performance, they fail in restoring the facial contours due to the limitation of local convolutions. In contrast, Transformer-based methods which use self-attention as the basic component, are expert in modeling long-range dependencies between image patches. However, learning long-range dependencies often deteriorates facial textures due to the lack of locality. Therefore, a question is naturally raised: <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">how to effectively combine the superiority of CNN and Transformer for better reconstructing faces?</i> To address this issue, we propose an Efficient Latent Style guided Transformer-CNN framework for FSR called <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ELSFace</b> , which can sufficiently integrate the advantages of CNN and Transformer. The framework consists of a Feature Preparation Stage and a Feature Carving Stage. Basic facial contours and textures are generated in the Feature Preparation Stage, and separately guided by latent styles, so that facial details are better represented in reconstruction. CNN and Transformer streams in the Feature Carving Stage are used to individually restore facial textures and facial contours, respectively in a parallel recursive way. Considering the negligence of high-frequency features when learning the long-range dependencies, we design the High-Frequency Enhancement Block (HFEB) in the Transformer stream. The Sharp Loss is also proposed for better perceptual quality in optimization. Extensive experimental results demonstrate that our ELSFace can achieve the best results among all metrics compared to the state-of-the-art CNN and Transformer-based methods on commonly used datasets and real-world tasks. Meanwhile, our ELSFace method has the least model parameters and running time. The codes are released at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/FVL2020/ELSFace</uri> . 

Zhi Jin

Papers

NTIRE 2021 Learning the Super-Resolution Space Challenge

Depth-guided asymmetric CycleGAN for rain synthesis and image deraining

An Efficient Latent Style Guided Transformer-CNN Framework for Face Super-Resolution

V-DixMatch: A Semi-Supervised Learning Method for Human Action Recognition in Night Video Sensing

Self-distillation framework for indoor and outdoor monocular depth estimation