Image super-resolution is one of the most popular computer vision problems with many important applications to mobile devices. While many solutions have been proposed for this task, they are usually not optimized even for common smartphone AI hardware, not to mention more constrained smart TV platforms that are often supporting INT8 inference only. To address this problem, we introduce the first Mobile AI challenge, where the target is to develop an end-to-end deep learning-based image super-resolution solutions that can demonstrate a real-time performance on mobile or edge NPUs. For this, the participants were provided with the DIV2K dataset and trained quantized models to do an efficient 3X image upscaling. The runtime of all models was evaluated on the Synaptics VS680 Smart Home board with a dedicated NPU capable of accelerating quantized neural networks. The proposed solutions are fully compatible with all major mobile AI accelerators and are capable of reconstructing Full HD images under 40-60 ms while achieving high fidelity results. A detailed description of all models developed in the challenge is provided in this paper.

/pdf/real-time-quantized-image-super-resolution-on-mobile-npus-2py2h7xu51.pdf

Real-Time Quantized Image Super-Resolution on Mobile NPUs, Mobile AI 2021 Challenge: Report

Video super-resolution has recently become one of the most important mobile-related problems due to the rise of video communication and streaming services. While many solutions have been proposed for this task, the majority of them are too computationally expensive to run on portable devices with limited hardware resources. To address this problem, we introduce the first Mobile AI challenge, where the target is to develop an end-to-end deep learning-based video super-resolution solutions that can achieve a real-time performance on mobile GPUs. The participants were provided with the REDS dataset and trained their models to do an efficient 4X video upscaling. The runtime of all models was evaluated on the OPPO Find X2 smartphone with the Snapdragon 865 SoC capable of accelerating floating-point networks on its Adreno GPU. The proposed solutions are fully compatible with any mobile GPU and can upscale videos to HD resolution at up to 80 FPS while demonstrating high fidelity results. A detailed description of all models developed in the challenge is provided in this paper.

/pdf/real-time-video-super-resolution-on-smartphones-with-deep-7oq4ldseqr.pdf

Real-Time Video Super-Resolution on Smartphones with Deep Learning, Mobile AI 2021 Challenge: Report

As the quality of mobile cameras starts to play a crucial role in modern smartphones, more and more attention is now being paid to ISP algorithms used to improve various perceptual aspects of mobile photos. In this Mobile AI challenge, the target was to develop an end-to-end deep learning-based image signal processing (ISP) pipeline that can replace classical hand-crafted ISPs and achieve nearly real-time performance on smartphone NPUs. For this, the participants were provided with a novel learned ISP dataset consisting of RAW-RGB image pairs captured with the Sony IMX586 Quad Bayer mobile sensor and a professional 102-megapixel medium format camera. The runtime of all models was evaluated on the MediaTek Dimensity 1000+ platform with a dedicated AI processing unit capable of accelerating both floating-point and quantized neural networks. The proposed solutions are fully compatible with the above NPU and are capable of processing Full HD photos under 60-100 milliseconds while achieving high fidelity results. A detailed description of all models developed in this challenge is provided in this paper.

/pdf/learned-smartphone-isp-on-mobile-npus-with-deep-learning-1g8vf668cf.pdf

Learned Smartphone ISP on Mobile NPUs with Deep Learning, Mobile AI 2021 Challenge: Report

Image denoising is one of the most critical problems in mobile photo processing. While many solutions have been proposed for this task, they are usually working with synthetic data and are too computationally expensive to run on mobile devices. To address this problem, we introduce the first Mobile AI challenge, where the target is to develop an end-to-end deep learning-based image denoising solution that can demonstrate high efficiency on smartphone GPUs. For this, the participants were provided with a novel large-scale dataset consisting of noisy-clean image pairs captured in the wild. The runtime of all models was evaluated on the Samsung Exynos 2100 chipset with a powerful Mali GPU capable of accelerating floating-point and quantized neural networks. The proposed solutions are fully compatible with any mobile GPU and are capable of processing 480p resolution images under 40-80 ms while achieving high fidelity results. A detailed description of all models developed in the challenge is provided in this paper.

/pdf/fast-camera-image-denoising-on-mobile-gpus-with-deep-4t2z43muim.pdf

Fast Camera Image Denoising on Mobile GPUs with Deep Learning, Mobile AI 2021 Challenge: Report

Image relighting is attracting increasing interest due to its various applications. From a research perspective, im-age relighting can be exploited to conduct both image normalization for domain adaptation, and also for data augmentation. It also has multiple direct uses for photo montage and aesthetic enhancement. In this paper, we review the NTIRE 2021 depth guided image relighting challenge.We rely on the VIDIT dataset for each of our two challenge tracks, including depth information. The first track is on one-to-one relighting where the goal is to transform the illumination setup of an input image (color temperature and light source position) to the target illumination setup. In the second track, the any-to-any relighting challenge, the objective is to transform the illumination settings of the in-put image to match those of another guide image, similar to style transfer. In both tracks, participants were given depth information about the captured scenes. We had nearly 250 registered participants, leading to 18 confirmed team sub-missions in the final competition stage. The competitions, methods, and final results are presented in this paper.

/pdf/ntire-2021-depth-guided-image-relighting-challenge-26esz7jp2m.pdf

NTIRE 2021 Depth Guided Image Relighting Challenge

Light is a mixture of multiple spectral components. An image is a response of the scene with respect to these spectra. Principal components are more compact representation of the data compared to any other representations. Hence accuracy of the estimated defocus parameter is higher in principal component representation than any other customary used representations. In this paper, we present comparison between principal component representation and customary gray scale representation for depth map creation. The presented results shows that the depth maps obtained using principal component are smoother than depth maps obtained using gray scale representation. Besides that, the noise estimation using principal components is much more accurate than using Wiener strategy.

Defocus map estimation from a single image using principal components

Compression of multimedia content is an important processing step and backbone of real life applications in terms of optimum resource utilization in transmission and storage. It is an established field of research with very little scope for further improvement in achieved compression through customary coding-based compression techniques. Consequently, non-customary compression methods have become an important area for future research. Based on the principle ‘Any information that can be restored can be compressed’, we propose a novel spectrum-based image compression technique to further reduce the data footprint with satisfactory quality metric for images. We first blur the image with a point spread function (PSF) determined using frequency content of the given image. Blurring increases the DC component in the image, which in turn gets further compressed compared with original image by DCT-based JPEG compression. To recover the image, we perform deconvolution using the known blur PSF. Results obtained show further improvement of $$20-30\%$$
 in achieved compression with respect to original JPEG compressed image with satisfactory quality of recovered image.

A novel non-customary method of image compression based on image spectrum

Custom and natural lighting conditions can be emulated in images of the scene during post-editing. Extraordinary capabilities of the deep learning framework can be utilized for such purpose. Deep image relighting allows automatic photo enhancement by illumination-specific retouching. Most of the state-of-the-art methods for relighting are run-time intensive and memory inefficient. In this paper, we propose an efficient, real-time framework Deep Stacked Relighting Network (DSRN) for image relighting by utilizing the aggregated features from input image at different scales. Our model is very lightweight with total size of about 42 MB and has an average inference time of about 0.0116s for image of resolution $1024 \times 1024$ which is faster as compared to other multi-scale models. Our solution is quite robust for translating image color temperature from input image to target image and also performs moderately for light gradient generation with respect to the target image. Additionally, we show that if images illuminated from opposite directions are used as input, the qualitative results improve over using a single input image.

Dsrn: an Efficient Deep Network for Image Relighting

Conventionally, the point spread function (PSF) is understood as a characteristic function of any optical system. It captures the information about the amount of blur present along all the directions for a point in the scene. However, the dependence of blur on the PSF is in the form of convolution for any object other than a point source present in the scene and hence their relationship is less explicit. The authors propose a blur parameter locus curve (BPLC) as a system representation which has a one to one relationship with blur. BPLC simply is a chart of blur amounts in all directions of a given PSF with respect to the selected measurement function. They further characterise the PSF by decomposing the variation of BPLC across all directions based on the study performed for different possible forms of the blur kernels. Such decomposition provides powerful tools for various analysis. As PSF can be anisotropic, the computation of BPLC becomes an essential intermediate step to obtain the scale map as at the same scale, blur is different in different directions. Furthermore, they demonstrate the use of BPLC to obtain other system characteristics function such as PSF.

Blur parameter locus curve and its applications

Focal Plane Ambiguity (FPA) is a fundamental limitation of the Depth from Defocus (DFD) technique and refers to ambiguity of two possible distances corresponding to a single defocus blur value. Since, mix-sided scenes exist frequently in images and image-sequences, the assumption of a one sided focused scene often does not hold true. This leads to errors in the estimated defocus map. However, the inherent ordering of defocus blurs at the edges due to chromatic aberration in the R, G and B color planes can be used to correct this ambiguity. But, in highly defocused regions the ordering of defocus blurs becomes unreliable as the detection of edges becomes erroneous. In this paper, we propose a novel region based method using Color Uniformity Principle (CUP) for detecting the ordering of defocus blurs in R, G and B color planes to resolve the FPA.

Himanshu Kumar

Papers

Defocus map estimation from a single image using principal components

A novel non-customary method of image compression based on image spectrum

Dsrn: an Efficient Deep Network for Image Relighting

Blur parameter locus curve and its applications

Resolving Focal Plane Ambiguity using Chromatic Aberration and Color Uniformity Principle