Neural networks are becoming central in several areas of computer vision and image processing and different architectures have been proposed to solve specific problems. The impact of the loss layer of neural networks, however, has not received much attention in the context of image processing: the default and virtually only choice is $\ell _2$ . In this paper, we bring attention to alternative choices for image restoration. In particular, we show the importance of perceptually-motivated losses when the resulting image is to be evaluated by a human observer. We compare the performance of several losses, and propose a novel, differentiable error function. We show that the quality of the results improves significantly with better loss functions, even when the network architecture is left unchanged.

Loss Functions for Image Restoration With Neural Networks

Sparse representation models code an image patch as a linear combination of a few atoms chosen out from an over-complete dictionary, and they have shown promising results in various image restoration applications. However, due to the degradation of the observed image (e.g., noisy, blurred, and/or down-sampled), the sparse representations by conventional models may not be accurate enough for a faithful reconstruction of the original image. To improve the performance of sparse representation-based image restoration, in this paper the concept of sparse coding noise is introduced, and the goal of image restoration turns to how to suppress the sparse coding noise. To this end, we exploit the image nonlocal self-similarity to obtain good estimates of the sparse coding coefficients of the original image, and then centralize the sparse coding coefficients of the observed image to those estimates. The so-called nonlocally centralized sparse representation (NCSR) model is as simple as the standard sparse representation model, while our extensive experiments on various types of image restoration problems, including denoising, deblurring and super-resolution, validate the generality and state-of-the-art performance of the proposed NCSR algorithm.

/pdf/nonlocally-centralized-sparse-representation-for-image-ibijdq0ay0.pdf

Nonlocally Centralized Sparse Representation for Image Restoration

Model-based optimization methods and discriminative learning methods have been the two dominant strategies for solving various inverse problems in low-level vision. Typically, those two kinds of methods have their respective merits and drawbacks, e.g., model-based optimization methods are flexible for handling different inverse problems but are usually time-consuming with sophisticated priors for the purpose of good performance, in the meanwhile, discriminative learning methods have fast testing speed but their application range is greatly restricted by the specialized task. Recent works have revealed that, with the aid of variable splitting techniques, denoiser prior can be plugged in as a modular part of model-based optimization methods to solve other inverse problems (e.g., deblurring). Such an integration induces considerable advantage when the denoiser is obtained via discriminative learning. However, the study of integration with fast discriminative denoiser prior is still lacking. To this end, this paper aims to train a set of fast and effective CNN (convolutional neural network) denoisers and integrate them into model-based optimization method to solve other inverse problems. Experimental results demonstrate that the learned set of denoisers can not only achieve promising Gaussian denoising results but also can be used as prior to deliver good performance for various low-level vision applications.

/pdf/learning-deep-cnn-denoiser-prior-for-image-restoration-2yp6tg9rtn.pdf

Learning Deep CNN Denoiser Prior for Image Restoration

Model-based optimization methods and discriminative learning methods have been the two dominant strategies for solving various inverse problems in low-level vision. Typically, those two kinds of methods have their respective merits and drawbacks, e.g., model-based optimization methods are flexible for handling different inverse problems but are usually time-consuming with sophisticated priors for the purpose of good performance; in the meanwhile, discriminative learning methods have fast testing speed but their application range is greatly restricted by the specialized task. Recent works have revealed that, with the aid of variable splitting techniques, denoiser prior can be plugged in as a modular part of model-based optimization methods to solve other inverse problems (e.g., deblurring). Such an integration induces considerable advantage when the denoiser is obtained via discriminative learning. However, the study of integration with fast discriminative denoiser prior is still lacking. To this end, this paper aims to train a set of fast and effective CNN (convolutional neural network) denoisers and integrate them into model-based optimization method to solve other inverse problems. Experimental results demonstrate that the learned set of denoisers not only achieve promising Gaussian denoising results but also can be used as prior to deliver good performance for various low-level vision applications.

This paper describes a recently created image database, TID2013, intended for evaluation of full-reference visual quality assessment metrics. With respect to TID2008, the new database contains a larger number (3000) of test images obtained from 25 reference images, 24 types of distortions for each reference image, and 5 levels for each type of distortion. Motivations for introducing 7 new types of distortions and one additional level of distortions are given; examples of distorted images are presented. Mean opinion scores (MOS) for the new database have been collected by performing 985 subjective experiments with volunteers (observers) from five countries (Finland, France, Italy, Ukraine, and USA). The availability of MOS allows the use of the designed database as a fundamental tool for assessing the effectiveness of visual quality. Furthermore, existing visual quality metrics have been tested with the proposed database and the collected results have been analyzed using rank order correlation coefficients between MOS and considered metrics. These correlation indices have been obtained both considering the full set of distorted images and specific image subsets, for highlighting advantages and drawbacks of existing, state of the art, quality metrics. Approaches to thorough performance analysis for a given metric are presented to detect practical situations or distortion types for which this metric is not adequate enough to human perception. The created image database and the collected MOS values are freely available for downloading and utilization for scientific purposes. We have created a new large database.This database contains larger number of distorted images and distortion types.MOS values for all images are obtained and provided.Analysis of correlation between MOS and a wide set of existing metrics is carried out.Methodology for determining drawbacks of existing visual quality metrics is described.

https://hal.archives-ouvertes.fr/hal-01109219/document

Image database TID2013

A family of the block matching 3-D (BM3D) algorithms for various imaging problems has been recently proposed within the framework of nonlocal patchwise image modeling , . In this paper, we construct analysis and synthesis frames, formalizing BM3D image modeling, and use these frames to develop novel iterative deblurring algorithms. We consider two different formulations of the deblurring problem, i.e., one given by the minimization of the single-objective function and another based on the generalized Nash equilibrium (GNE) balance of two objective functions. The latter results in the algorithm where deblurring and denoising operations are decoupled. The convergence of the developed algorithms is proved. Simulation experiments show that the decoupled algorithm derived from the GNE formulation demonstrates the best numerical and visual results and shows superiority with respect to the state of the art in the field, confirming a valuable potential of BM3D-frames as an advanced image modeling tool.

BM3D Frames and Variational Image Deblurring

A family of the Block Matching 3-D (BM3D) algorithms for various imaging problems has been recently proposed within the framework of nonlocal patch-wise image modeling [1], [2]. In this paper we construct analysis and synthesis frames, formalizing the BM3D image modeling and use these frames to develop novel iterative deblurring algorithms. We consider two different formulations of the deblurring problem: one given by minimization of the single objective function and another based on the Nash equilibrium balance of two objective functions. The latter results in an algorithm where the denoising and deblurring operations are decoupled. The convergence of the developed algorithms is proved. Simulation experiments show that the decoupled algorithm derived from the Nash equilibrium formulation demonstrates the best numerical and visual results and shows superiority with respect to the state of the art in the field, confirming a valuable potential of BM3D-frames as an advanced image modeling tool.

Color image reconstruction from noisy color filter array (CFA) data is considered. A modification of the Block Matching 3D (BM3D) [2] filter for CFA data denoising utilizing cross-color correlations is proposed. Denoised images are then demosaicked by algorithms developed for noise-free data leading to state-of-the-art performance for both Gaussian and Poissonian noise models.

/pdf/cross-color-bm3d-filtering-of-noisy-raw-data-1ulldiih5q.pdf

Cross-color BM3D filtering of noisy raw data

We consider the estimation of the variance of an additive white Gaussian noise corrupting an image.

/pdf/noise-variance-estimation-in-nonlocal-transform-domain-28fdcc194q.pdf

Noise variance estimation in nonlocal transform domain

1.1 Introduction : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 4 1.2 Iterative ltering as regularization : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 5 1.2.1 Spectral decomposition of the operator : : : : : : : : : : : : : : : : : : : : : : : : : : : 6 1.2.2 Non-local transform domain ltering : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 6 1.3 Compressed sensing : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 8 1.3.1 Observation model and notation : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 9 1.3.2 Iterative algorithm with stochastic approximation : : : : : : : : : : : : : : : 10 1.3.2.1 Comments to the algorithm : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 11 1.3.3 Experiments : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 13 1.3.3.1 Radon inversion from sparse projections : : : : : : : : : : : : : : : : 13 1.3.3.2 Limited-angle tomography : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 14 1.3.3.3 Reconstruction from low-frequency data : : : : : : : : : : : : : : : : 15 1.4 Super-resolution : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 16 1.4.1 Spectral decomposition for the super-resolution problem : : : : : : : : 18 1.4.2 Observation model : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 20 1.4.3 Scaling family of transforms : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 20 1.4.4 Multistage iterative reconstruction : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 21 1.4.5 Experiments : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 23 1.4.5.1 Implementation details : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 23 1.4.5.2 Super-resolution : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 24 1.4.5.3 Image upsampling : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 25 1.5 Conclusions : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 25

Aram Danielyan

Papers

BM3D Frames and Variational Image Deblurring

BM3D Frames and Variational Image Deblurring

Cross-color BM3D filtering of noisy raw data

Noise variance estimation in nonlocal transform domain

Spatially Adaptive Filtering as Regularization in Inverse Imaging: Compressive Sensing, Super-Resolution, and Upsampling