A general framework for image fusion based on multi-scale transform and sparse representation

In this article, we review recent uses of convolutional neural networks (CNNs) to solve inverse problems in imaging. It has recently become feasible to train deep CNNs on large databases of images, and they have shown outstanding performance on object classification and segmentation tasks. Motivated by these successes, researchers have begun to apply CNNs to the resolution of inverse problems such as denoising, deconvolution, superresolution, and medical image reconstruction, and they have started to report improvements over state-of-the-art methods, including sparsity-based techniques such as compressed sensing. Here, we review the recent experimental work in these areas, with a focus on the critical design decisions.

/pdf/convolutional-neural-networks-for-inverse-problems-in-50y28exfec.pdf

Convolutional Neural Networks for Inverse Problems in Imaging: A Review

Deep learning for pixel-level image fusion: Recent advances and future prospects

In this survey paper, we review recent uses of convolution neural networks (CNNs) to solve inverse problems in imaging. It has recently become feasible to train deep CNNs on large databases of images, and they have shown outstanding performance on object classification and segmentation tasks. Motivated by these successes, researchers have begun to apply CNNs to the resolution of inverse problems such as denoising, deconvolution, super-resolution, and medical image reconstruction, and they have started to report improvements over state-of-the-art methods, including sparsity-based techniques such as compressed sensing. Here, we review the recent experimental work in these areas, with a focus on the critical design decisions: Where does the training data come from? What is the architecture of the CNN? and How is the learning problem formulated and solved? We also bring together a few key theoretical papers that offer perspective on why CNNs are appropriate for inverse problems and point to some next steps in the field.

/pdf/a-review-of-convolutional-neural-networks-for-inverse-u3g8iywd5g.pdf

A Review of Convolutional Neural Networks for Inverse Problems in Imaging

Sparse representation based multi-sensor image fusion for multi-focus and multi-modality images: A review

Video superresolution (SR) techniques are of essential usages for high-resolution display devices due to the current lack of high-resolution videos. Although many algorithms have been proposed, video SR still remains a very challenging inverse problem under different conditions. In this paper, we propose a new method for video SR named motion compensation and residual net (MCResNet). We use optical flow algorithm for motion estimation and motion compensation as a preprocessing step. Then, we employ a novel deep residual convolutional neural network (CNN) to predict a high-resolution image using multiple motion compensated observations. The new residual CNN model preserves the low-frequency contents and facilitates the restoration of high-frequency details. Our method is able to handle large and complex motions adaptively. Extensive experimental results validate that our proposed method outperforms state-of-the-art single-image-based and multi-frame-based algorithms for video SR quantitatively and qualitatively.

Video Superresolution via Motion Compensation and Deep Residual Learning

Video super-resolution (SR) aims at restoring fine details and enhancing visual experience for low-resolution videos. In this paper, we propose a very deep non-simultaneous fully recurrent convolutional network for video SR. To make full use of temporal information, we employ motion compensation, very deep fully recurrent convolutional layers, and late fusion in our system. Residual connection is also employed in our recurrent structure for more accurate SR. Finally, a new model ensemble strategy is used to combine our method with a single-image SR method. Experimental results demonstrate that the proposed method is better than that of the state-of-the-art SR methods on quantitative visual quality assessment.

Video Super-Resolution Using Non-Simultaneous Fully Recurrent Convolutional Network

Pan-sharpening is an important remote sensing image pre-processing technique, which aims at obtaining a high-resolution multispectral (HRM) image by integrating the spectral information of a low-resolution multispectral (LRM) image and the spatial details of a high-resolution panchromatic (HRP) image. This paper proposes a new pan-sharpening method with sparse representation (SR) under the framework of wavelet transform. First, the wavelet transform is applied to the HRP image and the intensity component of LRM image. Then, the low-frequency components are fused based on SR to extract the spatial details in the HRP image as much as possible, and the dictionary is simply learned from high-quality nature images. Moreover, a novel strategy is also proposed to preserve the spectral information in the LRM image. On the other hand, the “numerous-but-sparse” high-frequency components are merged based on the local wavelet energy, which makes the algorithm more efficient than traditional SR-based methods. Finally, the fused result is obtained by performing inverse wavelet transform and inverse IHS transform. Experiments on WorldView-2 images demonstrate that the proposed method gives more spatial details and less spectral distortion compared with some conventional methods in terms of both visual quality and objective measurements.

A practical pan-sharpening method with wavelet transform and sparse representation

In this paper, we present a novel medical image fusion method by taking the complementary advantages of two powerful image representation theories: nonsubsampled contourlet transform (NSCT) and sparse representation (SR). In our fusion algorithm, the NSCT is firstly performed on each of the pre-registered source images to obtain the low-pass and high-pass coefficients. Then, the low-pass bands are merged with a SR-based fusion approach, and the high-pass bands are fused by employing the absolute values of coefficients as activity level measurement. Finally, the fused image is obtained by performing inverse NSCT on the merged coefficients. Several sets of medical source images with different combinations of modalities are used to test the effectiveness of the proposed method. Experimental results demonstrate that our method owns clear advantages over the fusion method based on NSCT or SR individually in terms of both visual quality and objective assessments.

Medical Image Fusion by Combining Nonsubsampled Contourlet Transform and Sparse Representation

Sparse representation (SR) has been widely used in many image processing applications including image fusion. As the contents vary significantly across different images, a highly redundant dictionary is always required in the sparse model, which reduces the algorithm stability and efficiency. This paper proposes a multi-focus image fusion method based on SR with adaptive sparse domain selection (SR-ASDS). Under SR-ASDS, numerous high-quality image patches are first classified into several categories according to their gradient information, and each category is applied into training a compact sub-dictionary. At the fusion process, a corresponding sub-dictionary is adaptively selected for a given pair of source image patches. Moreover, we present a general optimization framework for the merging rule design of the SR based image fusion. Numerous experiments on both clear images and the noisy ones demonstrate that the proposed method outperforms the fusion methods which use a single dictionary, in terms of several popular objective evaluation criteria.

Zengfu Wang

Papers

Video Superresolution via Motion Compensation and Deep Residual Learning

Video Super-Resolution Using Non-Simultaneous Fully Recurrent Convolutional Network

A practical pan-sharpening method with wavelet transform and sparse representation

Medical Image Fusion by Combining Nonsubsampled Contourlet Transform and Sparse Representation

Multi-focus Image Fusion Based on Sparse Representation with Adaptive Sparse Domain Selection