Proceedings ArticleDOI

IRGUN : Improved Residue Based Gradual Up-Scaling Network for Single Image Super Resolution

18 Jun 2018-pp 834-843

TL;DR: A novel Improved Residual based Gradual Up-Scaling Network (IRGUN) to improve the quality of the super-resolved image for a large magnification factor and recovers fine details effectively at large (8X) magnification factors.

AbstractConvolutional neural network based architectures have achieved decent perceptual quality super resolution on natural images for small scaling factors (2X and 4X). However, image super-resolution for large magnication factors (8X) is an extremely challenging problem for the computer vision community. In this paper, we propose a novel Improved Residual based Gradual Up-Scaling Network (IRGUN) to improve the quality of the super-resolved image for a large magnification factor. IRGUN has a Gradual Upsampling and Residue-based Enhancment Network (GUREN) which comprises of series of Up-scaling and Enhancement blocks (UEB) connected end-to-end and fine-tuned together to give a gradual magnification and enhancement. Due to the perceptual importance of the luminance in super-resolution, the model is trained on luminance (Y) channel of the YCbCr image. Whereas, the chrominance components (Cb and Cr) channel are up-scaled using bicubic interpolation and combined with super-resolved Y channel of the image, which is then converted to RGB. A cascaded 3D-RED architecture trained on RGB images is utilized to incorporate its inter-channel correlation. In addition to this, the training methodology is also presented in the paper. In the training procedure, the weights of the previous UEB are used in the next immediate UEB for faster and better convergence. Each UEB is trained on its respective scale by taking the output image of the previous UEB as input and corresponding HR image of the same scale as ground truth to the successive UEB. All the UEBs are then connected end-to-end and fine tuned. The IRGUN recovers fine details effectively at large (8X) magnification factors. The efficiency of IRGUN is presented on various benchmark datasets and at different magnification scales.

...read more

Content maybe subject to copyright    Report

Citations
More filters
Proceedings ArticleDOI

18 Jun 2018
TL;DR: This paper reviews the 2nd NTIRE challenge on single image super-resolution (restoration of rich details in a low resolution image) with focus on proposed solutions and results and gauges the state-of-the-art in single imagesuper-resolution.
Abstract: This paper reviews the 2nd NTIRE challenge on single image super-resolution (restoration of rich details in a low resolution image) with focus on proposed solutions and results. The challenge had 4 tracks. Track 1 employed the standard bicubic downscaling setup, while Tracks 2, 3 and 4 had realistic unknown downgrading operators simulating camera image acquisition pipeline. The operators were learnable through provided pairs of low and high resolution train images. The tracks had 145, 114, 101, and 113 registered participants, resp., and 31 teams competed in the final testing phase. They gauge the state-of-the-art in single image super-resolution.

222 citations

Cites background from "IRGUN : Improved Residue Based Grad..."

• ...CEERI team proposed an improved residual based gradual upscaling network (IRGUN) [29]....

[...]

• ...The IRGUN has a series of up-scaling and enhancement blocks (UEB) connected end-to-end and fine-tuned together to give a gradual magnification and enhancement....

[...]

• ...Title:: Improved residual based gradual upscaling network(IRGUN) Members:Manoj Sharma, Rudrabha Mukhopadhyay, Avinash Upadhyay, Sriharsha Koundinya, Ankit Shukla, Santanu Chaudhury Affiliation: CSIR-CEERI, India...

[...]

Journal ArticleDOI
, Hai Min1
TL;DR: A fast image upsampling method designed specifically for industrial applications at low magnification that can obtain performance comparable to that of some state-of-the-art methods for 720P-to-1080P magnification, but the computational cost is much lower.
Abstract: In recent years, many deep-network-based super-resolution techniques have been proposed and have achieved impressive results for 2 ${\times }$ and higher magnification factors. However, lower magnification factors encountered in some industrial applications have not received special attention, such as 720P-to-1080P (1.5 ${\times }$ magnification). Compared to traditional 2 ${\times }$ or higher magnification factors, these lower magnifications are much simpler, but reconstructions of high-definition images are time-consuming and computationally complex. Hence, in this paper, a fast image upsampling method is designed specifically for industrial applications at low magnification. In the proposed method, edge and nonedge areas are first distinguished and then reconstructed via different fast approaches. For the edge area, a local edge pattern encoding-based method is presented to recover sharp edges. For the nonedge area, a global iterative reconstruction with texture constraint is utilized. Moreover, some acceleration strategies are also presented to further reduce the complexity. The experimental results demonstrate that the proposed method can obtain performance comparable to that of some state-of-the-art methods for 720P-to-1080P magnification, but the computational cost is much lower.

7 citations

Proceedings ArticleDOI
01 Mar 2020
TL;DR: This work proposes a divide and conquer approach based wide and deep network (WDN) that divides the 4× up-sampling problem into 32 disjoint subproblems that can be solved simultaneously and independently of each other.
Abstract: Divide and Conquer is a well-established approach in the literature that has efficiently solved a variety of problems. However, it is yet to be explored in full in solving image super-resolution. To predict a sharp up-sampled image, this work proposes a divide and conquer approach based wide and deep network (WDN) that divides the 4× up-sampling problem into 32 disjoint subproblems that can be solved simultaneously and independently of each other Half of these subproblems deal with predicting the overall features of the high-resolution image, while the remaining are exclusively for predicting the finer details. Additionally, a technique that is found to be more effective in calibrating the pixel intensities has been proposed. Results obtained on multiple datasets demonstrate the improved performance of the proposed wide and deep network over state-of-the-art methods.

6 citations

Cites background from "IRGUN : Improved Residue Based Grad..."

• ...For instance, [58, 32, 87, 33, 36, 58, 15, 1, 64, 43, 26, 16, 12, 70, 92, 40, 53, 79, 22, 57, 59, 4, 60, 78, 66] are some deep networks for super-resolution....

[...]

Book ChapterDOI
, Deepak1
17 Dec 2019
TL;DR: A novel light-weight architecture-Gradually growing Residual and self-Attention based Dense Deep Back Projection Network (GRAD-DBPN) for large scale image super-resolution (SR) and overcomes the issue of vanishing gradient.
Abstract: Due to the strong capacity of deep learning in handling unstructured data, it has been utilized for the task of single image super-resolution (SISR). These algorithms have shown promising results for small scale super-resolution but are not robust to large scale super-resolution. In addition, these algorithms are computationally complex and require high-end computational devices. Developing large-scale super-resolution framework finds its application in smart-phones as these devices have limited computational power. In this context, we present a novel light-weight architecture-Gradually growing Residual and self-Attention based Dense Deep Back Projection Network (GRAD-DBPN) for large scale image super-resolution (SR). The network is made of cascaded self-Attention based Residual Dense Deep Back Projection Network (ARD-DBPN) blocks to perform super-resolution gradually. Where each block performs 2X super-resolution and fine tuned in an end to end manner. The residual architecture facilitates the faster convergence of network and overcomes the issue of vanishing gradient. Experimental results on different benchmark data-set have been presented to compare the efficacy and effectiveness of the architecture.
Patent
22 May 2020
TL;DR: In this article, an image processing method and device, including electronic equipment and a storage medium is described, which includes the following steps: carrying out M-level feature extraction on a to-be-processed image to obtain Mlevel first feature maps of the image, wherein the scales of all levels of first features are different, and M is an integer greater than 1; carrying out scale adjustment and fusion on the feature map groups corresponding to the first features at all levels to obtain the M levels of second features.
Abstract: The invention relates to an image processing method and device, electronic equipment and a storage medium. The method comprises the following steps: carrying out M-level feature extraction on a to-be-processed image to obtain M-level first feature maps of the to-be-processed image, wherein the scales of all levels of first feature maps in the M-level first feature maps are different, and M is an integer greater than 1; carrying out scale adjustment and fusion on the feature map groups corresponding to the first feature maps at all levels to obtain M levels of second feature maps, wherein eachfeature map group comprises the first feature map and a first feature map adjacent to the first feature map; and performing target detection on the M-level second feature map to obtain a target detection result of the to-be-processed image. According to the embodiment of the invention, the target detection effect can be improved.

References
More filters
Proceedings Article
, Jimmy Ba2
01 Jan 2015
TL;DR: This work introduces Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments, and provides a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework.
Abstract: We introduce Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments. The method is straightforward to implement, is computationally efficient, has little memory requirements, is invariant to diagonal rescaling of the gradients, and is well suited for problems that are large in terms of data and/or parameters. The method is also appropriate for non-stationary objectives and problems with very noisy and/or sparse gradients. The hyper-parameters have intuitive interpretations and typically require little tuning. Some connections to related algorithms, on which Adam was inspired, are discussed. We also analyze the theoretical convergence properties of the algorithm and provide a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework. Empirical results demonstrate that Adam works well in practice and compares favorably to other stochastic optimization methods. Finally, we discuss AdaMax, a variant of Adam based on the infinity norm.

78,539 citations

"IRGUN : Improved Residue Based Grad..." refers methods in this paper

• ...We trained our model with Adam optimizer [13]....

[...]

Proceedings ArticleDOI
, D. Tal1
07 Jul 2001
TL;DR: In this paper, the authors present a database containing ground truth segmentations produced by humans for images of a wide variety of natural scenes, and define an error measure which quantifies the consistency between segmentations of differing granularities.
Abstract: This paper presents a database containing 'ground truth' segmentations produced by humans for images of a wide variety of natural scenes. We define an error measure which quantifies the consistency between segmentations of differing granularities and find that different human segmentations of the same image are highly consistent. Use of this dataset is demonstrated in two applications: (1) evaluating the performance of segmentation algorithms and (2) measuring probability distributions associated with Gestalt grouping factors as well as statistics of image region properties.

6,077 citations

Journal ArticleDOI
TL;DR: Zhang et al. as discussed by the authors proposed a deep learning method for single image super-resolution (SR), which directly learns an end-to-end mapping between the low/high-resolution images.
Abstract: We propose a deep learning method for single image super-resolution (SR). Our method directly learns an end-to-end mapping between the low/high-resolution images. The mapping is represented as a deep convolutional neural network (CNN) that takes the low-resolution image as the input and outputs the high-resolution one. We further show that traditional sparse-coding-based SR methods can also be viewed as a deep convolutional network. But unlike traditional methods that handle each component separately, our method jointly optimizes all layers. Our deep CNN has a lightweight structure, yet demonstrates state-of-the-art restoration quality, and achieves fast speed for practical on-line usage. We explore different network structures and parameter settings to achieve trade-offs between performance and speed. Moreover, we extend our network to cope with three color channels simultaneously, and show better overall reconstruction quality.

4,680 citations

Posted Content

TL;DR: SRGAN, a generative adversarial network (GAN) for image super-resolution (SR), is presented, to its knowledge, the first framework capable of inferring photo-realistic natural images for 4x upscaling factors and a perceptual loss function which consists of an adversarial loss and a content loss.
Abstract: Despite the breakthroughs in accuracy and speed of single image super-resolution using faster and deeper convolutional neural networks, one central problem remains largely unsolved: how do we recover the finer texture details when we super-resolve at large upscaling factors? The behavior of optimization-based super-resolution methods is principally driven by the choice of the objective function. Recent work has largely focused on minimizing the mean squared reconstruction error. The resulting estimates have high peak signal-to-noise ratios, but they are often lacking high-frequency details and are perceptually unsatisfying in the sense that they fail to match the fidelity expected at the higher resolution. In this paper, we present SRGAN, a generative adversarial network (GAN) for image super-resolution (SR). To our knowledge, it is the first framework capable of inferring photo-realistic natural images for 4x upscaling factors. To achieve this, we propose a perceptual loss function which consists of an adversarial loss and a content loss. The adversarial loss pushes our solution to the natural image manifold using a discriminator network that is trained to differentiate between the super-resolved images and original photo-realistic images. In addition, we use a content loss motivated by perceptual similarity instead of similarity in pixel space. Our deep residual network is able to recover photo-realistic textures from heavily downsampled images on public benchmarks. An extensive mean-opinion-score (MOS) test shows hugely significant gains in perceptual quality using SRGAN. The MOS scores obtained with SRGAN are closer to those of the original high-resolution images than to those obtained with any state-of-the-art method.

4,397 citations

"IRGUN : Improved Residue Based Grad..." refers background or methods or result in this paper

• ...Red shows highest while blue shows second highest Dataset Scale Bicubic SRGAN [15] VDSR [11] GUN [38] RDN [27] EDSR [28] LapSRN [14] IRGUN...

[...]

• ...They are, SRGAN [15], VDSR [11], GUN [36], RDN [40], EDSR [26] and LapSRN [1]....

[...]

• ...However, in some of the models the increment in scale is also done using convolutional layers [1, 15, 18, 25, 28, 30]....

[...]

• ...Among the above mentioned frameworks, SRGAN gives good perceptual quality while its PSNR and SSIM metrics are poor in comparison to other methods....

[...]

• ...We compare results with SRGAN [15], VDSR [11], GUN [36], RDN [40], EDSR [26] and LapSRN [1]....

[...]

Journal ArticleDOI
, Yi Ma2
TL;DR: This paper presents a new approach to single-image superresolution, based upon sparse signal representation, which generates high-resolution images that are competitive or even superior in quality to images produced by other similar SR methods.
Abstract: This paper presents a new approach to single-image superresolution, based upon sparse signal representation. Research on image statistics suggests that image patches can be well-represented as a sparse linear combination of elements from an appropriately chosen over-complete dictionary. Inspired by this observation, we seek a sparse representation for each patch of the low-resolution input, and then use the coefficients of this representation to generate the high-resolution output. Theoretical results from compressed sensing suggest that under mild conditions, the sparse representation can be correctly recovered from the downsampled signals. By jointly training two dictionaries for the low- and high-resolution image patches, we can enforce the similarity of sparse representations between the low-resolution and high-resolution image patch pair with respect to their own dictionaries. Therefore, the sparse representation of a low-resolution image patch can be applied with the high-resolution image patch dictionary to generate a high-resolution image patch. The learned dictionary pair is a more compact representation of the patch pairs, compared to previous approaches, which simply sample a large amount of image patch pairs , reducing the computational cost substantially. The effectiveness of such a sparsity prior is demonstrated for both general image super-resolution (SR) and the special case of face hallucination. In both cases, our algorithm generates high-resolution images that are competitive or even superior in quality to images produced by other similar SR methods. In addition, the local sparse modeling of our approach is naturally robust to noise, and therefore the proposed algorithm can handle SR with noisy inputs in a more unified framework.

4,389 citations

"IRGUN : Improved Residue Based Grad..." refers methods in this paper

• ...Reconstruction of missing information with known LR/HR example pair uses learning based methods such as neighbor embedding based methods [2], local self-exemplar methods and sparse representation based methods [34, 36]....

[...]

02 Nov 2016
Wang Haoqian
17 Nov 2017
Cong Peng
21 Dec 2018
Lu Xiaobo