Image super-resolution
TL;DR: This paper aims to provide a review of SR from the perspective of techniques and applications, and especially the main contributions in recent years, and discusses the current obstacles for future research.
About: This article is published in Signal Processing.The article was published on 2016-11-01. It has received 378 citations till now.
Citations
More filters
TL;DR: In this article, the authors provide a short overview of recent advances and some associated challenges in machine learning applied to medical image processing and image analysis, and provide a starting point for people interested in experimenting and perhaps contributing to the field of machine learning for medical imaging.
Abstract: What has happened in machine learning lately, and what does it mean for the future of medical image analysis? Machine learning has witnessed a tremendous amount of attention over the last few years. The current boom started around 2009 when so-called deep artificial neural networks began outperforming other established models on a number of important benchmarks. Deep neural networks are now the state-of-the-art machine learning models across a variety of areas, from image analysis to natural language processing, and widely deployed in academia and industry. These developments have a huge potential for medical imaging technology, medical data analysis, medical diagnostics and healthcare in general, slowly being realized. We provide a short overview of recent advances and some associated challenges in machine learning applied to medical image processing and image analysis. As this has become a very broad and fast expanding field we will not survey the entire landscape of applications, but put particular focus on deep learning in MRI.
Our aim is threefold: (i) give a brief introduction to deep learning with pointers to core references; (ii) indicate how deep learning has been applied to the entire MRI processing chain, from acquisition to image retrieval, from segmentation to disease prediction; (iii) provide a starting point for people interested in experimenting and perhaps contributing to the field of machine learning for medical imaging by pointing out good educational resources, state-of-the-art open-source code, and interesting sources of data and problems related medical imaging.
991 citations
TL;DR: This paper indicates how deep learning has been applied to the entire MRI processing chain, from acquisition to image retrieval, from segmentation to disease prediction, and provides a starting point for people interested in experimenting and contributing to the field of deep learning for medical imaging.
Abstract: What has happened in machine learning lately, and what does it mean for the future of medical image analysis? Machine learning has witnessed a tremendous amount of attention over the last few years. The current boom started around 2009 when so-called deep artificial neural networks began outperforming other established models on a number of important benchmarks. Deep neural networks are now the state-of-the-art machine learning models across a variety of areas, from image analysis to natural language processing, and widely deployed in academia and industry. These developments have a huge potential for medical imaging technology, medical data analysis, medical diagnostics and healthcare in general, slowly being realized. We provide a short overview of recent advances and some associated challenges in machine learning applied to medical image processing and image analysis. As this has become a very broad and fast expanding field we will not survey the entire landscape of applications, but put particular focus on deep learning in MRI. Our aim is threefold: (i) give a brief introduction to deep learning with pointers to core references; (ii) indicate how deep learning has been applied to the entire MRI processing chain, from acquisition to image retrieval, from segmentation to disease prediction; (iii) provide a starting point for people interested in experimenting and perhaps contributing to the field of deep learning for medical imaging by pointing out good educational resources, state-of-the-art open-source code, and interesting sources of data and problems related medical imaging.
590 citations
Cites background from "Image super-resolution"
...Image super-resolution, reconstructing a higher-resolution image or image sequence from the observed low-resolution image [190], is an exciting application of deep learning methods....
[...]
TL;DR: In this article, a test-time augmentation-based aleatoric uncertainty was proposed to analyze the effect of different transformations of the input image on the segmentation output, and the results showed that the proposed test augmentation provides a better uncertainty estimation than calculating the testtime dropout-based model uncertainty alone and helps to reduce overconfident incorrect predictions.
305 citations
TL;DR: The proposed integrated fusion framework can achieve the integrated fusion of multisource observations to obtain high spatio-temporal-spectral resolution images, without limitations on the number of remote sensing sensors.
Abstract: Remote sensing satellite sensors feature a tradeoff between the spatial, temporal, and spectral resolutions. In this paper, we propose an integrated framework for the spatio–temporal–spectral fusion of remote sensing images. There are two main advantages of the proposed integrated fusion framework: it can accomplish different kinds of fusion tasks, such as multiview spatial fusion, spatio–spectral fusion, and spatio–temporal fusion, based on a single unified model, and it can achieve the integrated fusion of multisource observations to obtain high spatio–temporal–spectral resolution images, without limitations on the number of remote sensing sensors. The proposed integrated fusion framework was comprehensively tested and verified in a variety of image fusion experiments. In the experiments, a number of different remote sensing satellites were utilized, including IKONOS, the Enhanced Thematic Mapper Plus (ETM+), the Moderate Resolution Imaging Spectroradiometer (MODIS), the Hyperspectral Digital Imagery Collection Experiment (HYDICE), and Systeme Pour l' Observation de la Terre-5 (SPOT-5). The experimental results confirm the effectiveness of the proposed method.
240 citations
TL;DR: This letter proposes a new single-image super-resolution algorithm named local–global combined networks (LGCNet) for remote sensing images based on the deep CNNs, elaborately designed with its “multifork” structure to learn multilevel representations ofRemote sensing images including both local details and global environmental priors.
Abstract: Super-resolution is an image processing technology that recovers a high-resolution image from a single or sequential low-resolution images Recently deep convolutional neural networks (CNNs) have made a huge breakthrough in many tasks including super-resolution In this letter, we propose a new single-image super-resolution algorithm named local–global combined networks (LGCNet) for remote sensing images based on the deep CNNs Our LGCNet is elaborately designed with its “multifork” structure to learn multilevel representations of remote sensing images including both local details and global environmental priors Experimental results on a public remote sensing data set (UC Merced) demonstrate an overall improvement of both accuracy and visual performance over several state-of-the-art algorithms
203 citations
Cites background from "Image super-resolution"
...Instead of devoting to physical imaging technology, many researchers aim to recover highresolution images from low-resolution ones using an image processing technology called super-resolution [1]....
[...]
References
More filters
TL;DR: This paper adapt and expand kernel regression ideas for use in image denoising, upscaling, interpolation, fusion, and more and establishes key relationships with some popular existing methods and shows how several of these algorithms are special cases of the proposed framework.
Abstract: In this paper, we make contact with the field of nonparametric statistics and present a development and generalization of tools and results for use in image processing and reconstruction. In particular, we adapt and expand kernel regression ideas for use in image denoising, upscaling, interpolation, fusion, and more. Furthermore, we establish key relationships with some popular existing methods and show how several of these algorithms, including the recently popularized bilateral filter, are special cases of the proposed framework. The resulting algorithms and analyses are amply illustrated with practical examples
1,457 citations
"Image super-resolution" refers background or methods in this paper
...Therefore, most of the recent representative arti cles about SR have focused on regularized frameworks [1,47,49,53,54,68,72,73]....
[...]
...[54] adapted kernel regression to multi-frame SR, and this method is able to handle video sequences with general motion models....
[...]
...The popular spatial domain methods include non-uniform interpolation [35], iterative back-projection (IBP) [56], projection onto convex sets (POCS) [57,63,70], the regularized methods [34,40,43,47,53,54,58,62], and a number of hybrid algorithms [71]....
[...]
TL;DR: A family of photo-switchable fluorescent probes is introduced and multicolor stochastic optical reconstruction microscopy (STORM) is demonstrated, to facilitate direct visualization of molecular interactions at the nanometer scale.
Abstract: Recent advances in far-field optical nanoscopy have enabled fluorescence imaging with a spatial resolution of 20 to 50 nanometers. Multicolor super-resolution imaging, however, remains a challenging task. Here, we introduce a family of photo-switchable fluorescent probes and demonstrate multicolor stochastic optical reconstruction microscopy (STORM). Each probe consists of a photo-switchable "reporter" fluorophore that can be cycled between fluorescent and dark states, and an "activator" that facilitates photo-activation of the reporter. Combinatorial pairing of reporters and activators allows the creation of probes with many distinct colors. Iterative, color-specific activation of sparse subsets of these probes allows their localization with nanometer accuracy, enabling the construction of a super-resolution STORM image. Using this approach, we demonstrate multicolor imaging of DNA model samples and mammalian cells with 20- to 30-nanometer resolution. This technique will facilitate direct visualization of molecular interactions at the nanometer scale.
1,438 citations
TL;DR: This work derives a sequence of analytical results which show that the reconstruction constraints provide less and less useful information as the magnification factor increases, and proposes a super-resolution algorithm which attempts to recognize local features in the low-resolution images and then enhances their resolution in an appropriate manner.
Abstract: Nearly all super-resolution algorithms are based on the fundamental constraints that the super-resolution image should generate low resolution input images when appropriately warped and down-sampled to model the image formation process. (These reconstruction constraints are normally combined with some form of smoothness prior to regularize their solution.) We derive a sequence of analytical results which show that the reconstruction constraints provide less and less useful information as the magnification factor increases. We also validate these results empirically and show that, for large enough magnification factors, any smoothness prior leads to overly smooth results with very little high-frequency content. Next, we propose a super-resolution algorithm that uses a different kind of constraint in addition to the reconstruction constraints. The algorithm attempts to recognize local features in the low-resolution images and then enhances their resolution in an appropriate manner. We call such a super-resolution algorithm a hallucination or reconstruction algorithm. We tried our hallucination algorithm on two different data sets, frontal images of faces and printed Roman text. We obtained significantly better results than existing reconstruction-based algorithms, both qualitatively and in terms of RMS pixel error.
1,418 citations
TL;DR: This topic can be viewed as an extension of spectral graph theory and the diffusion geometry framework to functional analysis and PDE-like evolutions to define new types of flows and functionals for image processing and elsewhere.
Abstract: We propose the use of nonlocal operators to define new types of flows and functionals for image processing and elsewhere. A main advantage over classical PDE-based algorithms is the ability to handle better textures and repetitive structures. This topic can be viewed as an extension of spectral graph theory and the diffusion geometry framework to functional analysis and PDE-like evolutions. Some possible applications and numerical examples are given, as is a general framework for approximating Hamilton–Jacobi equations on arbitrary grids in high demensions, e.g., for control theory.
1,397 citations
"Image super-resolution" refers methods in this paper
...Compared with the TV model, the nonlocal-based model can make use of more information, and can thus prevent the staircase effect in flat regions, and can help restore the details [97,113]....
[...]
01 Dec 2013
TL;DR: This paper proposes fast super-resolution methods while making no compromise on quality, and supports the use of sparse learned dictionaries in combination with neighbor embedding methods, and proposes the anchored neighborhood regression.
Abstract: Recently there have been significant advances in image up scaling or image super-resolution based on a dictionary of low and high resolution exemplars. The running time of the methods is often ignored despite the fact that it is a critical factor for real applications. This paper proposes fast super-resolution methods while making no compromise on quality. First, we support the use of sparse learned dictionaries in combination with neighbor embedding methods. In this case, the nearest neighbors are computed using the correlation with the dictionary atoms rather than the Euclidean distance. Moreover, we show that most of the current approaches reach top performance for the right parameters. Second, we show that using global collaborative coding has considerable speed advantages, reducing the super-resolution mapping to a precomputed projective matrix. Third, we propose the anchored neighborhood regression. That is to anchor the neighborhood embedding of a low resolution patch to the nearest atom in the dictionary and to precompute the corresponding embedding matrix. These proposals are contrasted with current state-of-the-art methods on standard images. We obtain similar or improved quality and one or two orders of magnitude speed improvements.
1,276 citations
"Image super-resolution" refers methods in this paper
...Other researchers have developed regressionbased methods and some improved algorithms, such as the SR method with sparse NE [156], image SR using nonlocal autoregressive modeling [159], and anchored neighborhood regression [171] for fast example-based SR....
[...]