Super resolution problems are widely discussed in medical imaging Spatial resolution of medical images are not sufficient due to the constraints such as image acquisition time, low irradiation dose or hardware limits To address these problems, different super resolution methods have been proposed, such as optimization or learning-based approaches Recently, deep learning methods become a thriving technology and are developing at an exponential speed We think it is necessary to write a review to present the current situation of deep learning in medical imaging super resolution In this paper, we first briefly introduce deep learning methods, then present a number of important deep learning approaches to solve super resolution problems, different architectures as well as up-sampling operations will be introduced Afterwards, we focus on the applications of deep learning methods in medical imaging super resolution problems, the challenges to overcome will be presented as well

A Review of the Deep Learning Methods for Medical Images Super Resolution Problems

Recently, convolutional neural networks (CNNs) have been successfully applied to many remote sensing problems. However, deep learning techniques for multi-image super-resolution (SR) from multitemporal unregistered imagery have received little attention so far. This article proposes a novel CNN-based technique that exploits both spatial and temporal correlations to combine multiple images. This novel framework integrates the spatial registration task directly inside the CNN, and allows one to exploit the representation learning capabilities of the network to enhance registration accuracy. The entire SR process relies on a single CNN with three main stages: shared 2-D convolutions to extract high-dimensional features from the input images; a subnetwork proposing registration filters derived from the high-dimensional feature representations; 3-D convolutions for slow fusion of the features from multiple images. The whole network can be trained end-to-end to recover a single high-resolution image from multiple unregistered low-resolution images. The method presented in this article is the winner of the PROBA-V SR challenge issued by the European Space Agency (ESA).

DeepSUM: Deep Neural Network for Super-Resolution of Unregistered Multitemporal Images

A global analysis of the temporal availability of PlanetScope high spatial resolution multi-spectral imagery

Recently, convolutional neural networks (CNN) have been successfully applied to many remote sensing problems. However, deep learning techniques for multi-image super-resolution from multitemporal unregistered imagery have received little attention so far. This work proposes a novel CNN-based technique that exploits both spatial and temporal correlations to combine multiple images. This novel framework integrates the spatial registration task directly inside the CNN, and allows to exploit the representation learning capabilities of the network to enhance registration accuracy. The entire super-resolution process relies on a single CNN with three main stages: shared 2D convolutions to extract high-dimensional features from the input images; a subnetwork proposing registration filters derived from the high-dimensional feature representations; 3D convolutions for slow fusion of the features from multiple images. The whole network can be trained end-to-end to recover a single high resolution image from multiple unregistered low resolution images. The method presented in this paper is the winner of the PROBA-V super-resolution challenge issued by the European Space Agency.

DeepSUM: Deep neural network for Super-resolution of Unregistered Multitemporal images

Super-resolution (SR) reconstruction is a process aimed at enhancing the spatial resolution of images, either from a single observation, based on the learned relation between low and high resolution, or from multiple images presenting the same scene. SR is particularly important, if it is not feasible to acquire images at the desired resolution, while there are single or many observations available at lower resolution—this is inherent to a variety of remote sensing scenarios. Recently, we have witnessed substantial improvement in single-image SR attributed to the use of deep neural networks for learning the relation between low and high resolution. Importantly, deep learning has not been widely exploited for multiple-image super-resolution, which benefits from information fusion and in general allows for achieving higher reconstruction accuracy. In this letter, we introduce a new approach to combine the advantages of multiple-image fusion with learning the low-to-high resolution mapping using deep networks. The results of our extensive experiments indicate that the proposed framework outperforms the state-of-the-art SR methods.

Deep Learning for Multiple-Image Super-Resolution

Evaluating super-resolution reconstruction of satellite images

Super-resolution reconstruction (SRR) allows for enhancing image spatial resolution from low-resolution (LR) observations, which are assumed to have been derived from a hypothetical high-resolution image by applying a certain imaging model (IM). However, if the actual degradation is different from the assumed IM, which is often the case in real-world scenarios, then the reconstruction quality is affected. We introduce a genetic algorithm to optimize the SRR hyper-parameters and to discover the actual IM by evolving the kernels exploited in the IM. The reported experimental results indicate that our approach outperforms the state of the art for a variety of images, including difficult real-life satellite data.

Evolving imaging model for super-resolution reconstruction

Detecting anomalies in telemetry data captured on-board satellites is a pivotal step towards their safe operation. The data-driven algorithms for this task are often heavily parameterized, and the incorrect hyperparameters can deteriorate their performance. We tackle this issue and introduce a genetic algorithm for evolving a dynamic thresholding approach that follows a long short-term memory network in an unsupervised anomaly detection system. Our experiments show that the genetic algorithm improves the abilities of a detector operating on multi-channel satellite telemetry.

Detecting anomalies in spacecraft telemetry using evolutionary thresholding and LSTMs

Recent advancements in single-image super-resolution reconstruction (SRR) are attributed primarily to convolutional neural networks (CNNs), which effectively learn the relation between low and high resolution and allow for obtaining high-quality reconstruction within seconds. SRR from multiple images benefits from information fusion, which improves the reconstruction outcome compared with example-based methods. On the other hand, multiple-image SRR is computationally more demanding, mainly due to required subpixel registration of the input images. Here, we explore how to exploit CNNs in multiple-image SRR and we demonstrate that competitive reconstruction outcome can be obtained within seconds.

Pawel Benecki

Papers

Deep Learning for Multiple-Image Super-Resolution

Evaluating super-resolution reconstruction of satellite images

Evolving imaging model for super-resolution reconstruction

Detecting anomalies in spacecraft telemetry using evolutionary thresholding and LSTMs

Deep learning for fast super-resolution reconstruction from multiple images