Recently, several models based on deep neural networks have achieved great success in terms of both reconstruction accuracy and computational performance for single image super-resolution. In these methods, the low resolution (LR) input image is upscaled to the high resolution (HR) space using a single filter, commonly bicubic interpolation, before reconstruction. This means that the super-resolution (SR) operation is performed in HR space. We demonstrate that this is sub-optimal and adds computational complexity. In this paper, we present the first convolutional neural network (CNN) capable of real-time SR of 1080p videos on a single K2 GPU. To achieve this, we propose a novel CNN architecture where the feature maps are extracted in the LR space. In addition, we introduce an efficient sub-pixel convolution layer which learns an array of upscaling filters to upscale the final LR feature maps into the HR output. By doing so, we effectively replace the handcrafted bicubic filter in the SR pipeline with more complex upscaling filters specifically trained for each feature map, whilst also reducing the computational complexity of the overall SR operation. We evaluate the proposed approach using images and videos from publicly available datasets and show that it performs significantly better (+0.15dB on Images and +0.39dB on Videos) and is an order of magnitude faster than previous CNN-based methods.

/pdf/real-time-single-image-and-video-super-resolution-using-an-4ljje8ipch.pdf

Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network

Recently, Convolutional Neural Network (CNN) based models have achieved great success in Single Image Super-Resolution (SISR). Owing to the strength of deep networks, these CNN models learn an effective nonlinear mapping from the low-resolution input image to the high-resolution target image, at the cost of requiring enormous parameters. This paper proposes a very deep CNN model (up to 52 convolutional layers) named Deep Recursive Residual Network (DRRN) that strives for deep yet concise networks. Specifically, residual learning is adopted, both in global and local manners, to mitigate the difficulty of training very deep networks, recursive learning is used to control the model parameters while increasing the depth. Extensive benchmark evaluation shows that DRRN significantly outperforms state of the art in SISR, while utilizing far fewer parameters. Code is available at https://github.com/tyshiwo/DRRN_CVPR17.

/pdf/image-super-resolution-via-deep-recursive-residual-network-4npbhz4o8c.pdf

Image Super-Resolution via Deep Recursive Residual Network

In this report we demonstrate that with same parameters and computational budgets, models with wider features before ReLU activation have significantly better performance for single image super-resolution (SISR). The resulted SR residual network has a slim identity mapping pathway with wider (\(2\times\) to \(4\times\)) channels before activation in each residual block. To further widen activation (\(6\times\) to \(9\times\)) without computational overhead, we introduce linear low-rank convolution into SR networks and achieve even better accuracy-efficiency tradeoffs. In addition, compared with batch normalization or no normalization, we find training with weight normalization leads to better accuracy for deep super-resolution networks. Our proposed SR network \textit{WDSR} achieves better results on large-scale DIV2K image super-resolution benchmark in terms of PSNR with same or lower computational complexity. Based on WDSR, our method also won 1st places in NTIRE 2018 Challenge on Single Image Super-Resolution in all three realistic tracks. Experiments and ablation studies support the importance of wide activation for image super-resolution. Code is released at: this https URL

Wide Activation for Efficient and Accurate Image Super-Resolution.

Downscaling in remote sensing

Mapping rural land cover features, such as trees and hedgerows, for ecological applications is a desirable component of the creation of cartographic maps by the Ordnance Survey. Based on the phenomenon of spatial dependence, sub‐pixel mapping can provide increased mapping accuracy of such features. A simple pixel‐swapping algorithm for sub‐pixel mapping was applied to soft classified fine spatial resolution remotely sensed imagery. Initially, QuickbirdTM satellite sensor imagery with a spatial resolution of 2.6 m was acquired of the Christchurch area of Dorset, UK, and three field sites chosen. The imagery was soft classified using a supervised fuzzy c‐means algorithm and then super‐resolved into sub‐pixels using a zoom factor of five. Sub‐pixels within pixels were then iteratively swapped until the spatial correlation between sub‐pixels for the entire image was maximized. Mathematical morphology was used to suppress error in the super‐resolved output, increasing overall accuracy. Field data, including de...

M. W. Thornton

Papers

Sub‐pixel mapping of rural land cover objects from fine spatial resolution satellite sensor imagery using super‐resolution pixel‐swapping

A linearised pixel-swapping method for mapping rural linear land cover features from fine spatial resolution remotely sensed imagery

Super-resolution mapping of rural land cover features from fine spatial resolution satellite sensor imagery