Super-resolution, the process of obtaining one or more high-resolution images from one or more low-resolution observations, has been a very attractive research topic over the last two decades. It has found practical applications in many real-world problems in different fields, from satellite and aerial imaging to medical image processing, to facial image analysis, text image analysis, sign and number plates reading, and biometrics recognition, to name a few. This has resulted in many research papers, each developing a new super-resolution algorithm for a specific purpose. The current comprehensive survey provides an overview of most of these published works by grouping them in a broad taxonomy. For each of the groups in the taxonomy, the basic concepts of the algorithms are first explained and then the paths through which each of these groups have evolved are given in detail, by mentioning the contributions of different authors to the basic concepts of each group. Furthermore, common issues in super-resolution algorithms, such as imaging models and registration algorithms, optimization of the cost functions employed, dealing with color information, improvement factors, assessment of super-resolution algorithms, and the most commonly employed databases are discussed.

/pdf/super-resolution-a-comprehensive-survey-xjl42yt9nt.pdf

Super-resolution: a comprehensive survey

The key objective of super-resolution (SR) imaging is to reconstruct a higher-resolution image based on a set of images, acquired from the same scene and denoted as ‘low-resolution’ images, to overcome the limitation and/or ill-posed conditions of the image acquisition process for facilitating better content visualization and scene recognition. In this paper, we provide a comprehensive review of SR image and video reconstruction methods developed in the literature and highlight the future research challenges. The SR image approaches reconstruct a single higher-resolution image from a set of given lower-resolution images, and the SR video approaches reconstruct an image sequence with a higher-resolution from a group of adjacent lower-resolution image frames. Furthermore, several SR applications are discussed to contribute some insightful comments on future SR research directions. Specifically, the SR computations for multi-view images and the SR video computation in the temporal domain are discussed.

A survey on super-resolution imaging

Despite their high popularity, common high dynamic range (HDR) methods are still limited in their practical applicability: They assume that the input images are perfectly aligned, which is often violated in practise. Our paper does not only free the user from this unrealistic limitation, but even turns the missing alignment into an advantage: By exploiting the multiple exposures, we can create a super-resolution image. The alignment step is performed by a modern energy-based optic flow approach that takes into account the varying exposure conditions. Moreover, it produces dense displacement fields with subpixel precision. As a consequence, our approach can handle arbitrary complex motion patterns, caused by severe camera shake and moving objects. Additionally, it benefits from several advantages over existing strategies: (i) It is robust under outliers (noise, occlusions, saturation problems) and allows for sharp discontinuities in the displacement field. (ii) The alignment step neither requires camera calibration nor knowledge of the exposure times. (iii) It can be efficiently implemented on CPU and GPU architectures. After the alignment is performed, we use the obtained subpixel accurate displacement fields as input for an energy-based, joint super-resolution and HDR (SR-HDR) approach. It introduces robust data terms and anisotropic smoothness terms in the SR-HDR literature. Our experiments with challenging real world data demonstrate that these novelties are pivotal for the favourable performance of our approach.

Freehand HDR Imaging of Moving Scenes with Simultaneous Resolution Enhancement

An image acquisition sensor of a digital image acquisition apparatus is coupled to imaging optics for acquiring a sequence of images. Images acquired by the sensor are stored. A motion detector causes the sensor to cease capture of an image when the degree of movement in acquiring the image exceeds a threshold. A controller selectively transfers acquired images for storage. A motion extractor determines motion parameters of a selected, stored image. An image re-constructor corrects the selected image with associated motion parameters. A selected plurality of images nominally of the same scene are merged and corrected by the image re-constructor to produce a high quality image of the scene.

Image Acquisition Method and Apparatus

A method of combining image data from multiple frames to enhance one or more parameters of video image quality includes acquiring a first image at a first exposure duration, as well as acquiring a second image at a second exposure duration shorter than the first exposure duration and at a time just before, just after or overlapping in time with acquiring the first image, such that the first and second images include approximately a same first scene. In this way, the second image is relatively sharp and under-exposed, while the first image is relatively well-exposed and less sharp than the second image. Brightness and/or color information are extracted from the first image and applied to the second image to generate an enhanced version of the second image.

Low-light video frame enhancement

Super-resolution reconstruction is the process of reconstructing a high-resolution image from multiple low-resolution images. Most super-resolution reconstruction methods assume that exposure time is fixed for all observations, which is not necessarily true. In reality, cameras have limited dynamic range and nonlinear response to the quantity of light received, and exposure time might be adjusted automatically or manually to capture the desired portion of the scene's dynamic range. In this letter, we propose a Bayesian super-resolution algorithm based on an imaging model that includes camera response function, exposure time, sensor noise, and quantization error in addition to spatial blurring and sampling.

/pdf/high-resolution-image-reconstruction-from-multiple-2fw0vu3hi2.pdf

High-resolution image reconstruction from multiple differently exposed images

https://www.ece.lsu.edu/ipl/papers/Elsevier_CVIU2009.pdf

Illumination robust interest point detection

Superresolution (SR) is a well-known technique to increase the quality of an image using multiple overlapping pictures of a scene. SR requires accurate registration of the images, both geometrically and photometrically. Most of the SR articles in the literature have considered geometric registration only, assuming that images are captured under the same photometric conditions. This is not necessarily true as external illumination conditions and/or camera parameters (such as exposure time, aperture size, and white balancing) may vary for different input images. Therefore, photometric modeling is a necessary task for superresolution. In this paper, we investigate superresolution image reconstruction when there is photometric variation among input images.

/pdf/superresolution-under-photometric-diversity-of-images-2shprdic93.pdf

Superresolution under photometric diversity of images

Finding geometric and photometric relation among images is crucial in many computer vision tasks such as panoramic imaging, high dynamic range imaging, stereo imaging, and change detection. Most photometric registration algorithms require accurate geometric registration of images. On the other hand, geometric registration may fail when images are not aligned photometrically. There are two contributions of this paper: (i) A contrast invariant feature detection algorithm is proposed. This would allow geometric registration of images without photometric registration, (ii) A photometric registration algorithm that can handle scene occlusions is presented.

On Geometric and Photometric Registration of Images

Spatial resolution of digital images are limited due to optical/sensor blurring and sensor site density. In single-chip digital cameras, the resolution is further degraded because such devices use a color filter array to capture only one spectral component at a pixel location. The process of estimating the missing two color values at each pixel location is known as demosaicking. Demosaicking methods usually exploit the correlation among color channels. When there are multiple images, it is possible not only to have better estimates of the missing color values but also to improve the spatial resolution further (using super-resolution reconstruction). Previously, we have proposed a demosaicking algorithm based on the projection onto convex sets (POCS) technique. In this paper, we improve the results of that algorithm adding a new constraint set based on the spatio-intensity neighborhood. We extend the algorithm to image sequences for multi-frame demosaicking and super resolution.

Murat Gevrekci

Papers

High-resolution image reconstruction from multiple differently exposed images

Illumination robust interest point detection

Superresolution under photometric diversity of images

On Geometric and Photometric Registration of Images

POCS-Based Restoration of Bayer-Sampled Image Sequences