A new approach toward increasing spatial resolution is required to overcome the limitations of the sensors and optics manufacturing technology. One promising approach is to use signal processing techniques to obtain an high-resolution (HR) image (or sequence) from observed multiple low-resolution (LR) images. Such a resolution enhancement approach has been one of the most active research areas, and it is called super resolution (SR) (or HR) image reconstruction or simply resolution enhancement. In this article, we use the term "SR image reconstruction" to refer to a signal processing approach toward resolution enhancement because the term "super" in "super resolution" represents very well the characteristics of the technique overcoming the inherent resolution limitation of LR imaging systems. The major advantage of the signal processing approach is that it may cost less and the existing LR imaging systems can be still utilized. The SR image reconstruction is proved to be useful in many practical cases where multiple frames of the same scene can be obtained, including medical imaging, satellite imaging, and video applications. The goal of this article is to introduce the concept of SR algorithms to readers who are unfamiliar with this area and to provide a review for experts. To this purpose, we present the technical review of various existing SR methodologies which are often employed. Before presenting the review of existing SR algorithms, we first model the LR image acquisition process.

Super-resolution image reconstruction: a technical overview

Nearly all super-resolution algorithms are based on the fundamental constraints that the super-resolution image should generate low resolution input images when appropriately warped and down-sampled to model the image formation process. (These reconstruction constraints are normally combined with some form of smoothness prior to regularize their solution.) We derive a sequence of analytical results which show that the reconstruction constraints provide less and less useful information as the magnification factor increases. We also validate these results empirically and show that, for large enough magnification factors, any smoothness prior leads to overly smooth results with very little high-frequency content. Next, we propose a super-resolution algorithm that uses a different kind of constraint in addition to the reconstruction constraints. The algorithm attempts to recognize local features in the low-resolution images and then enhances their resolution in an appropriate manner. We call such a super-resolution algorithm a hallucination or reconstruction algorithm. We tried our hallucination algorithm on two different data sets, frontal images of faces and printed Roman text. We obtained significantly better results than existing reconstruction-based algorithms, both qualitatively and in terms of RMS pixel error.

/pdf/limits-on-super-resolution-and-how-to-break-them-401rdhxruv.pdf

Limits on super-resolution and how to break them

The three main tools in the single image restoration theory are the maximum likelihood (ML) estimator, the maximum a posteriori probability (MAP) estimator, and the set theoretic approach using projection onto convex sets (POCS). This paper utilizes the above known tools to propose a unified methodology toward the more complicated problem of superresolution restoration. In the superresolution restoration problem, an improved resolution image is restored from several geometrically warped, blurred, noisy and downsampled measured images. The superresolution restoration problem is modeled and analyzed from the ML, the MAP, and POCS points of view, yielding a generalization of the known superresolution restoration methods. The proposed restoration approach is general but assumes explicit knowledge of the linear space- and time-variant blur, the (additive Gaussian) noise, the different measured resolutions, and the (smooth) motion characteristics. A hybrid method combining the simplicity of the ML and the incorporation of nonellipsoid constraints is presented, giving improved restoration performance, compared with the ML and the POCS approaches. The hybrid method is shown to converge to the unique optimal solution of a new definition of the optimization problem. Superresolution restoration from motionless measurements is also discussed. Simulations demonstrate the power of the proposed methodology.

/pdf/restoration-of-a-single-superresolution-image-from-several-11jwkvrm8u.pdf

Restoration of a single superresolution image from several blurred, noisy, and undersampled measured images

We analyze the super-resolution reconstruction constraints. In particular we derive a sequence of results which all show that the constraints provide far less useful information as the magnification factor increases. It is well established that the use of a smoothness prior may help somewhat, however for large enough magnification factors any smoothness prior leads to overly smooth results. We therefore propose an algorithm that learns recognition-based priors for specific classes of scenes, the use of which gives far better super-resolution results for both faces and text.

The human visual system appears to be capable of temporally integrating information in a video sequence in such a way that the perceived spatial resolution of a sequence appears much higher than the spatial resolution of an individual frame. While the mechanisms in the human visual system that do this are unknown, the effect is not too surprising given that temporally adjacent frames in a video sequence contain slightly different, but unique, information. This paper addresses the use of both the spatial and temporal information present in a short image sequence to create a single high-resolution video frame. A novel observation model based on motion compensated subsampling is proposed for a video sequence. Since the reconstruction problem is ill-posed, Bayesian restoration with a discontinuity-preserving prior image model is used to extract a high-resolution video still given a short low-resolution sequence. Estimates computed from a low-resolution image sequence containing a subpixel camera pan show dramatic visual and quantitative improvements over bilinear, cubic B-spline, and Bayesian single frame interpolations. Visual and quantitative improvements are also shown for an image sequence containing objects moving with independent trajectories. Finally, the video frame extraction algorithm is used for the motion-compensated scan conversion of interlaced video data, with a visual comparison to the resolution enhancement obtained from progressively scanned frames.

Extraction of high-resolution frames from video sequences

Printing from an NTSC source and conversion of NTSC source material to high-definition television (HDTV) format are some of the applications that motivate superresolution (SR) image and video reconstruction from low-resolution (LR) and possibly blurred sources. Existing methods for SR image reconstruction are limited by the assumptions that the input LR images are sampled progressively, and that the aperture time of the camera is zero, thus ignoring the motion blur occurring during the aperture time. Because of the observed adverse effects of these assumptions for many common video sources, this paper proposes (i) a complete model of video acquisition with an arbitrary input sampling lattice and a nonzero aperture time, and (ii) an algorithm based on this model using the theory of projections onto convex sets to reconstruct SR still images or video from an LR time sequence of images. Experimental results with real video are provided, which clearly demonstrate that a significant increase in the image resolution can be achieved by taking the motion blurring into account especially when there exists large interframe motion.

Superresolution video reconstruction with arbitrary sampling lattices and nonzero aperture time

We address the problem of reconstruction of a high-resolution image from a sequence of low-resolution images containing arbitrary relative motion, excluding occlusion effects. We develop a formulation that simultaneously takes into account blurring due to relative sensor-object motion, sensor integration, and additive noise. We propose a POCS-based algorithm for performing the high-resolution reconstruction, and provide experimental results. >

High-resolution image reconstruction from a low-resolution image sequence in the presence of time-varying motion blur

We present robust algorithms which combine global motion compensation and motion adaption for deinterlacing in the presence of both dominant motion, such as camera zoom, pan, or jitter, and local motion, such as object motion. The dominant motion is modeled by a global affine warping and estimated by a gradient-based estimation method. Two alternative algorithms are proposed for compensation of the dominant motion: a bilinear interpolation based on the affine model, and a projections onto convex sets (POCS) based method that takes into account blurring in the image formation. It is important to note that the latter must be used if the blurring is severe enough to act as an anti-alias filter, which imposes an irreversible limit on the resolution improvement ability of any motion-compensated filter. Global motion-compensated images are then input to a motion-adaptive filter to detect and correct for those pixels where there exists local motion. A dynamic thresholding for motion detection is presented, with weighted directional-filtering for regions where motion is detected, to obtain the best results. Experimental results with application to obtaining high quality stills from video camcorders demonstrate the effectiveness of the proposed methods.

Robust methods for high-quality stills from interlaced video in the presence of dominant motion

With the advent of frame grabbers capable of acquiring multiple video frames, a great deal of attention is being directed at creating high-resolution (hi-res) imagery from interlaced or low-resolution (low-res) video. This is a multi-faceted problem, which generally necessitates standards conversion and hi-res reconstruction. Standards conversion is the problem of converting from one spatio-temporal sampling lattice to another, while hi-res image reconstruction involves increasing the spatial sampling density. Also of interest is removing degradations that occur during the image acquisition process. These tasks have all received considerable, yet separate, treatment in the literature. A unifying video formation model is presented which addresses these problems simultaneously. Then, a POCS-based algorithm for generating high-resolution imagery from video is delineated. Results with real imagery are included.

High resolution standards conversion of low resolution video

The present invention is a two-step, contour-sensitive deinterlacing technique. The first step of the technique determines for each missing pixel of an interlaced frame of image pixels whether the absolute difference between the pixels above and below the missing pixel is greater than a preselected threshold value. If it is decided that the missing pixel lies at a low-vertical frequency location, its value is estimated via vertical interpolation. Otherwise, the second step is carried out. The goal of the second step is to determine whether or not there is a well-defined contour passing through the missing pixel, and to determine its direction if there is one. In the presence of a well-defined contour, the missing pixel is obtained by averaging the intensity values along the direction of the contour in the field lines immediately above and below the missing field line. Otherwise the process effectively falls back to vertical interpolation. The presence of a well-defined contour is detected by comparing the summed absolute differences (SAD) of blocks of pixels along a predetermined set of candidate directions, including the vertical direction, and selecting the direction associated with the least SAD. The fall-back to vertical interpolation in the absence of a well-defined contour is implicit due to the weighting of the SAD corresponding to the vertical direction.

Andrew J Patti

Papers

Superresolution video reconstruction with arbitrary sampling lattices and nonzero aperture time

High-resolution image reconstruction from a low-resolution image sequence in the presence of time-varying motion blur

Robust methods for high-quality stills from interlaced video in the presence of dominant motion

High resolution standards conversion of low resolution video

Contour-sensitive, single-field deinterlacing method