Resolution Enhancement in Multi-Image Stereo
08 Apr 2010-IEEE Transactions on Pattern Analysis and Machine Intelligence (IEEE)-Vol. 32, Iss: 9, pp 1721-1728
TL;DR: This paper proposes an integrated approach to estimate the HR depth and the SR image from multiple LR stereo observations and demonstrates the efficacy of the proposed method in not only being able to bring out image details but also in enhancing theHR depth over its LR counterpart.
Abstract: Under stereo settings, the twin problems of image superresolution (SR) and high-resolution (HR) depth estimation are intertwined. The subpixel registration information required for image superresolution is tightly coupled to the 3D structure. The effects of parallax and pixel averaging (inherent in the downsampling process) preclude a priori estimation of pixel motion for superresolution. These factors also compound the correspondence problem at low resolution (LR), which in turn affects the quality of the LR depth estimates. In this paper, we propose an integrated approach to estimate the HR depth and the SR image from multiple LR stereo observations. Our results demonstrate the efficacy of the proposed method in not only being able to bring out image details but also in enhancing the HR depth over its LR counterpart.
Citations
More filters
TL;DR: This paper provides a comprehensive review of SR image and video reconstruction methods developed in the literature and highlights the future research challenges.
Abstract: The key objective of super-resolution (SR) imaging is to reconstruct a higher-resolution image based on a set of images, acquired from the same scene and denoted as ‘low-resolution’ images, to overcome the limitation and/or ill-posed conditions of the image acquisition process for facilitating better content visualization and scene recognition. In this paper, we provide a comprehensive review of SR image and video reconstruction methods developed in the literature and highlight the future research challenges. The SR image approaches reconstruct a single higher-resolution image from a set of given lower-resolution images, and the SR video approaches reconstruct an image sequence with a higher-resolution from a group of adjacent lower-resolution image frames. Furthermore, several SR applications are discussed to contribute some insightful comments on future SR research directions. Specifically, the SR computations for multi-view images and the SR video computation in the temporal domain are discussed.
255 citations
Cites background from "Resolution Enhancement in Multi-Ima..."
...investigation. Bhavsar and Rajagopalan [109, 110 ] proposed to exploit an iterated conditional modes (ICM) algorithm to reconstruct the higher-resolution images via estimating their MAP estimators....
[...]
15 Jun 2019
TL;DR: A parallax-attention mechanism with a global receptive field along the epipolar line to handle different stereo images with large disparity variations is introduced and a new and the largest dataset for stereo image SR is proposed.
Abstract: Stereo image pairs can be used to improve the performance of super-resolution (SR) since additional information is provided from a second viewpoint. However, it is challenging to incorporate this information for SR since disparities between stereo images vary significantly. In this paper, we propose a parallax-attention stereo superresolution network (PASSRnet) to integrate the information from a stereo image pair for SR. Specifically, we introduce a parallax-attention mechanism with a global receptive field along the epipolar line to handle different stereo images with large disparity variations. We also propose a new and the largest dataset for stereo image SR (namely, Flickr1024). Extensive experiments demonstrate that the parallax-attention mechanism can capture correspondence between stereo images to improve SR performance with a small computational and memory cost. Comparative results show that our PASSRnet achieves the state-of-the-art performance on the Middlebury, KITTI 2012 and KITTI 2015 datasets.
198 citations
Patent•
15 Feb 2010TL;DR: In this paper, a set of spectacles are disclosed for use in viewing images or videos, and include at least two lenses each having an adjustable optical property and an image capture device associated with the spectacles for receiving a digital image comprising a plurality of digital image channels each containing multiple pixels.
Abstract: Spectacles are disclosed for use in viewing images or videos, and include at least two lenses each having an adjustable optical property and an image capture device associated with the spectacles for receiving a digital image comprising a plurality of digital image channels each containing multiple pixels. The spectacles further include the image capture device for capturing a digital image and a processor for computing a feature vector from pixel values of the digital image channels wherein the feature vector includes information that would indicate whether the image is an anaglyph or a non-anaglyph image.
74 citations
18 Jun 2018
TL;DR: A novel method to learn a parallax prior from stereo image datasets by jointly training two-stage networks that enhances the spatial resolution of stereo images significantly more than single-image super-resolution methods.
Abstract: We present a novel method that can enhance the spatial resolution of stereo images using a parallax prior. While traditional stereo imaging has focused on estimating depth from stereo images, our method utilizes stereo images to enhance spatial resolution instead of estimating disparity. The critical challenge for enhancing spatial resolution from stereo images: how to register corresponding pixels with subpixel accuracy. Since disparity in traditional stereo imaging is calculated per pixel, it is directly inappropriate for enhancing spatial resolution. We, therefore, learn a parallax prior from stereo image datasets by jointly training two-stage networks. The first network learns how to enhance the spatial resolution of stereo images in luminance, and the second network learns how to reconstruct a high-resolution color image from high-resolution luminance and chrominance of the input image. Our two-stage joint network enhances the spatial resolution of stereo images significantly more than single-image super-resolution methods. The proposed method is directly applicable to any stereo depth imaging methods, enabling us to enhance the spatial resolution of stereo images.
74 citations
Cites background or methods from "Resolution Enhancement in Multi-Ima..."
...[5] enhance the spatial resolution but suffer from jaggy aliasing artifacts....
[...]
...Bhavsar and Rajagopalan [4, 5] and Park et al....
[...]
...Figure 10: Comparison of our method with a state-ofthe-art stereo super-resolution method by Bhavsar and Rajagopalan [5]....
[...]
...Bhavsar and Rajagopalan [4, 5] and Park et al. [41] utilize stereo block matching to search pixel correspondences....
[...]
...Stereo Image Input We evaluate our method with a state-of-the-art stereo super-resolution method proposed by Bhavsar and Rajagopalan [5]....
[...]
01 Oct 2019
TL;DR: Experimental results show that, as compared to the KITTI and Middlebury datasets, the Flickr1024 dataset can help to handle the over-fitting problem and significantly improves the performance of stereo SR methods.
Abstract: With the popularity of dual cameras in recently released smart phones, a growing number of super-resolution (SR) methods have been proposed to enhance the resolution of stereo image pairs. However, the lack of high-quality stereo datasets has limited the research in this area. To facilitate the training and evaluation of novel stereo SR algorithms, in this paper, we present a large-scale stereo dataset named Flickr1024, which contains 1024 pairs of high-quality images and covers diverse scenarios. We first introduce the data acquisition and processing pipeline, and then compare several popular stereo datasets. Finally, we conduct cross-dataset experiments to investigate the potential benefits introduced by our dataset. Experimental results show that, as compared to the KITTI and Middlebury datasets, our Flickr1024 dataset can help to handle the over-fitting problem and significantly improves the performance of stereo SR methods. The Flickr1024 dataset is available online at: https://yingqianwang.github.io/Flickr1024.
65 citations
Cites background from "Resolution Enhancement in Multi-Ima..."
...Using the complementary information provided by binocular systems, the resolution of image pairs can be enhanced by stereo super-resolution (SR) methods [2, 6, 17]....
[...]
References
More filters
18 Jun 1996
TL;DR: This work intensively analyzed the occlusion in the camera matrix stereo (SEA) and developed a simple but effective method to detect the presence of Occlusion and to eliminate its effect in the correspondence search.
Abstract: In stereo algorithms with more than two cameras, the improvement of accuracy is often reported since they are robust against noise. However, another important aspect of the polynocular stereo, that is the ability of occlusion detection, has been paid less attention. We intensively analyzed the occlusion in the camera matrix stereo (SEA) and developed a simple but effective method to detect the presence of occlusion and to eliminate its effect in the correspondence search. By considering several statistics on the occlusion and the accuracy in the SEA, we derived a few base masks which represent occlusion patterns and are effective for the detection of occlusion. Several experiments using typical indoor scenes showed quite good performance to obtain dense and accurate depth maps even at the occluding boundaries of objects.
125 citations
"Resolution Enhancement in Multi-Ima..." refers background in this paper
...[23] show that the degradation in accuracy resulting from declaring an invisible pixel as visible (I to V error) can be about 10 times that of labeling a visible pixel as invisible (V to I error)....
[...]
20 Jun 2005
TL;DR: A new model to overcome the occlusion problems coming from wide baseline multiple camera stereo is presented, which detects occlusions in the depth map obtained from regular efficient stereo matching algorithms.
Abstract: This paper presents a new model to overcome the occlusion problems coming from wide baseline multiple camera stereo. Rather than explicitly modeling occlusions in the matching cost function, it detects occlusions in the depth map obtained from regular efficient stereo matching algorithms. Occlusions are detected as inconsistencies of the depth map by computing the visibility of the map as it is reprojected into each camera. Our approach has the particularity of not discriminating between occluders and occludees. The matching cost function is modified according to the detected occlusions by removing the offending cameras from the computation of the matching cost. The algorithm gradually modifies the matching cost function according to the history of inconsistencies in the depth map, until convergence. While two graph-theoretic stereo algorithms are used in our experiments, our framework is general enough to be applied to many others. The validity of our framework is demonstrated using real imagery with different baselines.
123 citations
"Resolution Enhancement in Multi-Ima..." refers background in this paper
...It generalizes motion-based image superresolution to 3D scenes and also enables the estimation of depth at a resolution higher than that of the given observations....
[...]
TL;DR: In this paper, a mathematical framework and optimization algorithms that can be used to jointly estimate the displacements between the low-resolution images are presented, and numerical experiments are provided to illustrate the effectiveness of their approach.
Abstract: The process of combining, via mathematical software tools, a set of low-resolution images into a single high-resolution image is often referred to as super-resolution. Algorithms for super-resolution involve two key steps: registration and reconstruction. Most approaches proposed in the literature decouple these steps, solving each independently. This can be effective if there are very simple, linear displacements between the low-resolution images. However, for more complex, nonlinear, nonuniform transformations, estimating the displacements can be very difficult, leading to severe inaccuracies in the reconstructed high-resolution image. This paper presents a mathematical framework and optimization algorithms that can be used to jointly estimate these quantities. Efficient implementation details are considered, and numerical experiments are provided to illustrate the effectiveness of our approach.
92 citations
"Resolution Enhancement in Multi-Ima..." refers background in this paper
...In its simplest form, the images are related via a global translation defined by only two parameters....
[...]
TL;DR: The proposed approach constitutes a principled unified probabilistic framework for low level scene analysis and understanding, showing several key features with respect to the state of the art methods, as it extracts information at the lowest possible level (using only pixel gray-level temporal behavior), and is unsupervised in nature.
85 citations
"Resolution Enhancement in Multi-Ima..." refers methods or result in this paper
...Unlike in [1], [ 7 ], we use multiimage stereo geometry to better constrain the correspondence problem, and hence, the solution-space....
[...]
...Another important difference between our work and that of [ 7 ] is that we carry out the visibility computation based on scene geometry [4] as opposed to their statistical approach....
[...]
...Whereas the work in [ 7 ] adopts an image generation model, the approach in [1] iterates between flow computation on interpolated images and their improvement by a mean filter....
[...]
...Closely related to our work is that of [1], [ 7 ]....
[...]
24 Nov 2003
TL;DR: A different implementation using L/sub 1/ norm minimization and robust regularization to deal with different data and noise models is proposed, which is robust to errors in motion and blur estimation, and results in sharp edges.
Abstract: In the last two decades, many papers have been published, proposing a variety methods of multiframe resolution enhancement. These methods are usually very sensitive to their assumed model of data and noise, which limits their utility. This paper reviews some of these methods and addresses their shortcomings. We propose a different implementation using L/sub 1/ norm minimization and robust regularization to deal with different data and noise models. This computationally inexpensive method is robust to errors in motion and blur estimation, and results in sharp edges. Simulation results confirm the effectiveness of our method and demonstrate its superiority to other robust super-resolution methods.
51 citations
"Resolution Enhancement in Multi-Ima..." refers background in this paper
...The subpixel registration information required for image superresolution is tightly coupled to the 3D structure....
[...]