Resolution Enhancement in Multi-Image Stereo
08 Apr 2010-IEEE Transactions on Pattern Analysis and Machine Intelligence (IEEE)-Vol. 32, Iss: 9, pp 1721-1728
TL;DR: This paper proposes an integrated approach to estimate the HR depth and the SR image from multiple LR stereo observations and demonstrates the efficacy of the proposed method in not only being able to bring out image details but also in enhancing theHR depth over its LR counterpart.
Abstract: Under stereo settings, the twin problems of image superresolution (SR) and high-resolution (HR) depth estimation are intertwined. The subpixel registration information required for image superresolution is tightly coupled to the 3D structure. The effects of parallax and pixel averaging (inherent in the downsampling process) preclude a priori estimation of pixel motion for superresolution. These factors also compound the correspondence problem at low resolution (LR), which in turn affects the quality of the LR depth estimates. In this paper, we propose an integrated approach to estimate the HR depth and the SR image from multiple LR stereo observations. Our results demonstrate the efficacy of the proposed method in not only being able to bring out image details but also in enhancing the HR depth over its LR counterpart.
Citations
More filters
TL;DR: This paper provides a comprehensive review of SR image and video reconstruction methods developed in the literature and highlights the future research challenges.
Abstract: The key objective of super-resolution (SR) imaging is to reconstruct a higher-resolution image based on a set of images, acquired from the same scene and denoted as ‘low-resolution’ images, to overcome the limitation and/or ill-posed conditions of the image acquisition process for facilitating better content visualization and scene recognition. In this paper, we provide a comprehensive review of SR image and video reconstruction methods developed in the literature and highlight the future research challenges. The SR image approaches reconstruct a single higher-resolution image from a set of given lower-resolution images, and the SR video approaches reconstruct an image sequence with a higher-resolution from a group of adjacent lower-resolution image frames. Furthermore, several SR applications are discussed to contribute some insightful comments on future SR research directions. Specifically, the SR computations for multi-view images and the SR video computation in the temporal domain are discussed.
255 citations
Cites background from "Resolution Enhancement in Multi-Ima..."
...investigation. Bhavsar and Rajagopalan [109, 110 ] proposed to exploit an iterated conditional modes (ICM) algorithm to reconstruct the higher-resolution images via estimating their MAP estimators....
[...]
15 Jun 2019
TL;DR: A parallax-attention mechanism with a global receptive field along the epipolar line to handle different stereo images with large disparity variations is introduced and a new and the largest dataset for stereo image SR is proposed.
Abstract: Stereo image pairs can be used to improve the performance of super-resolution (SR) since additional information is provided from a second viewpoint. However, it is challenging to incorporate this information for SR since disparities between stereo images vary significantly. In this paper, we propose a parallax-attention stereo superresolution network (PASSRnet) to integrate the information from a stereo image pair for SR. Specifically, we introduce a parallax-attention mechanism with a global receptive field along the epipolar line to handle different stereo images with large disparity variations. We also propose a new and the largest dataset for stereo image SR (namely, Flickr1024). Extensive experiments demonstrate that the parallax-attention mechanism can capture correspondence between stereo images to improve SR performance with a small computational and memory cost. Comparative results show that our PASSRnet achieves the state-of-the-art performance on the Middlebury, KITTI 2012 and KITTI 2015 datasets.
198 citations
Patent•
15 Feb 2010TL;DR: In this paper, a set of spectacles are disclosed for use in viewing images or videos, and include at least two lenses each having an adjustable optical property and an image capture device associated with the spectacles for receiving a digital image comprising a plurality of digital image channels each containing multiple pixels.
Abstract: Spectacles are disclosed for use in viewing images or videos, and include at least two lenses each having an adjustable optical property and an image capture device associated with the spectacles for receiving a digital image comprising a plurality of digital image channels each containing multiple pixels. The spectacles further include the image capture device for capturing a digital image and a processor for computing a feature vector from pixel values of the digital image channels wherein the feature vector includes information that would indicate whether the image is an anaglyph or a non-anaglyph image.
74 citations
18 Jun 2018
TL;DR: A novel method to learn a parallax prior from stereo image datasets by jointly training two-stage networks that enhances the spatial resolution of stereo images significantly more than single-image super-resolution methods.
Abstract: We present a novel method that can enhance the spatial resolution of stereo images using a parallax prior. While traditional stereo imaging has focused on estimating depth from stereo images, our method utilizes stereo images to enhance spatial resolution instead of estimating disparity. The critical challenge for enhancing spatial resolution from stereo images: how to register corresponding pixels with subpixel accuracy. Since disparity in traditional stereo imaging is calculated per pixel, it is directly inappropriate for enhancing spatial resolution. We, therefore, learn a parallax prior from stereo image datasets by jointly training two-stage networks. The first network learns how to enhance the spatial resolution of stereo images in luminance, and the second network learns how to reconstruct a high-resolution color image from high-resolution luminance and chrominance of the input image. Our two-stage joint network enhances the spatial resolution of stereo images significantly more than single-image super-resolution methods. The proposed method is directly applicable to any stereo depth imaging methods, enabling us to enhance the spatial resolution of stereo images.
74 citations
Cites background or methods from "Resolution Enhancement in Multi-Ima..."
...[5] enhance the spatial resolution but suffer from jaggy aliasing artifacts....
[...]
...Bhavsar and Rajagopalan [4, 5] and Park et al....
[...]
...Figure 10: Comparison of our method with a state-ofthe-art stereo super-resolution method by Bhavsar and Rajagopalan [5]....
[...]
...Bhavsar and Rajagopalan [4, 5] and Park et al. [41] utilize stereo block matching to search pixel correspondences....
[...]
...Stereo Image Input We evaluate our method with a state-of-the-art stereo super-resolution method proposed by Bhavsar and Rajagopalan [5]....
[...]
01 Oct 2019
TL;DR: Experimental results show that, as compared to the KITTI and Middlebury datasets, the Flickr1024 dataset can help to handle the over-fitting problem and significantly improves the performance of stereo SR methods.
Abstract: With the popularity of dual cameras in recently released smart phones, a growing number of super-resolution (SR) methods have been proposed to enhance the resolution of stereo image pairs. However, the lack of high-quality stereo datasets has limited the research in this area. To facilitate the training and evaluation of novel stereo SR algorithms, in this paper, we present a large-scale stereo dataset named Flickr1024, which contains 1024 pairs of high-quality images and covers diverse scenarios. We first introduce the data acquisition and processing pipeline, and then compare several popular stereo datasets. Finally, we conduct cross-dataset experiments to investigate the potential benefits introduced by our dataset. Experimental results show that, as compared to the KITTI and Middlebury datasets, our Flickr1024 dataset can help to handle the over-fitting problem and significantly improves the performance of stereo SR methods. The Flickr1024 dataset is available online at: https://yingqianwang.github.io/Flickr1024.
65 citations
Cites background from "Resolution Enhancement in Multi-Ima..."
...Using the complementary information provided by binocular systems, the resolution of image pairs can be enhanced by stereo super-resolution (SR) methods [2, 6, 17]....
[...]
References
More filters
TL;DR: This paper has designed a stand-alone, flexible C++ implementation that enables the evaluation of individual components and that can easily be extended to include new algorithms.
Abstract: Stereo matching is one of the most active research areas in computer vision. While a large number of algorithms for stereo correspondence have been developed, relatively little work has been done on characterizing their performance. In this paper, we present a taxonomy of dense, two-frame stereo methods designed to assess the different components and design decisions made in individual stereo algorithms. Using this taxonomy, we compare existing stereo methods and present experiments evaluating the performance of many different variants. In order to establish a common software platform and a collection of data sets for easy evaluation, we have designed a stand-alone, flexible C++ implementation that enables the evaluation of individual components and that can be easily extended to include new algorithms. We have also produced several new multiframe stereo data sets with ground truth, and are making both the code and data sets available on the Web.
7,458 citations
TL;DR: This work presents two algorithms based on graph cuts that efficiently find a local minimum with respect to two types of large moves, namely expansion moves and swap moves that allow important cases of discontinuity preserving energies.
Abstract: Many tasks in computer vision involve assigning a label (such as disparity) to every pixel. A common constraint is that the labels should vary smoothly almost everywhere while preserving sharp discontinuities that may exist, e.g., at object boundaries. These tasks are naturally stated in terms of energy minimization. The authors consider a wide class of energies with various smoothness constraints. Global minimization of these energy functions is NP-hard even in the simplest discontinuity-preserving case. Therefore, our focus is on efficient approximation algorithms. We present two algorithms based on graph cuts that efficiently find a local minimum with respect to two types of large moves, namely expansion moves and swap moves. These moves can simultaneously change the labels of arbitrarily large sets of pixels. In contrast, many standard algorithms (including simulated annealing) use small moves where only one pixel changes its label at a time. Our expansion algorithm finds a labeling within a known factor of the global minimum, while our swap algorithm handles more general energy functions. Both of these algorithms allow important cases of discontinuity preserving energies. We experimentally demonstrate the effectiveness of our approach for image restoration, stereo and motion. On real data with ground truth, we achieve 98 percent accuracy.
7,413 citations
"Resolution Enhancement in Multi-Ima..." refers background or methods in this paper
...If the scene mainly consists of discontinuous depth planes, we choose a Potts model [ 2 ] (equivalent to choosing T as the minimum depth label)....
[...]
...The minimization for depth is carried out by � -expansion graph cuts [ 2 ] and that for the image is carried out using iterated conditional modes (ICM) [6], [33]....
[...]
...Graph cuts is one of the better performing contemporary algorithms for stereo disparity estimation and we too employ it. Its advantage is its efficiency and the guarantee of reaching a strong local minimum [ 2 ]....
[...]
17 Jun 2006
TL;DR: This paper first survey multi-view stereo algorithms and compare them qualitatively using a taxonomy that differentiates their key properties, then describes the process for acquiring and calibrating multiview image datasets with high-accuracy ground truth and introduces the evaluation methodology.
Abstract: This paper presents a quantitative comparison of several multi-view stereo reconstruction algorithms. Until now, the lack of suitable calibrated multi-view image datasets with known ground truth (3D shape models) has prevented such direct comparisons. In this paper, we first survey multi-view stereo algorithms and compare them qualitatively using a taxonomy that differentiates their key properties. We then describe our process for acquiring and calibrating multiview image datasets with high-accuracy ground truth and introduce our evaluation methodology. Finally, we present the results of our quantitative comparison of state-of-the-art multi-view stereo reconstruction algorithms on six benchmark datasets. The datasets, evaluation details, and instructions for submitting new models are available online at http://vision.middlebury.edu/mview.
2,556 citations
"Resolution Enhancement in Multi-Ima..." refers methods in this paper
...For the multiview temple scene, where we compute the depth map (instead of disparity), the depth step was computed using the resolution information provided on the Middlebury multiview Web page [27]....
[...]
...Finally, we provide results for SR by 2 on the Middlebury multiview temple data set [27]....
[...]
...We carried out experiments on the Middlebury [26], [27], [35] and CMU data sets [34], as well as images captured in our lab....
[...]
07 Mar 2001
TL;DR: This paper presents a new method which properly addresses occlusions, while preserving the advantages of graph cut algorithms, and gives experimental results for stereo as well as motion, which demonstrate that the method performs well both at detecting occlusion and computing disparities.
Abstract: Several new algorithms for visual correspondence based on graph cuts have recently been developed. While these methods give very strong results in practice, they do not handle occlusions properly. Specifically, they treat the two input images asymmetrically, and they do not ensure that a pixel corresponds to at most one pixel in the other image. In this paper, we present a new method which properly addresses occlusions, while preserving the advantages of graph cut algorithms. We give experimental results for stereo as well as motion, which demonstrate that our method performs well both at detecting occlusions and computing disparities.
1,334 citations
"Resolution Enhancement in Multi-Ima..." refers methods in this paper
...Many methods exist for handling visibility [4], [10], [14], [29], [31]....
[...]
Book•
01 Aug 1995TL;DR: This book presents a comprehensive study on the use of MRFs for solving computer vision problems, and covers the following parts essential to the subject: introduction to fundamental theories, formulations of MRF vision models, MRF parameter estimation, and optimization algorithms.
Abstract: From the Publisher:
Markov random field (MRF) theory provides a basis for modeling contextual constraints in visual processing and interpretation. It enables us to develop optimal vision algorithms systematically when used with optimization principles. This book presents a comprehensive study on the use of MRFs for solving computer vision problems. The book covers the following parts essential to the subject: introduction to fundamental theories, formulations of MRF vision models, MRF parameter estimation, and optimization algorithms. Various vision models are presented in a unified framework, including image restoration and reconstruction, edge and region segmentation, texture, stereo and motion, object matching and recognition, and pose estimation. This book is an excellent reference for researchers working in computer vision, image processing, statistical pattern recognition, and applications of MRFs. It is also suitable as a text for advanced courses in these areas.
1,333 citations
"Resolution Enhancement in Multi-Ima..." refers methods in this paper
...The termsEzðzÞ andExðxÞ correspond to the weighted MRF priors applied on the depth and the image, respectively....
[...]