This paper describes the semiglobal matching (SGM) stereo method. It uses a pixelwise, mutual information (Ml)-based matching cost for compensating radiometric differences of input images. Pixelwise matching is supported by a smoothness constraint that is usually expressed as a global cost function. SGM performs a fast approximation by pathwise optimizations from all directions. The discussion also addresses occlusion detection, subpixel refinement, and multibaseline matching. Additionally, postprocessing steps for removing outliers, recovering from specific problems of structured environments, and the interpolation of gaps are presented. Finally, strategies for processing almost arbitrarily large images and fusion of disparity images using orthographic projection are proposed. A comparison on standard stereo images shows that SGM is among the currently top-ranked algorithms and is best, if subpixel accuracy is considered. The complexity is linear to the number of pixels and disparity range, which results in a runtime of just 1-2 seconds on typical test images. An in depth evaluation of the Ml-based matching cost demonstrates a tolerance against a wide range of radiometric transformations. Finally, examples of reconstructions from huge aerial frame and pushbroom images demonstrate that the presented ideas are working well on practical problems.

/pdf/stereo-processing-by-semiglobal-matching-and-mutual-147l7nvp3i.pdf

Stereo Processing by Semiglobal Matching and Mutual Information

This paper considers the objectives of accurate stereo matching, especially at object boundaries, robustness against recording or illumination changes and efficiency of the calculation. These objectives lead to the proposed semi-global matching method that performs pixelwise matching based on mutual information and the approximation of a global smoothness constraint. Occlusions are detected and disparities determined with sub-pixel accuracy. Additionally, an extension for multi-baseline stereo images is presented. There are two novel contributions. Firstly, a hierarchical calculation of mutual information based matching is shown, which is almost as fast as intensity based matching. Secondly, an approximation of a global cost calculation is proposed that can be performed in a time that is linear to the number of pixels and disparities. The implementation requires just 1 second on typical images.

Accurate and efficient stereo processing by semi-global matching and mutual information

Extraction of three-dimensional structure of a scene from stereo images is a problem that has been studied by the computer vision community for decades. Early work focused on the fundamentals of image correspondence and stereo geometry. Stereo research has matured significantly throughout the years and many advances in computational stereo continue to be made, allowing stereo to be applied to new and more demanding problems. We review recent advances in computational stereo, focusing primarily on three important topics: correspondence methods, methods for occlusion, and real-time implementations. Throughout, we present tables that summarize and draw distinctions among key ideas and approaches. Where available, we provide comparative analyses and we make suggestions for analyses yet to be done.

https://www.cs.jhu.edu/~burschka/publications/StereoPAMI.pdf

Advances in computational stereo

Stereo correspondence methods rely on matching costs for computing the similarity of image locations. In this paper we evaluate the insensitivity of different matching costs with respect to radiometric variations of the input images. We consider both pixel-based and window-based variants and measure their performance in the presence of global intensity changes (e.g., due to gain and exposure differences), local intensity changes (e.g., due to vignetting, non-Lambertian surfaces, and varying lighting), and noise. Using existing stereo datasets with ground-truth disparities as well as six new datasets taken under controlled changes of exposure and lighting, we evaluate the different costs with a local, a semi-global, and a global stereo method.

/pdf/evaluation-of-cost-functions-for-stereo-matching-2dqk9akhtz.pdf

Evaluation of Cost Functions for Stereo Matching

Stereo correspondence methods rely on matching costs for computing the similarity of image locations. We evaluate the insensitivity of different costs for passive binocular stereo methods with respect to radiometric variations of the input images. We consider both pixel-based and window-based variants like the absolute difference, the sampling-insensitive absolute difference, and normalized cross correlation, as well as their zero-mean versions. We also consider filters like LoG, mean, and bilateral background subtraction (BilSub) and nonparametric measures like Rank, SoftRank, Census, and Ordinal. Finally, hierarchical mutual information (HMI) is considered as pixelwise cost. Using stereo data sets with ground-truth disparities taken under controlled changes of exposure and lighting, we evaluate the costs with a local, a semiglobal, and a global stereo method. We measure the performance of all costs in the presence of simulated and real radiometric differences, including exposure differences, vignetting, varying lighting, and noise. Overall, the ranking of methods across all data sets and experiments appears to be consistent. Among the best costs are BilSub, which performs consistently very well for low radiometric differences; HMI, which is slightly better as pixelwise matching cost in some cases and for strong image noise; and Census, which showed the best and most robust overall performance.

/pdf/evaluation-of-stereo-matching-costs-on-images-with-55ka8dolp2.pdf

Evaluation of Stereo Matching Costs on Images with Radiometric Differences

Binocular half-occlusion points are those that are visible in one of the two views provided by a binocular imaging system. Due to their importance in binocular matching as well as, subsequent interpretation tasks, a number of approaches have been developed for dealing with such points. In the current paper, we consider five methods that explicitly detect half-occlusions and report on a more uniform comparison than has previously been performed. Taking a disparity image and its associated match goodness image as input, we generate images that show the half-occluded points in the underlying scene. We quantitatively and qualitatively compare these methods under a variety of conditions.

/pdf/detecting-binocular-half-occlusions-empirical-comparisons-of-38eckwb1tr.pdf

Detecting binocular half-occlusions: empirical comparisons of five approaches

Traditional stereo systems often falter over changes in lighting between their two views. Unfortunately, such changes often occur when using stereo with a wide baseline or between images from different spectra. In this paper, we propose a new dense stereo correspondence similarity metric, mutual information, which has the potential to overcome such adverse conditions. We explore the strengths and weaknesses of this metric, both quantitatively and qualitatively, under a variety of conditions. Throughout the exploration, we compare mutual information to a more traditional cross-correlation stereo system. We show that mutual information performs under conditions in which traditional dense stereo fails.

/pdf/mutual-information-as-a-stereo-correspondence-measure-3ekwdqszgn.pdf

Mutual Information as a Stereo Correspondence Measure

A stereo confidence metric using single view imagery with comparison to five alternative approaches

/pdf/image-registration-using-mutual-information-3gxo8we1ve.pdf

Image Registration Using Mutual Information

Binocular half-occlusion points are those that are visible in one of the two views provided by a binocular imaging system. Due to their importance in binocular matching as well as subsequent interpretation tasks, a number of approaches have been developed for dealing with such points. In the current paper, we consider four methods that explicitly detect half-occlusions and report on a more uniform comparison than has previously been performed. Taking a disparity image and its associated match goodness image as input, we generate images that show the half-occluded points in the underlying scene. We quantitatively and qualitatively compare these methods under a variety of conditions.

/pdf/detecting-binocular-half-occlusions-empirical-comparisons-of-th2f2ttd5d.pdf

Geoffrey Egnal

Papers

Detecting binocular half-occlusions: empirical comparisons of five approaches

Mutual Information as a Stereo Correspondence Measure

A stereo confidence metric using single view imagery with comparison to five alternative approaches

Image Registration Using Mutual Information

Detecting binocular half-occlusions: empirical comparisons of four approaches