Stereo matching is one of the most active research areas in computer vision. While a large number of algorithms for stereo correspondence have been developed, relatively little work has been done on characterizing their performance. In this paper, we present a taxonomy of dense, two-frame stereo methods designed to assess the different components and design decisions made in individual stereo algorithms. Using this taxonomy, we compare existing stereo methods and present experiments evaluating the performance of many different variants. In order to establish a common software platform and a collection of data sets for easy evaluation, we have designed a stand-alone, flexible C++ implementation that enables the evaluation of individual components and that can be easily extended to include new algorithms. We have also produced several new multiframe stereo data sets with ground truth, and are making both the code and data sets available on the Web.

/pdf/a-taxonomy-and-evaluation-of-dense-two-frame-stereo-4c20etkubp.pdf

A taxonomy and evaluation of dense two-frame stereo correspondence algorithms

While different optical flow techniques continue to appear, there has been a lack of quantitative evaluation of existing methods. For a common set of real and synthetic image sequences, we report the results of a number of regularly cited optical flow techniques, including instances of differential, matching, energy-based, and phase-based methods. Our comparisons are primarily empirical, and concentrate on the accuracy, reliability, and density of the velocity measurements; they show that performance can differ significantly among the techniques we implemented.

Performance of optical flow techniques

In this paper, a new technique coined two-dimensional principal component analysis (2DPCA) is developed for image representation. As opposed to PCA, 2DPCA is based on 2D image matrices rather than 1D vectors so the image matrix does not need to be transformed into a vector prior to feature extraction. Instead, an image covariance matrix is constructed directly using the original image matrices, and its eigenvectors are derived for image feature extraction. To test 2DPCA and evaluate its performance, a series of experiments were performed on three face image databases: ORL, AR, and Yale face databases. The recognition rate across all trials was higher using 2DPCA than PCA. The experimental results also indicated that the extraction of image features is computationally more efficient using 2DPCA than PCA.

/pdf/two-dimensional-pca-a-new-approach-to-appearance-based-face-32pf68mvmi.pdf

Two-dimensional PCA: a new approach to appearance-based face representation and recognition

This paper describes the semiglobal matching (SGM) stereo method. It uses a pixelwise, mutual information (Ml)-based matching cost for compensating radiometric differences of input images. Pixelwise matching is supported by a smoothness constraint that is usually expressed as a global cost function. SGM performs a fast approximation by pathwise optimizations from all directions. The discussion also addresses occlusion detection, subpixel refinement, and multibaseline matching. Additionally, postprocessing steps for removing outliers, recovering from specific problems of structured environments, and the interpolation of gaps are presented. Finally, strategies for processing almost arbitrarily large images and fusion of disparity images using orthographic projection are proposed. A comparison on standard stereo images shows that SGM is among the currently top-ranked algorithms and is best, if subpixel accuracy is considered. The complexity is linear to the number of pixels and disparity range, which results in a runtime of just 1-2 seconds on typical test images. An in depth evaluation of the Ml-based matching cost demonstrates a tolerance against a wide range of radiometric transformations. Finally, examples of reconstructions from huge aerial frame and pushbroom images demonstrate that the presented ideas are working well on practical problems.

/pdf/stereo-processing-by-semiglobal-matching-and-mutual-147l7nvp3i.pdf

Stereo Processing by Semiglobal Matching and Mutual Information

Incremental Structure-from-Motion is a prevalent strategy for 3D reconstruction from unordered image collections. While incremental reconstruction systems have tremendously advanced in all regards, robustness, accuracy, completeness, and scalability remain the key problems towards building a truly general-purpose pipeline. We propose a new SfM technique that improves upon the state of the art to make a further step towards this ultimate goal. The full reconstruction pipeline is released to the public as an open-source implementation.

/pdf/structure-from-motion-revisited-10382wusrz.pdf

Structure-from-Motion Revisited

Image segmentation is the process by which an original image is partitioned into some homogeneous regions. In this paper, a novel multiresolution color image segmentation (MCIS) algorithm which uses Markov random fields (MRF's) is proposed. The proposed approach is a relaxation process that converges to the MAP (maximum a posteriori) estimate of the segmentation. The quadtree structure is used to implement the multiresolution framework, and the simulated annealing technique is employed to control the splitting and merging of nodes so as to minimize an energy function and therefore, maximize the MAP estimate. The multiresolution scheme enables the use of different dissimilarity measures at different resolution levels. Consequently, the proposed algorithm is noise resistant. Since the global clustering information of the image is required in the proposed approach, the scale space filter (SSF) is employed as the first step. The multiresolution approach is used to refine the segmentation. Experimental results of both the synthesized and real images are very encouraging. In order to evaluate experimental results of both synthesized images and real images quantitatively, a new evaluation criterion is proposed and developed. >

Multiresolution color image segmentation

First Sight, a vision system in labeling the outline of a moving human body, is proposed in this paper. The emphasis of First Sight is on the analysis of motion information gathered solely from the outline of a moving human object. Two main processes are implemented in First Sight. The first process uses a novel technique to extract the outline of a moving human body from an image sequence. The second process, which employs a new human body model, interprets the outline and produces a labeled two-dimensional human body stick figure for each frame of the image sequence. Extensive knowledge of the structure, shape, and posture of the human body is used in the model. The experimental results of applying the technique on unedited image sequences with self-occlusions and missing boundary lines are encouraging. >

First Sight: A human body outline labeling system

In this paper we present an algorithm that integrates spatial and temporal information for the tracking of moving nonrigid objects. In addition, we obtain outlines of the moving objects.

The background primal sketch: an approach for tracking moving objects

A near-real-time stereo matching technique is presented in this paper, which is based on the reliability-based dynamic programming algorithm we proposed earlier. The new algorithm can generate semi-dense disparity maps using only two dynamic programming passes, while our previous approach requires 20-30 passes. We also implement the algorithm on programmable graphics hardware, which further improves the processing speed. The experiments on the four Middlebury stereo datasets show that the new algorithm can produce dense (>85% of the pixels) and reliable (error rate <0.3%) matches in near real-time (0.05-0.1 sec). If needed, it can also be used to generate dense disparity maps. Based on the evaluation conducted by the Middlebury Stereo Vision Research Website, the new algorithm is ranked between the variable window and the graph cuts approaches and currently is the most accurate dynamic programming based technique. When more than one reference images are available, the accuracy can be further improved with little extra computation time.

Near real-time reliable stereo matching using programmable graphics hardware

In this paper, we propose a fully automatic image colorization method for grayscale images using neural network and optimization. For a determined training set including the gray images and its corresponding color images, our method segments grayscale images into superpixels and then extracts features of particular points of interest in each superpixel. The obtained features and their RGB values are given as input for, the training colorization neural network of each pixel. To achieve a better image colorization effect in shorter running time, our method further propagates the resulting color points to neighboring pixels for improved colorization results. In the propagation of color, we present a cost function to formalize the premise that neighboring pixels should have the maximum positive similarity of intensities and colors; we then propose our solution to solving the optimization problem. At last, a guided image filter is employed to refine the colorized image. Experiments on a wide variety of images show that the proposed algorithms can achieve superior performance over the state-of-the-art algorithms.

Yee-Hong Yang

Papers

Multiresolution color image segmentation

First Sight: A human body outline labeling system

The background primal sketch: an approach for tracking moving objects

Near real-time reliable stereo matching using programmable graphics hardware

Automated Colorization of a Grayscale Image With Seed Points Propagation