How does the proposed algorithm achieve consistency between scanlines?

where consistency between scanlines were imposed using only GCPs, the authors guarantee the consistency by the two-pass dynamic programming.

Why do the authors have scanline inconsistency problem?

Due to the use of GGCPs, the disparity maps provided by the PASS 1 show better inter-scanline consistency than the conventional single-pass dynamic programming, but the authors still have scanline inconsistency problem.

(Open Access) A dense stereo matching using two-pass dynamic programming with generalized ground control points (2005) | Jae-Chul Kim

Q: What is the motivation of this paper?

In this sense, the second motivation of their paper is to develop a fast matching algorithm, while achieving the accuracy comparable to the state-of-thearts [5, 9, 13, 19].

A Dense Stereo Matching Using Two-Pass Dynamic Programming with

Generalized Ground Control Points

Jae Chul Kim

, Kyoung Mu Lee

, Byoung Tae Choi

, Sang Uk Lee

1,3

Electronics and Telecommunications Research Institute, 305-350, Daejon, Korea

2,4

School of Electrical Eng., ASRI, Seoul National University, 151-600, Seoul Korea

e-mail: jaechul@etri.re.kr, kyoungmu@snu.ac.kr, btchoi@etri.re.kr, sanguk@ipl.snu.ac.kr

Abstract

A method for solving dense stereo matching

problem is presented in this paper. First, a new

generalized ground control points (GGCPs) scheme is

introduced, where one or more disparity candidates for

the true disparity of each pixel are assigned by local

matching using the oriented spatial filters. By allowing

“all” pixels to have multiple candidates for their true

disparities, GGCPs not only guarantee to provide a

sufficient number of starting pixels needed for guiding

the subsequent matching process, but also remarkably

reduce the risk of false match, improving the previous

GCP-based approaches where the number of the

selected control points tends to be inversely

proportional to the reliability. Second, by employing a

two-pass dynamic programming technique that

performs optimization both along and across the

scanlines, we solve the typical inter-scanline

inconsistency problem. Moreover, combined with the

GGCPs, the stability and efficiency of the optimization

are improved significantly. Experimental results for the

standard data sets show that the proposed algorithm

achieves comparable results to the state-of-the-arts

with much less computational cost.

1. Introduction

1.1. Motivation

Stereo matching is a problem to find corres-

pondences between two or more input images. It is one

of fundamental computer vision problems with a wide

range of applications, and hence it has been extensively

studied in the computer vision field for decades.

However, there still exist some difficult inherent

problems in stereo matching; for example, the presence

of homogeneously textured regions, and the occlusions

near the object boundaries that make the disparity

assignment very difficult.

To resolve these difficulties, numerous attempts

have been made to lessen the matching ambiguities by

propagating the reliable matching results [4, 8, 20, 22,

23]. In these reliability-based approaches, one of the

most important tasks is to select the reliably matched

pixels, i.e. ground control points (GCPs). It is known

that the false matches in GCPs could severely degrade

the final matching results. On the other hand, the

number of the obtained GCPs would decrease if stricter

constraints are enforced for outlier removals, which in

turn could lead to the lack of information needed for

appropriately guiding the subsequent matching process.

The first motivation of our paper is to solve those

problems of conventional GCP-based approaches. To

this end, we propose the generalized ground control

points (GGCPs) scheme in which unlike conventional

GCP-based approaches where only reliably matched

pixels are selected, multiple disparity candidates are

assigned to all pixels by local matching using the

oriented spatial filters. Using this scheme, the

probability of false match drops remarkably, and

furthermore sufficient information is always provided

for dense matching, since all pixels take part in guiding

the subsequent matching process without loss of

reliability.

GCPs or GGCPs can be applied to various matching

techniques [4, 8, 11, 22]. In this paper, GGCPs are

applied to global optimization using efficient dynamic

programming. In this sense, the second motivation of

our paper is to develop a fast matching algorithm, while

achieving the accuracy comparable to the state-of-the-

arts [5, 9, 13, 19]. So, we propose a two-pass dynamic

programming technique. The proposed two-pass

dynamic programming is designed to resolve the

inconsistency between scanlines, which is the typical

problem in conventional dynamic programming. It

performs optimization both along and across the

Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05)

scanlines. Furthermore, since the finite number of

disparity candidates of GGCPs not only reduce the

range of possible disparities to be searched, but also

provide good initial points for optimization, the

optimization becomes more efficient and stable.

1.2. Related works

The proposed algorithm has the workflow in which

first, the local matching using spatial filters is carried

out, and then the results of local matching is applied to

the global optimization. This approach has been already

adopted in several algorithms [1, 4, 11]. In particular,

our algorithm has the similar framework to Bobick et

al.’s algorithm [4], where GCPs were used together

with dynamic programming. But, we propose the

GGCPs as extension of the GCPs, and unlike the work

of Bobick et al. where consistency between scanlines

were imposed using only GCPs, we guarantee the

consistency by the two-pass dynamic programming.

These features bring about remarkable improvement in

matching accuracy.

In this paper, disparity candidates of each pixel, i.e.

GGCPs, are obtained from local matching by the

oriented spatial filters. These oriented filters have a few

advantages over the windows commonly used in stereo

matching. First, they can delineate the object

boundaries more clearly. Second, even when the

oriented filters are applied to the slanted plane, at least

one filter among the filters with various orientations

satisfies the fronto-parallel plane assumption, and

therefore more accurate matching results for the slanted

planes can be provided. Additionally, in order to take

the best advantages of the oriented filters for stereo, it

is desirable for the filters to have high resolution in

orientation. To this end, we adopt the oriented rod-

shaped filters, instead of the Gaussian-based filters that

used commonly in conventional algorithms [10, 12]. It

can be shown that the coefficients of the rod-shaped

filter are more concentrated along the orientation of

filter that leads to higher resolution in orientation (see

figure 1). The detailed description on the rod-shaped

filter will be presented in section 2.

Finally, there have been many works to solve the

scanline inconsistency problem of dynamic

programming [2, 3, 6, 14]. For examples, Birchfield et

al. [3] conducted a post-processing using heuristics,

and Cox et al. [6] locally dealt with the inconsistency

problem by minimizing the discontinuities between

neighboring scanlines. But, these algorithms only

offered partial remedy for the inconsistency problem.

The proposed algorithm carries out the two-pass

dynamic programming using the scanline optimization

[17] without consideration of the ordering constraint.

By excluding the ordering constraint from optimization

process, we can readily perform the optimization across

the scanlines, by the same manner as the one used in

the optimization along the scanlines. Here, we should

note that our idea on the two-pass dynamic

programming is inspired from the algorithm proposed

by Zickler et al. [24] who applied the two-pass dynamic

programming to binocular Helmholtz stereopsis.

However, we adapt the two-pass dynamic programming

for stereo matching. Furthermore, by combining the

two-pass dynamic programming with the information

from GGCPs, we can obtain a remarkably enhanced

solution for inter-scanline inconsistency problem.

2. Preliminaries

For convenience, we assume that input images are

rectified. Then, the correspondences between input

images are represented by a univalued disparity

function

(, )dxy with respect to a pixel (, )xy of the

reference image. The disparity function can take one of

integer values within the disparity ranges of the scene.

A pair of a pixel

(, )xy and its disparity d generates

a point

(, , )xyd , which constructs a 3D disparity space.

An initial matching cost

(, , )Cxyd measures the

pixel-based error of a match at the point ( , , )xyd . The

simplest matching cost uses absolute intensity

differences between a pixel ( , )xy of the reference

(left) image

and a pixel ( , )xdy− of the matching

(right) image

, i.e.

012

(, , ) (, ) ( , )Cxyd Ixy Ixdy=−−.

In the proposed algorithm, the rod-shaped spatial

filters with N orientations are used. Examples of the

filters are illustrated in Figure 1 where each filter is

rotated by

. Generally, the rod-shaped filter which

21l + pixels long, and inclined at

to the horizontal

axis can be numerically expressed as

1 sin cos if sin cos < 1,

(, )

0 otherwise

xy xy

fxy

θθ θθ

 −− −

(1)

for

<cosxl

and <sin .yl

To avoid the problem that filters are across the

object boundaries, we perform local matching using

three filters for each orientation, where the centers of

the filters are shifted to the three different positions,

and only the best filtering result is taken. The shifted

versions of the oriented filter are shown in figure 2. The

shiftable filtering is implemented by a cascade of a

Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05)

single oriented filtering and a minimum (or maximum)

finding for the three center points instead of three

repetitive filtering for each center point. So, only little

additional computation is required for the shiftable

filtering. For more information on the shiftable

filtering, interested readers may refer to [15, 17].

3. Local matching

In local matching, the disparity candidates for true

disparity of each pixel are obtained. And these

candidates are computed by the sequential operation of

following three steps: preprocessing, local aggregation,

and post-processing.

3.1. Preprocessing

The preprocessing aims to classify each pixel in the

reference image into two groups (homogeneous group

and heterogeneous group) according to the intensity

variation around each pixel. To compute the intensity

variation of each pixel, small-sized (for example,

dimension of 3

× 3) Laplacian of Gaussian filter is first

applied to the reference image, followed by shiftable

oriented filters with N orientations in order to take

account of the intensity variations along the

neighborhood of the each orientation where local

aggregation is performed. In the shiftable filtering, a

minimum finding (not maximum) is used for picking

the best result. Since the shiftable filtering is carried out

independently for N orientations, N minimum values

are assigned to each pixel as the intensity variations

around it. Finally, if at least one of the N values of a

pixel is over a threshold, the pixel is labeled as

heterogeneous pixel; otherwise, it is labeled as

homogeneous pixel. Thus, a homogeneous pixel has no

significant intensity variation for any orientation along

which the filtering is executed.

3.2. Local Aggregation

In this stage, candidates for the true disparity of each

pixel are provided by locally aggregating the initial

matching costs using the spatial filters. The detailed

procedure for determining the disparity candidates of

each pixel is as follows. First, the initial matching cost

(, , )Cxyd for each pair of a pixel ( , )xy and its

disparity d is evaluated by

(, , )

( , ) ( , ) if ( , )

( , ) ( , ) if ( , ) ,

Cxyd

I x y I x d y x y homogeneous

g I x y g I x d y x y heterogeneous

 −− ∈

⊗−⊗− ∈

(2)

Figure 1.

Examples of the rod-shaped oriented filters

Figure 2.

Diagram of the three shiftable oriented filters,

where the centers of the filters are marked in black.

where

denotes the Gaussian kernel, and

(1,2)

gIi⊗= represents the convolution. Note that the

initial costs of the heterogeneous pixels are computed

from the smoothed input images by Gaussian filter.

This low-pass filtering helps to suppress the sampling

artifacts in the intensity-varying areas, which are known

to be quite common in a kind of small-sized box filters

like our rod-shaped filters [21]. After computing the

initial cost, the aggregation of the initial matching cost

is implemented using the shiftable oriented filters.

However, note that in order to reduce the matching

ambiguities induced by the lack of intensity variation in

the homogeneous pixels, large-sized shiftable windows

are additionally applied to the homogeneous pixels. In

mathematical terms, the aggregation can be expressed

(, , ) ( , ) ( , , ), (3)

Cxyd f x my n C x my nd=−−⋅−−

¦¦

where

denotes a 2D spatial filter. Notice that in our

algorithm, the aggregation is performed for N

orientations, so that N aggregated costs are assigned to

each pixel-disparity pair. Of course, N+1 costs

including the one from the shiftable windows are given

to the homogeneous pixels. Finally, at each pixel the

best disparity associated with the minimum cost value

is selected. Since each pixel has N (or N+1)

aggregation results, the N (or N+1) best disparities are

stored in each pixel, and these best disparities are

established as the disparity candidates of each pixel. In

addition, the aggregated costs at the best disparities

become the matching costs of the disparity candidates.

Specially, if the same best disparity is produced from

multiple filters, the smallest one among each filter’s

aggregated costs is assigned to the disparity. The

Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05)

disparities other than candidates are excluded in the

subsequent matching process by setting their matching

costs to be very large.

3.3. Postprocessing

To enhance the reliability of local aggregation, some

heuristic methods are used; these are similar to those

used by Bobick et al. [4] to identify the GCPs.

Visibility test Visibility test confirms the

consistency of the bi-directional matching based on the

uniqueness assumption, and mainly aims to eliminate

the matching ambiguities occurred by occlusions. Let

min1

d be the winner-take-all disparity of a pixel ( , )xyof

the reference image, i.e.

min1

arg min ( , , )

dCxyd= ,

and

min 2

d be the winner-take-all disparity of a pixel

min1

(,)xd y− of the matching image, i.e.

min 2 min1

arg min ( , , ).

dCxddyd=−+

min1 min 2

,dd≠ the pixel ( , )xyfails to pass the visibility

test, and its local aggregation result is invalidated by

equalizing the matching costs of all candidates at the

pixel ( , )xy to an arbitrary value (zero in our algorithm),

which eliminates the difference between the

aggregation results of the candidates. Pay attention to

that the matching costs of the disparities other than the

disparity candidates are kept as an initially assigned

value, i.e. very large predefined value.

Detection of suspicious pixels Due to some reasons

such as specularity or perfectly homogeneous texture,

there exist pixels whose results of local aggregation

cannot be trusted. We separate such pixels according to

the following rules. First, if the minimum matching cost

of a pixel exceeds a threshold

t , the pixel is marked as

an suspicious one. In addition, for a homogeneous pixel,

if the difference between its first minimum matching

cost and the second one is below a threshold

t , that

pixel is also labeled as suspicious. The aggregation

results of these suspicious pixels are then invalidated by

the same manner used in the visibility test except for

one difference: the matching costs at all disparities are

annulled without any distinction between the disparity

candidates and the remainders. This is because the local

aggregation result for a suspicious pixel is provided in

such an unpredictable way that it is not credible that a

true disparity of the pixel exists in its disparity

candidates, whereas matching ambiguities of a pixel by

occlusions near the depth discontinuities arise just

between foreground disparity and background one, both

of which are generally included in the disparity

candidates of the pixel.

4. Global optimization

In this section, two-pass dynamic programming is

performed for global optimization using the scanline

optimization [17], rather than the typical dynamic

programming enforcing the ordering constraint. By

excluding the ordering constraint, the scanline

optimization makes it easy to optimize across the

scanlines, where it is impractical to impose the ordering

constraint.

The complexity of the scanline optimization is

()Omn for m pixels and n disparities, which is larger

than ( ),Omn the complexity of the typical dynamic

programming. But, the proposed algorithm can work

very efficiently because it only considers the disparity

candidates of each pixel, but not all disparities within

the search range.

4.1. PASS 1: Optimization along the scanlines

The optimization along the scanlines finds a path of

disparities that minimizes the following energy

functional,

h1 11

11 1

E((, )) (, , (, ))

( , ) ( ( , ) ( 1, ))

dxy Cxy dxy

xy dxy dx y

λρ

−+

(4)

for a scanline

y . In equation (4), C is the matching

cost obtained from the local aggregation,

is an

increasing function of the disparity difference between

adjacent pixels evaluating the smoothness of a disparity

function, and

is a weight function.

As a

function, Potts model [16] has been widely

used since it can handle the disparity jumps, and it is

adopted in our algorithm as well. The Potts model is

()

10.

ρα



≠

(5)

For homogenous pixels with valid matching costs (see

section 3.3), on the other hand, modified Potts model is

used to avoid the excessive smoothing in the

homogeneous textured regions. The modified Potts

model incorporates the disparity gradient constraint

into the original Potts model, and is written as

Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05)

0if 0

'( ) 0.5 if 1 or -1

1otherwise.

ρα α α



===

(6)

In contrast to the original Potts model preferring the

fronto-parallel planes (especially, this preference will

be intensified in the homogeneous pixels where the

smoothness constraint dominates the energy functional

of equation 3.), the modified Potts model encourages

the slanted planes by lowering the cost imposed on one-

pixel sized disparity difference to be one-half. This

strategy aids to diminish the influence of the

smoothness constraint on the slanted surface where

disparities of neighboring pixels commonly vary within

a smaller range than one-pixel difference.

The weight function

has a functional value

inversely proportional to the intensity gradient to help

align the disparity jumps with the intensity edges [4, 5,

7]. In our algorithm,

is defined as

112

11112

0.5 if ( , )

(, ) if (, )

2otherwise,

cIxyi

xy c i I xy i

∇>



=<∇<

(7)

where

c is a constant and

(, )

xy∇ denotes the

horizontal intensity gradient computed by 3 3× sized

horizontal Sobel operator. Both

i and

i are thresholds

for the intensity gradient.

For each pixel-disparity pair

(, , )xy d on a

scanline

y , a typical scanline optimization usually

proceeds to compute the minimum cost

(, , )

Cxydrequired for reaching each pair, and when

finally arriving at the end point

x , the optimal path is

decided as the one that gets to the

1min

(,, ()),

xyd x

where

min

()

dx=

arg min ( , , )

Cxyd. Our algorithm,

however, does not follow this procedure. Instead, the

cost

(, , )

Cxyd is incorporated into the energy

functional defined in the PASS 2 where the final

optimal disparities are chosen, while the optimal

disparities of the PASS 1 just bias the final disparities

toward them by lowering their costs by a constant.

In the typical scanline optimization, the cost

(, , )

Cxyd increases with x, but the same dimensions

with respect to the

x-direction are required to apply the

cost to the energy functional of the PASS 2. To achieve

this symmetry, we use the similar approach to the one

used by Zickler et al. [24]; the cost

(, , )

Cxyd is

obtained by summing

(, , )

Cxyd, the cost from the

computation in the increasing direction of

x, and

(, , )

Cxyd, the cost from the computation in the

reverse direction.

4.2. PASS 2: Optimization across the scanlines

In this pass, the objective is to provide a final

disparity path that minimizes an energy functional,

111 11

111

E((,)) (,,(,)) (,,(,))

( , ) ( ( , ) ( , 1)). (8)

dxy Cxydxy Cxydxy

xy dxy dxy

λρ

+−+

¦¦

for a vertical line

xx= . In equation (8), C is the

matching cost by the local aggregation, identical to the

one used in equation (4), and

C is the cost obtained

from the previous optimization pass, and

enforces a

smoothness on the disparity function as before, but this

time, the original Potts model is used instead of the

modified Potts model on the pragmatic grounds, since it

shows slightly better experimental results. A weight

function

is written as

11 2

11112

0.5 if ( , )

(,) if (,)

2otherwise,

cIxyi

xy c i Ixy i

∇>



=<∇<

(9)

where all parameters are identical to those of equation

(7) except that the intensity gradient is computed by the

33× sized

vertical (not horizontal) Sobel operator

denoted by

(,)

xy∇ . Note that the energy functional

of equation (8) wholly considers three factors: the local

matching result, the result from optimization along the

scanlines, and the smoothness of a disparity function in

the vertical direction. By minimizing this functional, we

can get the disparity map preserving the consistency

between scanlines.

The matching process is finished with the simple

sub-pixel refinement using the following rule: if the

number of disparity candidates of a pixel ( , )

xy is two,

and the disparity difference between candidates is one,

then the sub-pixel refinement for the pixel ( , )

xy is

carried out by the heuristic equation,

sub 0 1

(, ) 0.75 (, ) 0.25 (, ),dxy dxy dxy=+ (10)

Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05)

A dense stereo matching using two-pass dynamic programming with generalized ground control points

Figures

Citations

Review of stereo vision algorithms: From software to hardware

High-Quality Real-Time Stereo Using Adaptive Cost Aggregation and Dynamic Programming

Real-time Global Stereo Matching Using Hierarchical Belief Propagation.

Review of stereo vision algorithms and their suitability for resource-limited systems

Classification and evaluation of cost aggregation methods for stereo correspondence

References

A taxonomy and evaluation of dense two-frame stereo correspondence algorithms

Fast approximate energy minimization via graph cuts

Some generalized order - disorder transformations

High-accuracy stereo depth maps using structured light

Stereo matching using belief propagation

Related Papers (5)

A taxonomy and evaluation of dense two-frame stereo correspondence algorithms

Adaptive support-weight approach for correspondence search

Computing visual correspondence with occlusions using graph cuts

Fast approximate energy minimization via graph cuts

Stereo matching using belief propagation

Frequently Asked Questions (12)

Q1. What are the contributions in "A dense stereo matching using two-pass dynamic programming with generalized ground control points" ?

Q2. What is the main advantage of the oriented filters?

Q3. What is the important task in stereo matching?

Q4. How does the proposed algorithm achieve consistency between scanlines?

Q5. What is the way to optimize a stereo filter?

Q6. Why do the authors have scanline inconsistency problem?

Q7. What is the motivation of this paper?

Q8. What is the proposed algorithm for the scanline inconsistency problem?

Q9. How can the authors perform the optimization across the scanlines?

Q10. What are the problems of stereo matching?

Q11. What is the effect of the rod-shaped filter on the GGCPs?

Q12. What is the difference between the two oriented filters?