Robust Wide Baseline Stereo from
Maximally Stable Extremal Regions
J. Matas
1,2
, O. Chum
1
,M.Urban
1
,T.Pajdla
1
1
Center for Machine Perception, Dept. of Cybernetics, CTU Prague, Karlovo n´am 13, CZ 121 35
2
CVSSP, University of Surrey, Guildford GU2 7XH, UK
[matas, chum]@cmp.felk.cvut.cz
Abstract
The wide-baseline stereo problem, i.e. the problem of establishing correspon-
dences between a pair of images taken from different viewpoints is studied.
A new set of image elements that are put into correspondence, the so
called extremal regions, is introduced. Extremal regions possess highly de-
sirable properties: the set is closed under 1. continuous (and thus projective)
transformation of image coordinates and 2. monotonic transformation of im-
age intensities. An efficient (near linear complexity) and practically fast de-
tection algorithm (near frame rate) is presented for an affinely-invariant stable
subset of extremal regions, the maximally stable extremal regions (MSER).
A new robust similarity measure for establishing tentative correspon-
dences is proposed. The robustness ensures that invariants from multiple
measurement regions (regions obtained by invariant constructions from ex-
tremal regions), some that are significantly larger (and hence discriminative)
than the MSERs, may be used to establish tentative correspondences.
The high utility of MSERs, multiple measurement regions and the robust
metric is demonstrated in wide-baseline experiments on image pairs from
both indoor and outdoor scenes. Significant change of scale (3.5×), illumi-
nation conditions, out-of-plane rotation, occlusion , locally anisotropic scale
change and 3D translation of the viewpoint are all present in the test prob-
lems. Good estimates of epipolar geometry (average distance from corre-
sponding points to the epipolar line below 0.09 of the inter-pixel distance)
are obtained.
1 Introduction
Finding reliable correspondences in two images of a scene taken from arbitrary view-
points viewed with possibly different cameras and in different illumination conditions is a
difficult and critical step towards fully automatic reconstruction of 3D scenes [5]. A cru-
cial issue is the choice of elements whose correspondence is sought. In the wide-baseline
set-up, local image deformations cannot be realistically approximated by translation or
translation with rotation and a full affine model is required. Correspondence cannot be
therefore established by comparing regions of a fixed (Euclidean) shape like rectangles or
circles since their shape is not preserved under affine transformation.
In most images there are regions that can be detected with high repeatability since they
posses some distinguishing, invariant and stable property. We argue that such regions of,
384
BMVC 2002 doi:10.5244/C.16.36
in general, data-dependent shape, called distinguished regions (DRs) in the paper, may
serve as the elements to be put into correspondence either in stereo matching or object
recognition.
The first contribution of the paper is the introduction of a new set of distinguished
regions, the so called extremal regions. Extremal regions have two desirable properties.
The set is closed under continuous (and thus perspective) transformation of image coor-
dinates and, secondly, it is closed under monotonic transformation of image intensities.
An efficient (near linear complexity) and practically fast detection algorithm is presented
for an affinely-invariant stable subset of extremal regions, the maximally stable extremal
regions (MSER). Robustness of a particular type of DR depends on the image data and
must be tested experimentally. Successful wide-baseline experiments on indoor and out-
door datasets presented in Section 4 demonstrate the potential of MSERs.
Reliable extraction of a manageable number of potentially corresponding image ele-
ments is a necessary but certainly not a sufficient prerequisite for successful wide-baseline
matching. With two sets of distinguished regions, the matching problem can be posed as
a search in the correspondence space [3]. Forming a complete bipartite graph on the two
sets of DRs and searching for a globally consistent subset of correspondences is clearly
out of question for computational reasons. Recently, a whole class of stereo matching
and object recognition algorithms with common structure has emerged [9, 15, 1, 16, 2,
13, 7, 6]. These methods exploit local invariant descriptors to limit the number of tenta-
tive correspondences. Important design decisions at this stage include: 1. the choice of
measurement regions, i.e. the parts of the image on which invariants are computed, 2. the
method of selecting tentative correspondences given the invariant description and 3. the
choice of invariants.
Typically, distinguished regions or their scaled version serve as measurement regions
and tentative correspondences are established by comparing invariants using Mahalanobis
distance [10, 16, 11]. As a second novelty of the presented approach, a robust similar-
ity measure for establishing tentative correspondences is proposed to replace the Maha-
lanobis distance. The robustness of the proposed similarity measure allows us to use
invariants from a collection of measurement regions, even some that are much larger than
the associated distinguished region. Measurements from large regions are either very
discriminative (it is very unlikely that two large parts of the image are identical) or com-
pletely wrong (e.g. if orientation or depth discontinuity becomes part of the region). The
former helps establishing reliable tentative (local) correspondences, the influence of the
latter is limited due to the robustness of the approach.
Finding epipolar geometry consistent with the largest number of tentative (local) cor-
respondences is the final step of all wide-baseline algorithms.
RANSAC has been by far
the most widely adopted method since [14]. The presented algorithm takes novel steps
to increase the number of matched regions and the precision of the epipolar geometry.
The rough epipolar geometry estimated from tentative correspondences is used to guide
the search for further region matches. It restricts location to epipolar lines and provides
an estimate of affine mapping between corresponding regions. This mapping allows the
use of correlation to filter out mismatches. The process significantly increases precision
of the EG estimate; the final average inlier distance-from-epipolar-line is below 0.1 pixel.
For details see Section 3.
Related work. Since the influential paper by Schmid and Mohr [11] many image
matching and wide-baseline stereo algorithms have been proposed, most commonly using
385
Image I is a mapping I : D⊂Z
2
→S. Extremal regions are well defined on images if:
1. S is totally ordered, i.e. reflexive, antisymmetric and transitive binary relation ≤
exists. In this paper only S = {0, 1,...,255} is considered, but extremal regions
can be defined on e.g. real-valued images (S = R).
2. An adjacency (neighbourhood) relation A ⊂D×Dis defined. In this paper
4-neighbourhoods are used, i.e. p, q ∈Dare adjacent (pAq)iff
d
i=1
|p
i
−q
i
|≤1.
Region Q is a contiguous subset of D, i.e. for each p, q ∈Qthere is a sequence
p, a
1
,a
2
,...,a
n
,qand pAa
1
,a
i
Aa
i+1
,a
n
Aq.
(Outer) Region Boundary ∂Q = {q ∈D\Q: ∃p ∈Q: qAp}, i.e. the boundary ∂Q of
Q is the set of pixels being adjacent to at least one pixel of Q but not belonging to Q.
Extremal Region Q⊂D is a region such that for all p ∈Q,q ∈ ∂Q : I(p) >I(q)
(maximum intensity region) or I(p) <I(q) (minimum intensity region).
Maximally Stable Extremal Region (MSER). Let Q
1
,...,Q
i−1
, Q
i
,...be a sequence
of nested extremal regions, i.e. Q
i
⊂Q
i+1
.ExtremalregionQ
i
∗
is maximally stable iff
q(i)=|Q
i+∆
\Q
i−∆
|/|Q
i
| has a local minimum at i
∗
(|.| denotes cardinality). ∆ ∈S
is a parameter of the method.
Table 1: Definitions used in Section 2
Harris interest points as distinguished regions. Tell and Carlsson [13] proposed a method
where line segments connecting Harris interest points form measurement regions. The
measurements are characterised by scale invariant Fourier coefficients. The Harris interest
detector is stable over a range of scales, but defines no scale or affine invariant measure-
ment region. Baumberg [1] applied an iterative scheme originally proposed by Lindeberg
and Garding to associate affine-invariant measurement regions with Harris interest points.
In [7], Mikolajczyk and Schmid show that a scale-invariant MR can be found around
Harris interest points. In [9], Pritchett and Zisserman form groups of line segments and
estimate local homographies using parallelograms as measurement regions. Tuytelaars
and Van Gool introduced two new classes of affine-invariant distinguished regions, one
based on local intensity extrema [16] the other using point and curve features [15]. In
the latter approach, DRs are characterised by measurements from inside an ellipse, con-
structed in an affine invariant manner. Lowe [6] describes the ’Scale Invariant Feature
Transform’ approach which produces a scale and orientation-invariant characterisation of
interest points.
The rest of the paper is structured as follows. Maximally Stable Extremal Regions
are defined and their detection algorithm is described in Section 2. In Section 3, details
of a novel robust matching algorithm are given. Experimental results on outdoor and
indoor images taken with an uncalibrated camera are presented in Section 4. Presented
experiments are summarized and the contributions of the paper are reviewed in Section 5.
2 Maximally Stable Extremal Regions
In this section, we introduce a new type of image elements useful in wide-baseline match-
ing — the Maximally Stable Extremal Regions. The regions are defined solely by an
extremal property of the intensity function in the region and on its outer boundary.
The concept can be explained informally as follows. Imagine all possible threshold-
ings of a gray-level image I. We will refer to the pixels below a threshold as ’black’ and
386
to those above or equal as ’white’. If we were shown a movie of thresholded images I
t
,
with frame t corresponding to threshold t, we would see first a white image. Subsequently
black spots corresponding to local intensity minima will appear and grow. At some point
regions corresponding to two local minima will merge. Finally, the last image will be
black. The set of all connected components of all frames of the movie is the set of all
maximal regions; minimal regions could be obtained by inverting the intensity of I and
running the same process. The formal definition of the MSER concept and the necessary
auxiliary definitions are given in Table 1.
In many images, local binarization is stable over a large range of thresholds in certain
regions. Such regions are of interest since they posses the following properties:
• Invariance to affine transformation of image intensities.
• Covariance to adjacency preserving (continuous) transformation T : D→Don
the image domain.
• Stability, since only extremal regions whose support is virtually unchanged over a
range of thresholds is selected.
• Multi-scale detection. Since no smoothing is involved, both very fine and very
large structure is detected.
• The set of all extremal regions can be enumerated in O(n log log n),wheren is
the number of pixels in the image.
Enumeration of extremal regions proceeds as follows. First, pixels are sorted by inten-
sity. The computational complexity of this step is O(n) if the range of image values S is
small, e.g. the typical {0,...,255}, since the sort can be implemented as
BINSORT [12].
After sorting, pixels are placed in the image (either in decreasing or increasing order) and
the list of connected components and their areas is maintained using the efficient union-
find algorithm [12]. The complexity of our union-find implementation is O(n log log n),
i.e. almost linear
1
. Importantly, the algorithm is very fast in practice. The MSER detec-
tion takes only 0.14 seconds on a Linux PC with the Athlon XP 1600+ processor for an
530x350 image (n = 185500).
The process produces a data structure storing the area of each connected component as
a function of intensity. A merge of two components is viewed as termination of existence
of the smaller component and an insertion of all pixels of the smaller component into the
larger one. Finally, intensity levels that are local minima of the rate of change of the area
function are selected as thresholds producing maximally stable extremal regions. In the
output, each MSER is represented by position of a local intensity minimum (or maximum)
and a threshold.
Notes. The structure of the above algorithm and of an efficient watershed algorithm
[17] is essentially identical. However, the structure of the output of the two algorithms
is different. The watershed is a partitioning of D, i.e. a set of regions R
i
:
R
i
=
D, R
j
∩R
k
= ∅. In watershed computation, focus is on the thresholds where regions
merge (and two watersheds touch). Such threshold are of little interest here, since they are
highly unstable – after merge, the region area jumps. In MSER detection, we seek a range
of thresholds that leaves the watershed basin effectively unchanged. Detection of MSER
is also related to thresholding. Every extremal region is a connected component of a
1
even faster (but more complex) connected component algorithms exist with O(nα(n)) complexity, where
α is the inverse Ackerman function; α(n) ≤ 4 for all practical n.
387
thresholded image. However, no global or ’optimal’ threshold is sought, all thresholds
are tested and the stability of the connected components evaluated. The output of the
MSER detector is not a binarized image. For some parts of the image, multiple stable
thresholds exist and a system of nested subsets is output in this case. Finally we remark
that MSERs can be defined on any image (even high-dimensional) whose pixel values are
from a totally ordered set.
3 The proposed robust wide-baseline algorithm
Distinguished region detection. As a first step, the DRs are detected - the MSERs com-
puted on the intensity image (MSER+) and on the inverted image (MSER-).
Measurement regions. A measurement region of arbitrary size may be associated with
each DR, if the construction is affine-covariant. Smaller measurement regions are both
more likely to satisfy the planarity condition and not to cross a discontinuity in depth or
orientation. On the other hand, small regions are less discriminative, i. e. they are much
less likely to be unique. Increasing the size of a measurement region carries the risk of
including parts of background that are completely different in the two images considered.
Clearly, the optimal size of a MR depends on the scene content and it is different for each
DR. In [16], Tuytelaars at al. double the elliptical DR to increase discriminability, while
keeping the probability of crossing object boundaries at an acceptable level.
In the proposed algorithm, measurement regions are selected at multiple scales: the
DR itself, 1.5, 2 and 3 times scaled convex hull of the DR. Since matching is accomplished
in a robust manner, we benefit from the increase of distinctiveness of large regions with-
out being severely affected by clutter or non-planarity of the DR’s pre-image. This is a
novelty of our approach. Commonly, Mahalanobis distance has been used in MR match-
ing. However, the non-robustness of this metric means that matching may fail because of
a single corrupted measurement (this happened in the experiments reported below).
Invariant description. In all experiments, rotational invariants (based on complex
moments) were used after applying a transformation that diagonalises the regions covari-
ance matrix of the DR. In combination, this is an affinely-invariant procedure. Combi-
nation of rotational and affinely invariant generalised colour moments [8] gave a similar
result. On their own, these affine invariants failed on problems with a large scale change.
Robust matching. A measurement taken from an almost planar patch of the scene
with stable invariant description will be referred to as a ’good measurement’. Unstable
measurements or those computed on non-planar surfaces or at discontinuities in depth or
orientation will be referred to as ’corrupted measurements’.
The robust similarity is computed as follows. For each measurement M
i
A
on region
A, k regions B
1
,...,B
k
from the other image with the corresponding i-th measurement
M
i
B
1
,...,M
i
B
k
nearest to M
i
A
are found and a vote is cast suggesting correspondence of
A and each of B
1
,...,B
k
.
Votes are summed over all measurements. In the current implementation 216 invari-
ants at each scale, i.e. a total of 864 measurements are used (i ∈ [1, 864]). The DRs with
the largest number of votes are the candidates for tentative correspondences. Experimen-
tally, we found that k set to 1% of the number of regions gives good results.
Probabilistic analysis of the likelihood of the success of the procedure is not simple,
since the distribution of invariants and their noise is image-dependent. We therefore only
suppose that corrupted measurements spread their votes randomly, not conspiring to cre-
ate a high score and that good measurements are more likely to vote for correct matches.
388