A Comparison of Affine Region Detectors
K. Mikolajczyk
1
, T. Tuytelaars
2
, C. Schmid
4
, A. Zisserman
1
,
J. Matas
3
, F. Schaffalitzky
1
, T. Kadir
1
, L. Van Gool
2
1
University of Oxford, OX1 3PJ Oxford, United Kingdom
2
University of Leuven, Kasteelpark Arenberg 10, 3001 Leuven, Belgium
3
Czech Technical University, Karlovo Namesti 13, 121 35, Prague, Czech Republic
4
INRIA, GRAVIR-CNRS, 655, av. de l’Europe, 38330 Montbonnot, France
km@robots.ox.ac.uk, tuytelaa@esat.kuleuven.ac.be, schmid@inrialpes.fr,
az@robots.ox.ac.uk, matas@cmp.felk.cvut.cz, fsm@robots.ox.ac.uk,
tk@robots.ox.ac.uk, vangool@esat.kuleuven.ac.be
Abstract
The pap er gives a snapshot of the state of the art in affine covariant region detectors, and
compares their p erformance on a set of test images under varying imaging conditions. Six
types of detectors are included: detectors based on affine normalization around Harris [24,
34] and Hessian points [24], as proposed by Mikolajczyk and Schmid and by Schaffalitzky
and Zisserman; a detector of ‘maximally stable extremal regions’, proposed by Matas et
al. [21]; an edge-based region detector [45] and a detector based on intensity extrema [47],
prop os ed by Tuytelaars and Van Gool; and a detector of ‘salient regions’, proposed by Kadir,
Zisserman and Brady [12]. The performance is measured against changes in viewpoint, scale,
illumination, defocus and image compression.
The objective of this paper is also to establish a reference test set of images and perfor-
mance software, so that future detectors can be evaluated in the same framework.
1 Introduction
Detecting regions covariant with a class of transformations has now reached some maturity in the
computer vision literature. These regions have been used in quite varied applications including:
wide baseline matching for stereo pairs [1, 21, 31, 47], reconstructing cameras for sets of disparate
views [34], image retrieval from large databases [36, 45], model based recognition [7, 18, 29, 32],
object retrieval in video [39, 40], visual data m ining [41], texture recognition [13, 14], shot lo-
cation [35], robot localization [37] and servoing [46], building panoramas [2], symmetry detec-
tion [44], and object categorization [4, 5, 6, 30].
The requirement for these regions is that they should correspond to the same pre-image for
different viewpoints, i.e., their shape is not fixed but automatically adapts, based on the un-
derlying image intensities, so that they are the projection of the same 3D surface patch. In
particular, consider images from two viewpoints and the geometric transformation between the
images induced by the viewpoint change. Regions detected after the viewpoint change should
be the same, modulo noise, as the transformed versions of the regions detected in the original
image – image transformation and region detection commute. As such, even though they have
often been called invariant regions in the literature (e.g., [5, 13, 41, 45]), in principle they should
be termed covariant regions since they change covariantly with the transformation. The confu-
sion probably arises from the fact that, even though the regions themselves are covariant, the
1
(a) (b) (c)
(d) (e) (f)
Figure 1: Class of transformations needed to cope with viewpoint changes. (a) First
viewpoint; (b,c) second viewpoint. Fixed size circular patches (a,b) clearly do not suffice to deal
with general viewpoint changes. What is needed is an anisotropic rescaling, i.e., an affinity (c).
Bottom row shows close-up of the images with surface corresponding patches.
normalized image pattern they cover and the feature descriptors derived from them are typically
invariant.
Note, our use of the term ‘region’ simply refers to a set of pixels, i.e. any subset of the image.
This differs from classical segmentation since the region boundaries do not have to correspond to
changes in image appearance such as colour or te xture. All the detectors presented here produce
simply connected regions, but in general this need not be the case.
For viewpoint changes, the transformation of most interest is an affinity. This is illustrated in
figure 1. Clearly, a region with fixed shape (a circular example is shown in figure 1(a) and (b))
cannot cope with the geometric deformations caused by the change in viewpoint. We can observe
that the circle does not cover the same image content, i.e., the same physical surface. Instead, the
shape of the region has to be adaptive, or covariant with respect to affinities (figure 1(c) – close-
ups shown in figure 1(d)–(f)). Indeed, an affinity is sufficient to locally model image distortions
arising from viewpoint changes, provided that (1) the scene surface c an be locally approximated
by a plane or in case of a rotating camera, and (2) perspective effects are ignored, which are
typically small on a local scale anyway. Aside from the geometric deformations, also photometric
deformations need to be taken into account. These can be modeled by a linear transformation
of the intensities.
To further illustrate these issues, and how affine covariant regions can be exploited to cope
with the geometric and photometric deformation between wide baseline images, consider the
example shown in figure 2. Unlike the example of figure 1 (where a circular region was chosen for
one viewpoint) the elliptical image regions here are detected independently in each viewpoint. As
is evident, the pre-image of these affine covariant regions correspond to the same surface region.
Given such an affine covariant region, it is then p os sible to normalize against the geometric and
photometric deformations (shown in figure 2(d)(e)) and to obtain a viewpoint and illumination
2
(a) (b) (c) (d) (e)
Figure 2: Affine covariant regions offer a solution to viewpoint and illumination
changes. First row: one viewpoint; second row: other viewpoint. (a) Original images, (b)
detected affine covariant regions, (c) close-up of the detected regions. (d) Geometric normal-
ization to circles. The regions are the same up to rotation. (e) Photometric and geometric
normalization. The slight residual difference in rotation is due to an estimation error.
invariant description of the intensity pattern within the region.
In a typical matching application, the regions are used as follows. First, a set of covariant
regions is detected in an image. Often a large number, perhaps hundreds or thousands, of
possibly overlapping regions are obtained. A vector descriptor is then associated with each
region, computed from the intensity pattern within the region. This descriptor is chosen to be
invariant to viewpoint changes and, to some extent, illumination changes, and to discriminate
between the regions. Correspondences may then be established with another image of the same
scene, by first detecting and representing regions (independently) in the new image; and then
matching the regions based on their descriptors. By design the regions commute with viewpoint
change, so by design, corresponding regions in the two images will have s imilar (ideally identical)
vector descriptors. The benefits are that correspondences can then be easily established and,
since there are multiple regions, the method is robust to partial occlusions.
This paper gives a snapshot of the state of the art in affine covariant region detection. We will
describe and compare six methods of detecting these regions on images. These detectors have been
designed and implemented by a number of researchers and the comparison is carried out using
binaries supplied by the authors. The detectors are: (i) the ‘Harris-Affine’ dete ctor [24, 27, 34];
(ii) the ‘Hessian-Affine’ detector [24, 27]; (iii) the ‘maximally stable extremal region’ detector (or
MSER, for short) [21, 22]; (iv) an edge-based region detector [45, 48] (referred to as EBR); (v) an
intensity extrema-based region detector [47, 48] (referred to as IBR); and (vi) an entropy-based
region detector [12] (referred to as salient regions).
To limit the scope of the paper we have not included methods for detecting regions which
are covariant only to similarity transformations (i.e., in particular sc ale), such as [18, 19, 23,
26], or other methods of computing affine invariant descriptors, such as image lines connecting
interest points [20, 42, 43] or invariant vertical line segments [9]. Also the detectors proposed
by Lindeberg [16] and Baumberg [1] have not been included, as they come very close to the
Harris-Affine and Hessian-Affine detectors.
The six detectors are described in section 2. They are compared on the data set shown
in figure 9. This data set includes structured and textured scenes as well as different types
of transformations: viewpoint changes, scale changes, illumination changes, blur and JPEG
3
compression. It is described in more detail in section 3. Two types of comparisons are carried
out. First, in section 4, the repeatability of the detector is measured: how well does the detector
determine corresponding sc ene regions? This is measured by comparing the overlap between
the ground truth and detected regions, in a manner similar to the evaluation test used in [24],
but with special attention paid to the effect of the different scales (region sizes) of the various
detectors’ output. Here, we also measure the accuracy of the regions shape, scale and localization.
Second, the distinctiveness of the detected regions is assessed: how distinguishable are the regions
detected? Following [25, 28], we use the SIFT descriptor developed by Lowe [18], which is an 128-
dimensional vector, to describe the intensity pattern within the image regions. This descriptor
has been demonstrated to be superior to others used in literature on a number of measures [25].
Our intention is that the image s and tests described here will be a benchmark against which
future affine covariant region detectors can be assessed. The images, Matlab code to carry out
the performance tests, and binaries of the detectors are available from
http://www.robots.ox.ac.uk/
˜
vgg/research/affine.
2 Affine covariant detectors
In this section we give a brief description of the six region detectors used in the comparison.
Section 2.1 describes the related methods Harris-Affine and Hessian-Affine. Sections 2.2 and 2.3
describe methods for detecting edge-based regions and intensity extrema-based regions. Finally,
sections 2.4 and 2.5 describe MSER and salient regions.
For the purpose of the comparisons the output region of all detector types are represented by
a common shape, which is an ellipse. Figures 3 and 4 show the ellipses for all detectors on one
pair of images. In order not to overload the images, only some of the corresponding regions that
were actually detected in both images have been shown. This selection is obtained by increasing
the threshold.
In fact, for most of the detectors the output shape is an ellipse. Howeve r, for two of the
detectors (edge-based regions and MSER) it is not, and information is lost by this representation,
as ellipses can only be matched up to a rotational degree of freedom. Examples of the original
regions detected by these two methods are given in figure 5. These are parallelogram-shaped
regions for the edge-based region detector, and arbitrarily shap ed regions for the MSER detector.
In the following the representing ellipse is chosen to have the same first and second moments as
the originally detected region, which is an affine covariant construction method.
2.1 Detectors based on affine normalization – Harris-Affine & Hessian-
Affine
We describe here two related methods which detect interest points in scale-space, and then
determine an elliptical region for each point. Interest points are either detected with the Harris
detector or with a detector based on the Hessian matrix. In both cases scale-selection is based
on the Laplacian, and the shape of the elliptical region is determined with the second moment
matrix of the intensity gradient [1, 16].
The second moment matrix, also called the auto-correlation matrix, is often used for feature
detection or for describing local image structures. Here it is used both in the Harris detector
and the elliptical shape estimation. This matrix describes the gradient distribution in a local
neighbourhood of a point:
M = µ(x, σ
I
, σ
D
) =
µ
11
µ
12
µ
21
µ
22
= σ
2
D
g(σ
I
) ∗
I
2
x
(x, σ
D
) I
x
I
y
(x, σ
D
)
I
x
I
y
(x, σ
D
) I
2
y
(x, σ
D
)
(1)
The local image derivatives are computed with Gaussian kernels of scale σ
D
(differentiation
scale). The derivatives are then averaged in the neighbourhood of the point by smoothing with
a Gaussian window of scale σ
I
(integration scale). The eigenvalues of this matrix represent two
4
(a) Harris-Affine
(b) Hessian-Affine
(c) MSER
Figure 3: Regions generated by different detectors on corresponding sub-parts of the
first and third graffiti images of figure 9(a). The ellipses show the original detection size.
5