A Multi-Scale Bilateral Structure Tensor
Based Corner Detector
Lin Zhang, Lei Zhang
1
and David Zhang
Biometrics Research Center, Department of Computing
The Hong Kong Polytechnic University
Hong Kong, China
{cslinzhang, cslzhang, csdzhang}@comp.polyu.edu.hk
Abstract. In this paper, a novel multi-scale nonlinear structure tensor based
corner detection algorithm is proposed to improve effectively the classical Har-
ris corner detector. By considering both the spatial and gradient distances of
neighboring pixels, a nonlinear bilateral structure tensor is constructed to ex-
amine the image local pattern. It can be seen that the linear structure tensor
used in the original Harris corner detector is a special case of the proposed bila-
teral one by considering only the spatial distance. Moreover, a multi-scale fil-
tering scheme is developed to tell the trivial structures from true corners based
on their different characteristics in multiple scales. The comparison between
the proposed approach and four representative and state-of-the-art corner detec-
tors shows that our method has much better performance in terms of both detec-
tion rate and localization accuracy.
Keywords: Harris, corner detector, bilateral structure tensor
1 Introduction
Corner detection is a critical task in various machine vision and image processing
systems because corners play an important role in describing object unique features
for recognition and identification. Applications that rely on corners include motion
tracking, object recognition, 3D object modeling, and stereo matching, etc.
Considerable research has been carried out on corner detection. One of the earliest
successful corner detectors can be Harris corner detector [1]. Harris et al. [1] calcu-
lated the first-order derivatives of the image along horizontal and vertical directions,
with which a 22 structure tensor was formed. The corner detection was accom-
plished by analyzing the eigenvalues of the structure tensor at each pixel. However,
computing derivatives is sensitive to noise, and the Harris corner detector has poor
localization performance because it needs to smooth the derivatives for noise reduc-
tion. Thus, several methods [2-3] have been proposed to improve its performance.
1
Corresponding author. Email: cslzhang@comp.polyu.edu.hk. Tel: 852-27667355.
Apart from Harris corner detector and its variants, many other corner detectors
have also been proposed by researchers. Kitchen and Rosenfeld [4] proposed a cor-
nerness measure based on the change of gradient direction along an edge contour
multiplied by the local gradient magnitude. Smith and Brady [5] proposed the
SUSAN scheme. In SUSAN, a circular mask is taken around the examined pixel and
this pixel is considered as the nucleus of the mask. Then “USAN” (Univalue Segment
Assimilating Nucleus) is defined as an area of the mask which has the similar bright-
ness as the nucleus. Smith et al. [5] assumed that the USAN would reach a minimum
when the nucleus lies on a corner point. Wang and Brady [6] proposed a corner de-
tection algorithm based on the measurement of surface curvature. In [7] and [8],
Mokhtarian et al. proposed two CSS (Curvature Scale Space) based corner detectors.
In these two algorithms, edge contours are first extracted and then corners are de-
tected as the positions with high curvatures on edge contours. In [9], Zheng et al.’s
cornerness measure was simply the gradient module of the image gradient direction.
This paper presents a novel effective evolution of the classical Harris corner detec-
tor. In the original Harris corner detector, an isotropic Gaussian kernel is used to
smooth each of the four elements in the 22 structure tensor over a local window
before calculating the eigenvalues. Such a smoothing operation will have two disad-
vantages. First, some weak corners will be smoothed out. Second, the localization
accuracy is much degraded. Inspired by the success of bilateral filters [10] in image
denoising, which consider both the spatial and the intensity similarities in averaging
neighboring pixels for noise removal, in this paper we construct a nonlinear bilateral
structure tensor and use it to detect corner points.
The basic idea of the proposed method lies in that both the spatial and gradient dis-
tances should be involved in smoothing the structure tensor elements. The neighbor-
ing pixels that have shorter spatial and gradient distances to the given one should
have higher weights in the averaging. In this way, a nonlinear structure tensor, which
is adaptive to image local structures, could be constructed and hence the image local
pattern could be better distinguished. It can be seen that the classical Harris corner
detector is a special case of the proposed method by exploiting only the spatial dis-
tance in the structure tensor smoothing. However, the proposed nonlinear structure
tensor has much higher sensitivity to corner-like fine structures than the linear struc-
ture tensor. Therefore, it may respond strongly to some trivial feature points in the
image. In order to get rid of the possible false corners detected at fine image scales,
we propose a multi-scale filtering scheme based on the different characteristics of true
corners and trivial structures in multiple scales.
The rest of the paper is organized as follows. Section 2 briefly reviews the Harris
corner detector. Section 3 presents the new corner detector in detail. Experimental
results are presented in section 4 and the conclusion is made in section 5.
2 Harris Corner Detector
Harris corner detector [1] has been very widely used in machine vision applications.
Consider a 2D gray-scale image I. Denote by W∈I an image patch centered on (x
0
,
y
0
). The sum of square differences between W and a shifted window W
(
△
x,
△
y)
is calcu-
lated as
2
(,)
(( , ) ( , ))
ii
ii i i
xy W
SIxyIxxyy
(1)
By approximating the shifted patch using a Taylor expansion truncated to the first
order terms, we have:
,
x
SxyA
y
(2)
where
2
(,) (,)
2
(,) (,)
()
()
ii ii
ii ii
hhv
iii
xy W xy W
vh v
ii i
xy W xy W
A
and
h
i
and
v
i
represent the first order partial
derivatives of image I along horizontal and vertical directions at pixel (x
i
, y
i
).
In practice matrix A is computed by averaging the tensor product
II
( I
denotes the gradient image of I) over the window W with a weighting function
K
,
i.e.
2
(,) (,)
2
(,) (,)
()( ) ()
() ()( )
ii ii
ii ii
hhv
iii
xy W xy W
vh v
ii i
xy W xy W
Ki Ki
A
Ki Ki
(3)
Usually
K
is set as a Gaussian function
2
2
1
() exp
2
2
i
d
Ki
, where
222
00
()( )
ii i
dxx yy and ρ is the standard deviation of the Gaussian kernel.
A
ρ
is symmetric and positive semi-definite. Its main modes of variation correspond
to the partial derivatives in orthogonal directions and they are reflected by the eigen-
values λ
1
and λ
2
of A
ρ
. The two eigenvalues can form a rotation-invariant description
of the local pattern. Under the situation of corner detection, three distinct cases are
considered. 1) Both the eigenvalues are small. This means that the local area is flat
around the examined pixel. 2) One eigenvalue is large and the other one is small. The
local neighborhood is ridge-shaped. 3) Both the eigenvalues are rather large. This
indicates that a small shift in any direction can cause significant change of the image
at the examined pixel. Thus a corner is detected at this pixel.
Harris suggested that the exact eigenvalue computation can be avoided by calcu-
lating the response function
2
() () ()RA detA k trace A
(4)
where det(A
ρ
) is the determinant of A
ρ
, trace(A
ρ
) is the trace of A
ρ
, and k is a tunable
parameter.
3 Bilateral Structure Tensor Based Corner Detection
This section presents the proposed multi-scale nonlinear bilateral structure tensor
based corner detector in detail. Our algorithm differs from the original Harris corner
detector mainly in two aspects. First, a nonlinear structure tensor is constructed to
substitute for the linear one used in the Harris corner detector; second, a multi-scale
filtering scheme is proposed to filter out the false and trivial corners detected at small
scales.
3.1 Construction of the Bilateral Structure Tensor
The structure tensor for a gray level image I is a 22 symmetric matrix that contains
in each element the orientation and intensity information in a local area. Denote by
I the gradient image of I. The initial matrix field can be computed as the tensor
product
0
J
II
. To incorporate the neighboring structural information into the
given position, an averaging kernel could be used to smooth each element of J
0
.
Usually a Gaussian kernel K
ρ
with standard deviation
is employed for this purpose:
0
*
J
KJ
(5)
where symbol “*” means convolution. Since convolution is a linear operator, the
structure tensor J
ρ
is referred to as linear structure tensor [11]. It is a symmetric, posi-
tive semi-definite matrix. Comparing Eq. (3) with Eq. (5), we see that the matrix A
ρ
in
Harris corner detector is actually the linear structure tensor J
ρ
at pixel (x
0
, y
0
).
In Harris corner detector [1], the “cornerness” of a pixel (x,y) is totally determined
by its local structure tensor J
ρ
(x,y). However, the smoothing kernel K
ρ
has two prob-
lems. First, the isotropic smoothing operation will smooth some weak corner features
out so that the detection capability is decreased. Second, the localization accuracy of
detected corner points will be reduced, which is a well-known problem of the Harris
corner detector. Intuitively, if the local structure tensor can better preserve the local
structural information at (x,y), the cornerness measured from it should be more relia-
ble and accurate.
Fig. 1: Weight distributions in a neighborhood of a corner pixel. (a) An artificial image with
an ideal corner (red circle); (b) weights distribution by using the Gaussian kernel
K
ρ
; (c)
weights distribution by using the proposed bilateral weighting function
N
ρ,σ
.
As an early denoising technique, Gaussian smoothing is simple but it will over-
blur the image details. The Gaussian weighting kernel only uses the notation of spa-
tial location in the weights assignment. The greater the spatial distance from a neigh-
boring pixel to the central pixel, the smaller the averaging weight will be assigned.
The intensity similarity between the pixels is not exploited in Gaussian smoothing. In
[10], the bilateral filter was proposed, which employs both the spatial and intensity
similarities between pixels in averaging weight design. It has been shown that bilater-
al filtering could significantly improve the edge structure preservation while remov-
ing noise [10].
Inspired by the success of bilateral filters in image denoising, in this paper we
construct a bilateral structure tensor for better corner detection performance. There
are two basic factors in the formation of a local pattern: the relative positions between
neighboring pixels and the intensity variations between them. Therefore, in the
smoothing of J
0
, we should consider both the spatial distance and the gradient dis-
tance in the averaging weight assignment. In the original Harris corner detector, only
the spatial distance is considered by applying a Gaussian smoothing kernel K
ρ
to
II
. In this paper, we will also involve the gradient distance in the smoothing of
II
.
Here, the gradient distance from the position (x
i
, y
i
) to the central position (x
0
, y
0
)
is defined as:
22
00
ghhvv
ii i
d
(6)
The spatial distance from (x
i
, y
i
) to (x
0
, y
0
) is the same as in the original Harris corner
detector:
22
00
s
ii i
dxxyy
(7)
By considering both the spatial and gradient distances into the assignment of averag-
ing weight, we define the following bilateral weighting function for each pixel (x
i
, y
i
)
∈W:
22
,
22
,
1() ()
() exp exp
22
sg
ii
dd
Ni
C
(8)
where
and
are the parameters to control the decaying speeds over spatial and
gradient distances, and
22
,
22
() ()
exp exp
22
sg
ii
W
dd
C
(9)
is the normalization factor.
Fig. 1 shows an example to illustrate the weight distributions by using the Gaus-
sian kernel K
ρ
and the proposed function N
ρ,σ
. Fig. 1-a is an artificial image with an
ideal corner in the center, which is marked by a red circle. The size of local window
W for smoothing is set as 2121. Figs. 1-b and 1-c illustrate the weight distributions
for the pixels within W by using the Gaussian kernel K
ρ
and the proposed bilateral
weighting function N
ρ,σ
, respectively. It is clearly seen that K
ρ
is isotropic and is inde-
pendent of the image local structure, while N
ρ,σ
is anisotropic and is adaptive to the
image local pattern. In this example, the edge pixels have higher weights than the
non-edge pixels because they are more similar to the examined corner pixel in terms
of gradient. Meanwhile, for the pixels lying on the same edge, the ones near to the
corner pixel have higher weights than the others because they have shorter spatial
distances to the corner point.
With the nonlinear bilateral weighting function N
ρ,σ
, the nonlinear bilateral struc-
ture tensor is defined as:
