University of Groningen
Distance sets for shape filters and shape recognition
Grigorescu, Cosmin; Petkov, Nicolai
Published in:
Ieee transactions on image processing
DOI:
10.1109/TIP.2003.816010
IMPORTANT NOTE: You are advised to consult the publisher's version (publisher's PDF) if you wish to cite from
it. Please check the document version below.
Document Version
Publisher's PDF, also known as Version of record
Publication date:
2003
Link to publication in University of Groningen/UMCG research database
Citation for published version (APA):
Grigorescu, C., & Petkov, N. (2003). Distance sets for shape filters and shape recognition.
Ieee
transactions on image processing
,
12
(10), 1274-1286. https://doi.org/10.1109/TIP.2003.816010
Copyright
Other than for strictly personal use, it is not permitted to download or to forward/distribute the text or part of it without the consent of the
author(s) and/or copyright holder(s), unless the work is under an open content license (like Creative Commons).
The publication may also be distributed here under the terms of Article 25fa of the Dutch Copyright Act, indicated by the “Taverne” license.
More information can be found on the University of Groningen website: https://www.rug.nl/library/open-access/self-archiving-pure/taverne-
amendment.
Take-down policy
If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately
and investigate your claim.
Downloaded from the University of Groningen/UMCG research database (Pure): http://www.rug.nl/research/portal. For technical reasons the
number of authors shown on this cover page is limited to 10 maximum.
Download date: 10-08-2022
1274 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 12, NO. 10, OCTOBER 2003
Distance Sets for Shape Filters
and Shape Recognition
Cosmin Grigorescu, Student Member, IEEE, and Nicolai Petkov
Abstract—We introduceanovelrich local descriptor of an image
point, we call the (labeled) distance set, which is determined by the
spatial arrangement of image features around that point. We de-
scribe a two-dimensional (2-D) visual object by the set of (labeled)
distance sets associated with the featurepoints of that object. Based
on a dissimilarity measure between (labeled) distance sets and a
dissimilarity measure between sets of (labeled) distance sets, we
address two problems that are often encountered in object recog-
nition: object segmentation, for which we formulate a distance sets
shape filter, and shape matching. The use of the shape filter is il-
lustrated on printed and handwritten character recognition and
detection of traffic signs in complex scenes. The shape compar-
ison procedure is illustrated on handwritten character classifica-
tion, COIL-20 database object recognition and MPEG-7 silhouette
database retrieval.
Index Terms—Character recognition, distance set, image data-
base retrieval, MPEG-7, object recognition, segmentation, shape
descriptor, shape filter, traffic sign recognition.
I. INTRODUCTION
T
HE world, as we visually perceive it, is full of information
that we effortlessly interpret as colors, lines, edges, con-
tours, textures, etc. It is not only the mere presence of these bits
and pieces of information that plays a role in our perception, but
also their spatial interrelations which enable us to distinguish
between surrounding objects. Let us consider, for example, the
binary images shown in Fig. 1(a). The black pixels taken indi-
vidually, especially those in the interior of the objects shown, do
not carry any information whether they are part of a representa-
tion of a bird or a rabbit; it is their spatial interrelations which
make us recognize the shape of a bird or a rabbit. When only
the contours or even parts of the contours of the objects are left,
like in Fig. 1(b) and (c), we are still able to discriminate between
the two objects. In this particular case, it is the edge points be-
longing to the contours which are perceptually important and it
is their spatial interrelations which define our perception of two
different objects. A failure of the structures of the brain respon-
sible for the detection of elementary features, such as lines and
edges, or of the higher structures responsible forsetting relations
between these features and integrating them into perception of
specific objects leads to a medical condition referred to by neu-
rologists as form blindness or object agnosia [1]. While many
aspects of detection of elementary visual features by the brain
Manuscript received August 14, 2002; revised March 17, 2003. The associate
editor coordinating the review of this manuscript and approving it for publica-
tion was Dr. Robert D. Nowak.
The authors are with the Institute of Mathematics and Computing Science,
University of Groningen, 9700 AV Groningen, The Netherlands (e-mail:
cosmin@cs.rug.nl; petkov@cs.rug.nl).
Digital Object Identifier 10.1109/TIP.2003.816010
Fig. 1. (a) Two familiar objects, (b) their contours, and (c) parts of their
contours.
are fairly well understood [2]–[11], and a large number of algo-
rithms for feature extraction have been developed in computer
vision (see, e.g., [12]–[16]), the neurophysiology of feature in-
tegration for object perception and recognition is still unknown
and the development of computer vision algorithms for this pur-
pose is an ongoing activity [17]–[21].
In this paper, we propose to describe the spatial interrela-
tions between perceptually significant points, to be called fea-
ture points, by associating with each such point a data structure
containing the set of distances to a certain number of neigh-
boring feature points. As features can be of different types, we
associate different labels with such distances and call the re-
sulting data structure the labeled distance set. We propose the
set of distance sets associated with all feature points of an ob-
ject as a data structure integrating information about the spa-
tial interrelations of elementary features and show that this data
structure can effectively be used to segment objects in complex
scenes and to classify objects.
A number which characterizes the image content in the sur-
roundings of a point is often called a local image descriptor.
The value of a pixel in a grey-level image can be considered
as the simplest type of local descriptor, comprising merely one
number. An elaborate local descriptor can indicate the presence
of a perceptually important feature,often calleda salient feature,
such as an edge [22], a corner, or a specific type of junction [23]
at a given location in the image.
A rich descriptor is a set of numbers or, more generally, a data
structure computed on the neighborhood of a point and presents
a more informative characteristic of the local image contents.
The adjective “rich” refers to the use of many values as op-
posed to one single value, with the degree of richness being
related to the size (cardinality) of the descriptor and its com-
plexity as a data structure. Kruizinga and Petkov [24] used as
a rich local descriptor a set of arrays of intensity values vis-
ible in multiscale windows centered on the concerned point.
Wiskott et al. [25] considered as a local descriptor a vector of
complex Gabor wavelet coefficientscomputed for several scales
and orientations. Belongie et al. [20] proposed as a local de-
scriptor of a point the two-dimensional (2-D) histogram (with
arguments log-distance and polar angle in relative image coor-
dinates) of the number of object contour points in the surround-
ings of the concerned point. Amit et al. [26] used a set of tags,
1057-7149/03$17.00 © 2003 IEEE
GRIGORESCU AND PETKOV: DISTANCE SETS FOR SHAPE FILTERS AND SHAPE RECOGNITION 1275
each tag specifying the occurrence of a given combination of
binary pixel values at certain positions relative to the concerned
point. The (labeled) distance set we propose below comprises
multiple values and thus also fallsin the class of rich local image
descriptors.
Local descriptors are used for different purposes; one can,
for instance, select only those points of an image whose local
descriptors fulfill a certain condition. For grey-level images,
thresholding can be considered as such an operation; the condi-
tion to be fulfilled by the local descriptor (in this case the grey-
level value of a pixel) is that it is larger than a given threshold
value. This type of use of local descriptors can be regarded as
filtering: a result image that represents a binary map of points
which satisfy a given condition versus points that do not fulfill
the condition is computed from an input image.
In the following we define a dissimilarity measure between
(labeled) distance sets and, based on this measure, we intro-
duce a novel filtering procedure aimed at detecting instances
of a given object (or parts of an object) in a complex scene. We
refer to the proposed filtering method as distance set shape fil-
tering.
Another use of local descriptors, typical of rich local descrip-
tors, is for solving the correspondence problem, i.e., finding
counterpart points in two images. We address this problem by
representing an object as a set of (labeled) distance sets and by
computing a dissimilarity measure between two such sets. We
use this measure for pairwise shape comparison and classifica-
tion.
Image descriptors based on distances between image features
have been previously proposed in the literature for object detec-
tion [22], [27], shape comparison [20], [28], handwritten digit
classification [26], face recognition [29]. Despite this, the con-
cept of a (labeled) distance set, and the associated filtering oper-
ator and shape comparison method are, to our best knowledge,
novel.
A brief overview of other methods for shape extraction and
shape comparison methods based on rich local descriptors is
given in Section II. In Section III we introduce the concepts of
a distance set and a labeled distance set, and associated dissim-
ilarity measures. Sets of (labeled) distance sets together with a
measure of dissimilarity between two sets of (labeled) distance
sets are introduced in Section IV. The distance sets shape filter
is introduced in Section V; this filter is related to certain mor-
phological filters. We illustrate the applicability of the distance
sets shape filtering to handwritten character recognition and the
detection of instances of a given object in complex scenes.
A shape comparison procedure based on a dissimilarity
measure between two sets of (labeled) distance sets is presented
in Section VI. We evaluate the performance of the proposed
comparison method in three applications: handwritten char-
acter classification, COIL-20 database object recognition, and
MPEG-7 silhouette database retrieval. Section VII summarizes
the results and concludes the paper.
II. O
VERVIEW OF OTHER SHAPE EXTRACTION AND SHAPE
COMPARISON METHODS
We consider shape as a property or characteristic of a set of
points that define a visual object whereby said property is deter-
mined by the geometric relations between the involved points in
such a way that it is invariant for translations, rotations, reflec-
tions and distance scaling of the point set as a whole.
A substantial body of work in shape analysis assumes that
shape is a characteristic of a binary image region. Such methods
use either the boundary of an object or its interior to compute
a shape descriptor. Their applicability is envisaged mainly in
situations in which a binary object is already available or can
be computed by some preprocessing steps like pixel-based seg-
mentation, edge detection, skeletonization, etc. An overview of
such shape analysis methods can be found in [17] and [30].
Recent developments in shape analysis describe shape more
generally, as a property of a collection of feature points, and
associate with each such point a local image descriptor. These
descriptors are subsequently used to find the occurrences of a
reference object in an image (segmentation), or to evaluate how
similar two objects are (comparison). When comparing two ob-
jects, one often tries to determine a transformation which casts
one of them into another. Since rigid, affine or projective trans-
formations are sensitive to irregular shape deformations or par-
tial occlusion, a more general model, that of nonrigid trans-
formations, is assumed. With this formulation, shape compar-
ison implies either finding pairs of corresponding feature points
(i.e., solving a correspondence problem) and/or determining the
transformation which maps one point set into the other. A shape
(dis)similarity measure can be computed from the solution of
the correspondence problem and/or from the nonrigid transfor-
mation.
As the proposed (labeled) distance set shape descriptor, the
associated shape filtering operator and the shape comparison
procedure we propose are based on sets of feature points, we
restrict our overview only to similar methods—nonrigid point-
based shape analysis methods—previously reported in the liter-
ature.
Non-rigid shape matching methods have been introduced in
computer vision by Fischler and Elschlager [31], who formu-
lated the shape matching as an energy minimization problem
in a mass-spring model. A feature-based correspondence ap-
proach using eigenvectors is presented by Shapiro and Brady
in [32]. Modal matching proposed by Sclaroff and Pentland
[33] describes shape in terms of generalized symmetries defined
by object’s eigenmodes. The shape similarity between two ob-
jects is expressed by the amount of modal energy deformation
needed to align the objects. Chui and Rangarajan [34] use an in-
tegrated approach for solving the point correspondence problem
and finding a nonrigid transformation between points from the
contours of two objects. Their iterative regularization proce-
dure uses a softassign [35], [36] for the correspondences and
a thin-plate spline model [37] for the nonrigid mapping. The
method proves to be robust to noise and outliers.
Other approaches select only a number of points from
the outline of an object and approximate it with curve seg-
ments which pass through those points. Usually, these points
have some specific properties, such as minimum curvature,
inflections between convex and concave parts of an object
contour, etc. Petrakis et al. [38], for instance, extract only
inflection points and approximate the shape by B-splines
curve segments. A dissimilarity cost of associating a group
of curve segments from a source shape to another group of
curve segments originating from a target shape is computed
by dynamic programming. In a similar way, Sebastian et al.
1276 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 12, NO. 10, OCTOBER 2003
[21] use length and curvature to define a similarity metric
between curve segments. This metric is subsequently employed
by a dynamic programming algorithm to solve the optimum
alignment (correspondence) problem. Such methods have a
limited applicability because they can be used for the pairwise
comparison of single curves only. Objects that are defined by
multiple curves, e.g., an icon of a face with separate curves for
head outline, nose, mouth and eyes, cannot be compared since
the mutual geometric arrangement of the constituent curves
that define such an object is not taken into account.
Shape characterization based on distances between points
which do not necessarily originate from the object outline is
used by a number of authors. A shape extraction method based
on the Hausdorff distance between two point sets is proposed
by Huttenlocher et al. [22], [27]. Although this method does
not explicitly find the point correspondences, it is capable of
detecting multiple occurrences of the same reference object
even in case of partial matching caused by occlusion and
clutter. The discrimination is improved by introducing as
additional information the local edge orientation [39]. Amit
and Kong [40] find geometric arrangements of landmark points
in an image by matching decomposable graphs. An extended
approach [26], [41] considers as local descriptors topographic
codes and compares the descriptors originating from different
shapes using multiple classification trees. Gavrila [42] proposes
a method for shape extraction based on the distance transform
of the edge map of an image. The correlation between this
distance transform and the edge map of a reference object is an
indicator whether the reference object is present or not in the
image. Since the distance transform assigns to each image point
as a local descriptor the distance to the nearest edge pixel, this
formulation can be considered as a special case of a distance
set descriptor with cardinality one.
A local descriptor based on the spatial arrangement of feature
points in a fixed-size neighborhood of a point is the shape con-
text of Belongie et al. [20], [28]. The shape context of a point
is a two-dimensional histogram (with axes the log-distance and
polar angle) of the number of contour points in the surround-
ings of the concerned point. These authors developed a method
for shape comparison by first finding the point correspondences
as the solution of an assignment problem and then evaluating
a nonlinear alignment transformation using a thin-plate spline
model. With respect to radial (distance) information, the shape
context may be regarded as a simplified, coarser version of the
distance set descriptor: the coarseness is due to the histogram
binning process. As to the angular information present in the
shape context, such information can be included in a labeled dis-
tance set by assigning to each distance an orientation label—the
polar angle of the concerned feature point. As proposed in [20],
the shape context uses only contour information and is intended
only for shape comparison. In contrast, the labeled distance set
descriptor incorporates multiple feature types. Next to shape
comparison, we use this descriptor also for shape segmentation.
III. D
ISTANCE SETS AND LABELED DISTANCE SETS
A. Distance Sets
Let
be a set of perceptually significant
points in an image, which we will call feature points. The spatial
relation of a given feature point to other feature points can be
characterized by the set of its distances to these other points.
For a given point
one can select only a subset of
nearest neighboring points and still have a good description of
its spatial relation to the other points
1
. Let be the
distance between point
and its -nearest neighbor from ,
. We call the local descriptor
(1)
the distance set, more precisely the
-distance set, of point to
its first
nearest neighbors within . Note that the distance set
of a point is not affected by a rotation of the image as a whole.
Given two points
and from two images and
their associated distance sets
and ,
, we define the relative difference between the
-neighbor and -neighbor distances of and , respectively,
as
2
(2)
Let
be a one-to-one mapping from the set
to the set , such that , ,
and let
be the set of all such mappings.
We introduce the dissimilarity between two distance sets
and as follows:
(3)
The dissimilarity is thus the cost of the optimal mapping of the
distance set
onto the distance set .
Iff
, the distance set is a
subset of
:
(4)
In the special case
, the two sets are identical:
(5)
In this way, the quantity
indicates how dis-
similar two points
and are with respect to their distances to
feature points in their respective neighborhoods.
The distance set
of a point , together with the
dissimilarity measure defined above, can be an effective means
for discrimination of points according to their similarity to a
given point. As an example, we consider as feature points the
pixels which define printed characters on a computer screen.
Fig. 2(a) shows a set of points
that define the character “ ,”
a point
from this set and its associated distance set
to the first neighbors. (Euclidian distance
is used in this and the followingexamples unless otherwise spec-
ified.) Fig. 2(b) shows points from the word “alphabet,” that
comprise a set
. For each point we compute its cor-
responding distance set
. In Fig. 2(b), the points for
1
The point
p
need actually not be a point from
S
, it can be any point in the
image.
2
The use of relative differencesis motivated by Weber-Fechner law of percep-
tion. Alternative definitions with similar effect are possible, e.g.,
j
ln
d
(
p
)
0
ln
d
(
q
)
j
, but we consider this as a technical detail.
GRIGORESCU AND PETKOV: DISTANCE SETS FOR SHAPE FILTERS AND SHAPE RECOGNITION 1277
Fig. 2. (a) A point
p
2
S
(shed black) from theprinted character “
a
” together
with its set
DS
(
p
)
of distances to the first
N
=5
nearest neighbors
within
S
. (b) The points
q
2
S
(shed also black) in the word “alphabet” which
have 5-distance sets (within
S
) identical with that of
p
,
D
(
p; q
)=0
.
(c) Points
q
2
S
for which
D
(
p; q
)=0
,the same points are obtained
also for
D
(
p; q
)=0
.
which holds are shown black. From the
total of 173 points of the word “alphabet,” only 20 have 5-dis-
tance sets identicalto that of the concerned point ofthe character
“
.” The discrimination is improvedby increasing the size of the
involveddistance sets, Fig. 2(c): if, for instance, the distancesets
contain
distances, only the real counterparts of
the selected point are found to have the same distance sets.
In a second example, we consider a point
, where
is the set of points comprising the contour of a handwritten
character “
,” and the distance set associated with
it, Fig. 3(a). The set
consists of the points from the con-
tour of a handwritten word “alphabet.” For each
we
compute the associated distance set
and its dis-
similarity
to the distance set of
the concerned point
from the handwritten character “ .” The
requirement
, which was used in the
printed character examplegivenabove,turns out to be toostrong
and not fulfilled by any point
. A weaker requirement
, , can be imposed on the points of
.For , only 35 from the total number of 216 points of
the word “alphabet” fulfill this requirement; the corresponding
points are shed black in Fig. 3(b). If an even stricter condition
is imposed,
, only 11 points are found to satisfy the
condition, Fig. 3(c).
B. Labeled Distance Sets
The feature points in an image can be of different types. In
the character “
,” for instance, one can consider the points at
the extremities of the contour to be of an “end-of-line” type
and all other points to be of another, “line” or “contour” type.
As a matter of fact, similar, evidently perceptually significant
features are extracted in the visual cortex by so-called simple
and complex cells (for lines and edges) [2], [4]–[6], and end-
stopped cells (for ends of lines) [3], [4], [7]–[9]. The distance
Fig. 3. (a) A point
p
2
S
(shed black) from the handwritten character
“
a
” together with its distance set
DS
(
p
)
to the first
N
=15
nearest
neighbors within
S
. (b-c) The points
q
2
S
in the handwritten word
“alphabet” (shed also black) for which holds (b)
D
(
p;q
)
<
0
:
3
and (c)
D
(
p; q
)
<
0
:
25
.
from a feature point to another feature point can thus be labeled
by the type of feature to which the distance is measured.
Let
be the set of possible feature labels and let be one
such label. We define the labeled distance subset
of a point to its first neighbor feature points of type as
follows:
(6)
where
is the set of feature points of type and is the
distance from point
to its -th nearest neighbor feature point
from that set.
A labeled distance set is the set of tuples of labeled distance
subsets and their corresponding labels:
(7)
Let
be the dissimilarity between two la-
beled distance subsets of type
computed according to (3).
Since (in a given context) certain feature points can be perceptu-
ally more important than others, foreach label type
we can
assign a different weight
and define the dissimilarity be-
tween two labeled distance sets
and associated with and as
follows:
(8)
