EXPLOITING DISCRIMINANT INFORMATION IN ELASTIC GRAPH MATCHING
Stefanos Zafeiriou , Anastasios Tefas and Ioannis Pitas
Dept. of Informatics, Aristotle University of Thessaloniki, Box 451, 54124 Thessaloniki, Greece
e-mail: {dralbert,tefas,pitas}@zeus.csd.auth.gr
ABSTRACT
In this paper, we investigate the use of discriminant tech-
niques in the elastic graph matching (EGM) algorithm. First
we use discriminant analysis in the feature vectors of the
nodes in order to find the most discriminant features. The
similarity measure for discriminant feature vectors and the
node deformation are combined in a discriminant manner
in order to form a local similarity measure between nodes.
Moreover, the local similarity values at the nodes of the
elastic graph, are weighted by coefficients that are also de-
rived by some discriminant analysis in order to form a to-
tal similarity measure between faces. We illustrate the im-
provements in performance in frontal face verification using
a modified multiscale morphological analysis.
1. INTRODUCTION
A popular class of techniques used for frontal face recogni-
tion/verification is EGM [1]. In EGM the reference object
graph is created by projecting the object’s image onto a rect-
angular elastic sparse graph where a Gabor wavelet bank
response is measured at each node. The graph matching
procedure is implemented by a coarse-to-fine stochastic op-
timization of a cost function which takes into account both
jet similarities and node deformation [1].
A variant of the standard EGM, the so-called morpho-
logical elastic graph matching (MEGM), has been proposed
for frontal face verification [2]. In MEGM the Gabor analy-
sis has been superseded by multiscale morphological dilation-
erosion by a scaled structuring function [2].
Discriminant techniques have been employed in order
to enhance the recognition and verification performance of
the EGM. The use of linear discriminant techniques at the
feature vectors for selecting the most discriminant features
has been proposed in [1, 2]. Several schemes that aim at
weighting the graph nodes according to their discriminatory
power have been proposed [2, 3]. In [3] it has been shown
This work is funded by the integrated project BioSec IST-2002-
001766 (Biometric Security, http://www.biosec.org), under Information
Society Technologies (IST) priority of the 6th Framework Programme of
the European Community.
that the verification performance of the EGM can be highly
improved by proper node weighting strategies.
In this paper we illustrate where and how discriminant
techniques can be employed in the EGM. More precisely,
each node is considered as a local expert and discriminant
feature selection techniques are employed for enhancing its
recognition/verification performance. The deformation of
each node is considered as a second local similarity met-
ric that can quantify the relationships with its neighboring
nodes. The new local similarity value at each node is pro-
duced by discriminant weighting of both the feature vector
similarity measure and the node deformation. As a final
discriminant step the local similarity measures at grid nodes
are weighted by coefficients according to their discriminant
power. The problem of frontal face verification is used in
the following of the paper in order to describe in detail the
different discriminant steps.
2. ELASTIC GRAPH MATCHING
In this Section we will briefly outline the problem of frontal
face verification and the framework under which EGM per-
forms face verification. Let U be a facial image database
and each facial image u ∈ U belongs to one of the C per-
son classes {U
1
, U
2
, . . . , U
C
} with U =
S
C
i=1
U
i
. For a
face verification system that uses the database U a genuine
(or client) claim is performed when a person t provides its
facial image, u, claiming that u ∈ U
r
and t = r. When
a person t provides its facial image u while claiming that
u ∈ U
r
, with t 6= r, an impostor claim occurs. The scope of
a face verification system is to handle properly these claims
by accepting the genuine claims and rejecting the impostor
ones.
The first step of EGM is to analyze the facial image re-
gion of the image u. Then, a set of local descriptors is ex-
tracted at each graph node. In the standard EGM a 2D Ga-
bor based filter bank has been used for image analysis. The
output of multiscale morphological dilation-erosion opera-
tions is a nonlinear alternative of the Gabor filters for multi-
scale analysis and has been successfully used for facial im-
age analysis [2]. At each graph node that is located at image
coordinates x a jet j(x) is formed as:
j(x) = (f
1
(x), . . . , f
S
(x)), (1)
where f
i
(x) denotes the output of a local operator applied
to the image f at the ith scale or at the ith pair (scale, orien-
tation) and S is the dimensionality of the jet.
The next step of the EGM is to translate and deform the
reference graph on the test image in order to find the corre-
spondences of the reference graph nodes on the test image.
This is accomplished by minimizing a cost function that em-
ploys node jet similarities and in the same time preserves
the node relationships. Let the superscripts t and r denote
a test and a reference person (or graph), respectively. The
L
2
norm between the feature vectors at the l-th graph node
of the reference and the test graph is used as a similarity
measure between jets, i.e.:
C
f
(j(x
l
t
), j(x
l
r
)) = ||j(x
l
r
) − j(x
l
t
)||. (2)
Let V be the set of graph vertices. Let also H(l) be the
four-connected neighborhood of node l. In order to quan-
tify the node neighborhood relationships using a metric, the
local node deformation is used:
C
d
(x
l
t
, x
l
r
) =
X
ξ∈H(l)
||(x
l
t
− x
l
r
) − (x
ξ
t
− x
ξ
r
)||, ξ ∈ H(l).
(3)
The objective is to find a set of vertices {x
l
t
(r), l ∈ V}
in the test image that minimize the cost function:
C({x
l
t
(r)}) =
X
l∈V
{C
f
(j(x
l
t
), j(x
l
r
)) +λC
d
(x
l
t
, x
l
r
)}. (4)
The jet of the l-th node that has been produced after the
matching procedure of the graph of the reference person r
in the image of the test person t is denoted as j(x
l
t
(r)). The
optimization of (4) has been interpreted in [2] as a simulated
annealing with additional penalties imposed by the graph
deformations. Accordingly, (4) can be simplified to:
D
t
(r) =
P
l∈V
{C
f
(j(x
l
t
), j(x
l
r
))} subject to
x
l
t
= x
l
r
+ s + δ
l
, ||δ
l
|| ≤ δ
max
,
(5)
where s is a global translation of the graph and δ
l
denotes a
local perturbation of the graph nodes. The choices of δ
max
in (5) and of λ in (4) control the rigidity/plasticity of the
graph [1],[2]. Obviously, both functions (4) and (5) define a
similarity measure between two faces.
3. FEATURE VECTOR DISCRIMINANT ANALYSIS
It is obvious that the standard EGM treats uniformly all the
different features that form the jets. Thus, it sounds rea-
sonable to use discriminant techniques in order to find the
most discriminant features. In other words, we should learn
a person and node specific discriminant function g
l
r
, for the
l-th node of the reference person r, that transforms the jets
j(x
l
t
(r)):
´
j(x
l
t
(r)) = g
l
r
(j(x
l
t
(r)). (6)
We will use linear techniques for finding the transform
g
l
r
but non-linear techniques can be also used. Before cal-
culating the linear projections we normalize all the jets that
have been produced during the match of the graphs of the
reference person r to all other facial images in the train-
ing set in order to have zero mean and unit magnitude. Let
b
j(x
l
t
(r)) be the normalized jet at l-th node. Let F
l
C
(r) and
F
l
I
(r) be the sets of the normalized jets of the l-th node that
correspond to genuine claims and impostor claims related
to person r, respectively.
We use the same criterion as [1],[2] that can give more
than one discriminant directions. Let W
l
(r) and B
l
(r) be
the matrices:
W
l
(r) =
X
b
j(x
l
t
(r))∈F
l
I
(r)
(
b
j(x
l
t
(r))−m(F
l
C
(r))(
b
j(x
l
t
(r))−m(F
l
C
(r))
T
(7)
and
B
l
(r) =
X
b
j(x
l
t
(r))∈F
l
C
(r)
(
b
j(x
l
t
(r))−m(F
l
C
(r))(
b
j(x
l
t
(r))−m(F
l
C
(r))
T
.
(8)
The optimal discriminative directions
´
Ψ
l
(r) are given by
maximizing the criterion:
J(Ψ
l
(r)) =
tr[Ψ
l
(r)
T
W
l
(r)Ψ
l
(r)]
tr[Ψ
l
(r)
T
B
l
(r)Ψ
l
(r)]
(9)
where tr[R] is the trace of the matrix R. This criterion is
well suited for the face verification problem due to the fact
that it tries to find the feature projections that maximize the
distance of impostor jets from the genuine class center while
minimizing the distance of genuine jets from genuine class
center. If B
l
(r) is not singular then (9) is maximized when
the column vectors of the projection matrix,
´
Ψ
l
(r), are the
eigenvectors of B
l
(r)
−1
W
l
(r).
In order to proceed to feature dimensionality reduction
in M < S dimensions the matrix
´
Ψ
l
(r) should be com-
prised by the eigenvectors of B
l
(r)
−1
W
l
(r) that correspond
to the M greatest eigenvalues. The feature vector after dis-
criminant dimensionality reduction is:
´
j(x
l
t
(r)) = g
r
l
(
b
j(x
l
t
(r)) =
´
Ψ
l
(r)
T
b
j(x
l
t
(r)), (10)
The similarity measure of the new feature vectors can
be given by a simple distance metric. We have used the L
2
norm for forming the new feature vector similarity measure
in the final multidimensional space:
C
f
(
´
j(x
l
t
(r)),
´
j(x
l
r
)) = ||
´
j(x
l
t
(r)) −
´
j(x
l
r
)||. (11)
4. LOCAL SIMILARITY MEASURE
DISCRIMINANT WEIGHTING
In [1, 2] only the jet similarity measure has been consid-
ered when forming the total similarity measure between two
graph nodes. The node deformation was only employed im-
plicitly in the matching stage by imposing additional rigid-
ity/plasticity penalties. We propose to combine the feature
vector similarity distance and the node deformation in a dis-
criminant manner in order to form the new local similarity
measure. The node feature similarity measure between the
reference person r and the test person t for the l-th node is
f
l
t
(r) = C
f
(
´
j(x
l
t
(r)),
´
j(x
l
r
)) and the node deformation is
d
l
t
(r) = C
d
(x
l
t
(r), x
l
r
). Let d
l
t
(r) ∈ ℜ
2
be a column vec-
tor that is comprised by the two similarity measures for the
node l between the test person t and the reference person r,
i.e.:
d
l
t
(r) =
f
l
t
(r)
d
l
t
(r)
(12)
According to the standard EGM [1] the node similarity value
after the matching procedure is be given by:
c
l
t
(r) = f
l
t
(r) + λd
l
t
(r) =
1 λ
d
l
t
(r) = e
T
d
l
t
(r)
(13)
where λ is the constant that controls the rigidity/plasticity of
the graph [1]. In general e
T
does not contain any discrim-
inant information. Thus, when forming the local similarity
measure the vector e
T
should be superseded by a discrimi-
nant function µ
r
l
that is person and node specific. The new
local similarity measure is:
c
l
t
(r) = µ
l
r
(d
l
t
(r)). (14)
The discriminant transforms can be constructed by using
linear or non-linear methods for building discriminant func-
tion. We have used LDA in order to find the discriminant
transform µ
l
r
.
Let L
l
C
(r) and L
l
I
(r) be the sets of local similarity vec-
tors d
l
t
(r) that correspond to genuine and impostor claims,
respectively. In order to form the optimization criterion, the
between class scatter matrix, D
l
S
(r), and the within class
scatter matrix, D
l
W
(r), of the local similarity vectors d
l
t
(r)
are employed. The optimization criterion used for finding
the discriminant weighting vector
´
q
l
(r) :
J(q
l
(r)) =
q
l
(r)
T
D
l
S
(r)q
l
(r)
q
l
(r)
T
D
l
W
(r)q
l
(r)
. (15)
The optimal weighting coefficients are given by [4]:
´
q
l
(r) =
D
l
W
(r)
−1
(m(L
l
I
(r)) − m(L
l
C
(r))
||D
l
W
(r)
−1
(m(L
l
I
(r)) − m(L
l
C
(r))||
. (16)
The new similarity value between the l-th node of the refer-
ence graph and the same node of the test graph is now:
c
l
t
(r) = µ
l
r
(d
l
t
(r)) =
´
q
l
(r)
T
d
l
t
(r). (17)
5. DISCRIMINANT NODE WEIGHTING
In the standard EGM all nodes are treated uniformly when
forming the final similarity measure between faces. Thus,
it sounds reasonable to weight the similarity measures of
nodes that correspond to different fiducial points with weights
that correspond to their discriminant power. The weights
should be person specific due to the fact that different per-
sons have different discriminant fiducial points. Let c
t
(r) ∈
ℜ
L
be a column vector comprised by the new local similar-
ity values at every node:
c
t
(r) =
c
1
t
(r)
c
2
t
(r)
.
.
.
c
L
t
(r)
(18)
where L is the number of graph nodes. The vector c
t
(r) is
the total similarity vector between the reference face r and a
test face t. The standard EGM algorithm approach [1] treats
uniformly all the similarity values c
l
t
(r). That is, the total
similarity measure between a reference person r and a test
person t is simple the sum of all node similarity measures:
D
t
(r) =
L
X
i=1
c
i
t
(r) = 1
T
c
t
(r), (19)
where 1 is an L × 1 vector of ones. The algorithm should
learn a discriminant function β
r
that is person specific and
form the total similarity measure between faces:
´
D
t
(r) = β
r
(c
t
(r)). (20)
The transform β
r
could be just a weighting vector or a
more complicated nonlinear support vector machine [3]. We
will use LDA to create a total similarity measure between
the reference person r and a test person t.
Let T
C
(r) and T
I
(r) be the sets of the total similarity
vectors for the genuine and impostor claims of the refer-
ence person r, respectively. Let the within-class scatter ma-
trix and and the between-class scatter for the total similarity
vectors c
t
(r) be V
W
(r) and V
B
(r), respectively. The op-
timal weighting coefficients that are derived from the maxi-
mization of:
J(w(r)) =
w(r)
T
V
B
(r)w(r)
w(r)
T
V
W
(r)w(r)
(21)
are the elements of the vector
´
w(r) [4]:
´
w(r) =
V
W
(r)
−1
(m(T
I
(r)) − m(T
C
(r))
||V
W
(r)
−1
(m(T
I
(r)) − m(T
C
(r))||
. (22)
The similarity distance between the reference person r and
the test person t, after all the successively discriminant steps,
is given by:
´
D
t
(r) = β
r
(c
t
(r)) =
´
w(r)
T
c
t
(r). (23)
Table 1. Error Rates according to XM2VTS protocol for Con-
figuration I
Algorithm
Configuration I
Evaluation set Test set
FAE=FRE FAE(FRE=0) FRE(FAE=0)
FAE=FRE FRE=0 FAE=0 Total Error Rate(TER)
FA FR FA FR FA FR FAE=FRE FRE=0 FAE=0
EGM 9.2 98.2 65.0 7.9 5.0 98.8 0.0 0.0 61.0 12.9 98.8 61.0
EGM-ND 6.3 62.8 56.3 6.7 4.2 63.8 0.0 0.0 61.0 10.7 63.8 61.0
EGM-LD 5.2 45.5 20.0 5.2 4.0 45.0 0.5 0.0 17.0 9.2 45.5 17.0
EGM-FD 2.5 29.9 55.3 2.5 3.2 11.2 0.2 0.2 14.7 5.7 11.4 14.9
DEGM 0.2 0.7 6.5 1.6 1.2 10.2 0.0 0.0 13.1 2.8 10.2 13.1
6. EXPERIMENTAL RESULTS
The experiments were conducted in the XM2VTS database
using the protocol described in [5]. The images were aligned
using an automatic alignment method. A 8×8 graph and a
modified morphological analysis was used. The training set
is used for calculating for each reference person r and for
each node l a matrix
´
Ψ
l
(r) for feature selection. A PCA
step is used prior to discriminant analysis in order to obtain
the invertibility of B
l
(r).
The evaluation set is used for learning the discriminant
vector
´
q
l
(r) for weighting the local similarity vector and
the vector,
´
w(r), that weights the total similarity vector of
the graph nodes. The evaluation set is also used for learning
the thresholds. Table 1 shows the error rates according to
the protocol described in [5].
The EGM using no discriminant step has given an TER
equal to 12.9% in the test set of Configuration I. The best
TER achieved, using only feature vector discriminant anal-
ysis, was 5.7% and was achieved when we kept the first 3
discriminant projections. The step of the discriminant fea-
ture selection using the EGM will denoted as EGM-FD.
We also investigated the contribution of the discriminant
weighting of the local similarity vector. This was conducted
by using no feature projections and by treating uniformly
all the local similarity measures. That way we achieved an
TER equal to 9.2%. When only discrimination between lo-
cal similarity distances is considered we will use the acronym
EGM-LD.
The contribution of weighting the local similarity mea-
sure with coefficients that are derived by LDA without other
discriminant steps was also investigated. To do so, we ap-
plied only discriminant weighting in the graph level by cal-
culating,
´
w
r
, without applying prior discriminant analysis.
The TER obtained was 10.7%. EGM-ND will denote the
EGM when only discriminant weighting of the total similar-
ity vector is performed. The best TER achieved was 2.8%
using successively all the discriminant steps. These results
are the best that have been reported using an
automatic alignment method [6]. The acronym DEGM will
be used when all the discriminant steps were used.
7. CONCLUSIONS
The use of discriminant techniques in the EGM framework
is explored. The different phases of EGM that discriminant
information can be used are indicated. The successively dis-
criminant steps are applied in modified morphological EGM
algorithm.
8. REFERENCES
[1] B. Duc, S. Fischer, and J. Big
¨
un, “Face authentication
with Gabor information on deformable graphs.,” IEEE
Transactions on Image Processing, vol. 8, no. 4, pp.
504–516, Apr. 1999.
[2] C. Kotropoulos, A. Tefas, and I. Pitas, “Frontal face
authentication using discriminating grids with morpho-
logical feature vectors.,” IEEE Transactions on Multi-
media, vol. 2, no. 1, pp. 14–26, Mar. 2000.
[3] A. Tefas, C. Kotropoulos, and I. Pitas, “Using support
vector machines to enhance the performance of elastic
graph matching for frontal face authentication,” IEEE
Transactions on Pattern Analysis and Machine Intelli-
gence, vol. 23, no. 7, pp. 735–746, 2001.
[4] K. Fukunaga, Statistical Pattern Recognition, CA: Aca-
demic, San Diego, 1990.
[5] K. Messer, J. Matas, J.V. Kittler, J. Luettin, and
G. Maitre, “Xm2vtsdb: The extended m2vts database,”
in AVBPA’99, 1999, pp. 72–77.
[6] K. Messer, J.V. Kittler, M. Sadeghi, S. Marcel, C. Mar-
cel, S. Bengio, F. Cardinaux, C. Sanderson, J. Czyz,
L. Vandendorpe, S. Srisuk, M. Petrou, W. Kurutach,
A. Kadyrov, R. Paredes, B. Kepenekci, F.B. Tek, G.B.
Akar, F. Deravi, and N. Mavity, “Face verification com-
petition on the xm2vts database,” in AVBPA03, 2003,
pp. 964–974.