Graph embedding discriminant analysis on Grassmannian
manifolds for improved image set matching
Author
Harandi, Mehrtash T, Sanderson, Conrad, Shirazi, Sareh, Lovell, Brian C
Published
2011
Conference Title
CVPR 2011
Version
Accepted Manuscript (AM)
DOI
https://doi.org/10.1109/cvpr.2011.5995564
Copyright Statement
© 2011 IEEE. Personal use of this material is permitted. Permission from IEEE must be
obtained for all other uses, in any current or future media, including reprinting/republishing this
material for advertising or promotional purposes, creating new collective works, for resale or
redistribution to servers or lists, or reuse of any copyrighted component of this work in other
works.
Downloaded from
http://hdl.handle.net/10072/401034
Griffith Research Online
https://research-repository.griffith.edu.au
Graph Embedding Discriminant Analysis on Grassmannian Manifolds
for Improved Image Set Matching
Mehrtash T. Harandi, Conrad Sanderson, Sareh Shirazi, Brian C. Lovell
NICTA, PO Box 6020, St Lucia, QLD 4067, Australia
The University of Queensland, School of ITEE, QLD 4072, Australia
Abstract
A convenient way of dealing with image sets is to represent
them as points on Grassmannian manifolds. While several
recent studies explored the applicability of discriminant
analysis on such manifolds, the conventional formalism of
discriminant analysis suffers from not considering the local
structure of the data. We propose a discriminant analysis
approach on Grassmannian manifolds, based on a graph-
embedding framework. We show that by introducing within-
class and between-class similarity graphs to characterise
intra-class compactness and inter-class separability, the ge-
ometrical structure of data can be exploited. Experiments
on several image datasets (PIE, BANCA, MoBo, ETH-80)
show that the proposed algorithm obtains considerable im-
provements in discrimination accuracy, in comparison to
three recent methods: Grassmann Discriminant Analysis
(GDA), Kernel GDA, and the kernel version of Affine Hull
Image Set Distance. We further propose a Grassmannian
kernel, based on canonical correlation between subspaces,
which can increase discrimination accuracy when used in
combination with previous Grassmannian kernels.
1. Introduction
In contrast to object recognition approaches based on
considering one image at a time, there has been a recent
surge of interest in techniques based on explicit image set
matching [9, 16, 25, 26]. This is mainly driven by the need
for superior discrimination accuracy as well as increased
robustness to practical issues such as pose variations, mis-
alignment and varying environmental conditions (for exam-
ple, as present in realistic face recognition scenarios [21]).
While image set matching can be accomplished through
probability-density based methods [3, 8] and aggregation
methods [17], it has been shown that better performance can
be attained through modelling image sets via linear struc-
tures (ie., subspaces) [25, 29]. Subspaces appear to be ap-
∗
Acknowledgements: NICTA is funded by the Australian Government
as represented by the Department of Broadband, Communications and the
Digital Economy, as well as the Australian Research Council through the
ICT Centre of Excellence program. The second and third authors con-
tributed equally. We thank Prof. Terry Caelli for useful discussions.
propriate models for this task since they are able to accom-
modate the effects of various image variations. For exam-
ple, an acceptable and widely used approximation for pho-
tometric invariance, under conditions of no shadowing and
Lambertian reflectance, is a 4 dimensional linear space [1].
A convenient way of dealing with subspaces is
to represent them as points on Grassmannian mani-
folds [11, 13, 19, 25]. Recently, several studies explored the
applicability of discriminant analysis (DA) on such man-
ifolds [13, 26]. Given subspaces that are represented as
points on a Grassmannian manifold M, the underlying idea
is to map them to another Grassmannian manifold M
′
, such
that a measure of discriminatory power on M
′
is maximised
(see Fig. 1 for a conceptual example).
While the approaches presented in [13, 26] show promis-
ing results, the conventional formalism of DA suffers from
not being able to take into account the local structure of
data [10, 15]. For example, outliers and multi-modal classes
can adversely affect the discrimination and/or generalisa-
tion ability of models based on conventional DA.
Motivated by advances in DA over Euclidean vector
spaces [30, 24], we propose a novel DA on Grassmannian
manifolds, based on a graph-embedding framework [30].
We show that considerable gains in discrimination accu-
racy can be obtained by exploiting the geometrical structure
and local information on Grassmannian manifolds. This
is achieved by introducing within-class and between-class
similarity graphs to characterise intra-class compactness
and inter-class separability, respectively.
The proposed method for DA on Grassmannian mani-
folds is somewhat related to distance metric learning meth-
ods [28]. The main points of difference include the use of
graphs and manifolds in contrast to the typical use of vector
spaces in distance metric learning. Overall, the proposed
method can be considered as an extension of both graph-
embedding and distance metric learning to higher order data
structures.
We also propose a new kernel, based on canonical cor-
relation between subspaces, for measuring the similarity of
two points on a Grassmannian manifold. We empirically
show that, in combination with previous Grassmannian ker-
nels, the new kernel can result in considerable discrimina-
tion accuracy improvements.
θ
D
ℝ
(a) (b) (c)
Figure 1. A conceptual illustration of the proposed approach. (a) Image-sets can be described in R
D
by linear subspaces. To compare two
linear subspaces, the principal angles between them can be used. For clarity just two subspaces are shown. (b) Linear subspaces in R
D
can be represented as points on the Grassmannian manifold M. Having a proper geodesic distance between the points on the manifold,
it is possible to convert the image-set matching problem into a point to point classification problem. (c) By having a Grassmannian
kernel in hand, points on the Grassmannian manifold can be mapped into another Grassmannian manifold where not only certain local
properties have been retained but also the discriminatory power between classes has been increased. Unlike the conventional formalism
of discriminant analysis, the proposed method preserves the geometrical structure and local information on Grassmannian manifolds by
exploiting within-class and between-class similarity graphs.
We continue the paper as follows. Section 2 provides an
overview of Grassmannian analysis, which leads to the pro-
posed graph embedding discriminant analysis in Section 3.
We introduce the Grassmannian canonical correlation ker-
nel in Section 4. In Section 5 we briefly describe the overall
computational complexity of the proposed method. In Sec-
tion 6 we compare the performance of the proposed method
and kernel with previous approaches on several object and
face datasets. The main findings and possible future direc-
tions are summarised in Section 7.
2. Grassmannian Analysis
Manifold analysis has been extensively considered with
success by various disciplines. Amari and Nagaoka state
that many important structures in information theory and
statistics can be treated as structures in differential geom-
etry by regarding a space of probabilities as a Riemannian
manifold [2]. A manifold is a topological space that is lo-
cally similar to Euclidean space. At an intuitive level, man-
ifolds can be thought of as smooth, curved surfaces embed-
ded in higher dimensional Euclidean spaces. Riemannian
manifolds are endowed with a distance measure which al-
lows us to measure how similar two points are. In this work
we are interested in a particular class of Riemannian mani-
folds, known as Grassmannian manifolds [11].
Points on a Grassmannian manifold, G
D,m
, can be
viewed as the set of m-dimensional subspaces of R
D
and
are represented by orthonormal matrices, each with a size of
D × m. Two points on a Grassmannian manifold are equiv-
alent if one can be mapped into the other one by a m × m
orthogonal matrix [11].
Grassmannian analysis provides a natural way to tackle
the problem of image set matching. Specifically, as G
D,m
is the manifold parameterising m-dimensional real vector
subspaces of the D-dimensional vector space R
D
, the clas-
sification problem of matching sets comprising m images,
where each image is described by D pixels, can be trans-
formed to a point classification problem on G
D,m
.
During the past decade the concept of angles between
subspaces, ie., principal angles has been widely used for im-
age set matching [29]. Since Grassmannian manifolds are
curved and the shortest distance between points is geodesic,
it is not surprising to see that distances over Grassmannian
manifolds may outperform methods based on principal an-
gles. We note that principal angles can be considered as a
simple form of geodesic distance on Grassmannian mani-
folds [19].
Grassmannian kernels [13, 14, 27] allow us to treat the
Grassmannian space as if it were a Euclidean vector space.
As a result, learning algorithms in vector spaces can be ex-
tended to their counterparts on Grassmannian manifolds,
eg., kernel discriminant analysis [13, 26]. In the following
section we will demonstrate how Grassmannian kernels can
be employed to map points on a Grassmannian manifold
onto another Grassmannian manifold, where a measure of
discriminatory power between classes has been maximised.
3. Graph Embedding Discriminant Analysis
Linear Discriminant Analysis (LDA) is a supervised sta-
tistical learning method that seeks a linear projection by si-
multaneously maximising the between-class dissimilarities
and minimising the within-class dissimilarities [6]. While
LDA has been successfully applied to various computer
vision problems, eg., face recognition [5], it suffers from
not being able to naturally capture the local structure of
data [10, 24]. For example, LDA has problems handling
multi-modal classes (where each class is comprised of sev-
eral separate clusters) or when there are outliers in the data.
This stems from treating all data points in the same manner
(during the calculation of within-class and between-class
scatter matrices), no matter how they are related to their
classes.
To alleviate the above problem, a graph-embedding
framework can be used [7, 24, 30]. A graph (V , W ) in our
context refers to a collection of vertices or nodes, V , and a
collection of edges that connect pairs of vertices. We note
that W is a symmetric matrix with elements describing the
similarity between pairs of vertices. Moreover, the diagonal
matrix D and the Laplacian matrix L of a graph are defined
as L = D − W , with the diagonal elements of D obtained
as D(i, i) =
P
j6=i
W (i, j).
Given a graph in a vector space, the purpose of graph-
embedding DA is to maximise a measure of discriminatory
power by mapping the underlying data into another vec-
tor space (usually with lower dimensionality) while pre-
serving similarities between vertex pairs. This problem
can be solved through a generalised eigen-analysis frame-
work [30]. In the following text, we formulate the discrim-
inant analysis over Grassmannian manifolds based on the
graph-embedding framework.
Given N labelled points X = {(X
i
, l
i
)}
N
i=1
from the un-
derlying Grassmannian manifold M, where X
i
∈ R
D×m
and l
i
∈ {1, 2, · · · , C}, with C denoting the number of
classes, the local geometrical structure of M can be mod-
elled by building a within-class similarity graph W
w
and
a between-class similarity graph W
b
. The simplest forms
of W
w
and W
b
are based on the nearest neighbour graphs
defined in Eqns. (1) and (2):
W
w
(i, j) =
1, if X
i
∈ N
w
(X
j
) or X
j
∈ N
w
(X
i
)
0, otherwise
(1)
W
b
(i, j) =
1, if X
i
∈ N
b
(X
j
) or X
j
∈ N
b
(X
i
)
0, otherwise
(2)
In Eqn. (1), N
w
(X
i
) is the set of v neighbours
˘
X
1
i
, X
2
i
, ..., X
v
i
¯
, sharing the same label as l
i
. Similarly
in Eqn. (2), N
b
(X
i
) contains v neighbours having different
labels. We note that more complex similarity graphs, like
heat kernel graphs, can also be used to encode distances be-
tween points on Grassmannian manifolds [20].
Our aim is to maximise discriminatory power while si-
multaneously preserving geometry, by mapping the points
on M to a new manifold M
′
, ie., α : X
i
→ Y
i
. A suitable
transform would place the connected points of W
w
as close
as possible, while moving the connected points of W
b
as far
as possible. Such a mapping can be described by optimising
the following two objective functions:
f
1
= min
1
2
X
i,j
(Y
i
− Y
j
)
2
W
w
(i, j) (3)
f
2
= max
1
2
X
i,j
(Y
i
− Y
j
)
2
W
b
(i, j) (4)
Eqn. (3) punishes neighbours in the same class if they are
mapped far away in M
′
, while Eqn. (4) punishes points of
different classes if they are mapped close together in M
′
.
Assume that points on the manifold are implicitly known
and only a measure of similarity between them is available
through a Grassmannian kernel
1
, k
ij
= hX
i
, X
j
i.
Confining the solution to be linear, ie.,
α
i
=
P
N
j=1
a
ij
X
j
, we will have:
Y
i
= (hα
1
, X
i
i , hα
2
, X
i
i , · · · , hα
r
, X
i
i)
T
(5)
By defining A
l
=(a
l1
, a
l2
, · · ·, a
lN
)
T
and K
i
=(k
i1
, k
i2
, · · ·, k
iN
)
T
it can be shown that hα
l
, X
i
i = A
T
l
K
i
. Hence Eqn. (3) can
be simplified to:
1
2
P
i,j
(Y
i
− Y
j
)
2
W
w
(i, j)
=
1
2
P
i,j
`
A
T
i
K
i
− A
T
j
K
j
´
2
W
w
(i, j)
=
P
i
A
T
i
K
i
K
T
i
A
T
i
W
w
(i, i)
−
P
i,j
A
T
i
K
j
K
T
i
A
T
i
W
w
(i, j)
= A
T
KD
w
K
T
A − A
T
KW
w
K
T
A
(6)
where A = [A
1
|A
2
| · · · |A
r
] and K = [K
1
|K
2
| · · · |K
N
].
Considering that L
b
= D
b
− W
b
, in a similar manner it can
be shown that Eqn. (4) can be simplified to:
1
2
P
i,j
(Y
i
− Y
j
)
2
W
b
(i, j)
= A
T
KD
b
K
T
A − A
T
KW
b
K
T
A
= A
T
KL
b
K
T
A
(7)
Following [7, 30], a constraint is imposed on Eqn. (3) and
the minimisation problem is converted to a maximisation
one. Specifically, by forcing A
T
KD
w
K
T
A to be a constant
such as 1, Eqn. (3) becomes the following maximisation
problem:
min
˘
A
T
KD
w
K
T
A − A
T
KW
w
K
T
A
¯
= min
˘
1 − A
T
KW
w
K
T
A
¯
= max
˘
A
T
KW
w
K
T
A
¯
(8)
subject to
A
T
KD
w
K
T
A = 1 (9)
By converting both problems into maximisation, the overall
optimisation problem is hence:
max
˘
(A
T
K(L
b
+ βW
w
)K
T
A
¯
subject to A
T
KD
w
K
T
A = 1
(10)
where β is a Lagrangian multiplier that acts as a regulari-
sation parameter in the final solution. The solution of (10)
can be found through the following generalised eigenvalue
problem:
K {L
b
+ βW
w
} K
T
A = λKD
w
K
T
A (11)
More specifically, the desired projection matrix A, is equal
to the r largest eigenvectors of the Rayleigh quotient:
KD
w
K
T
K {L
b
+ βW
w
} K
T
(12)
1
We use the notation hX
i
, X
j
i to indicate a similarity measure
between points X
i
and X
j
on a Grassmannian manifold. This is similar
in principle to an inner product in Hilbert space, as used in kernel-based
methods [22].
Fig. 2 outlines the proposed graph embedding method on
Grassmannian manifolds. The proposed algorithm uses the
points on the Grassmannian manifold implicitly (ie., via
measuring similarities through a kernel) to obtain a map-
ping, A = [A
1
|A
2
| · · · |A
r
] that maximises a quotient similar
to discriminant analysis, while retaining the overall geomet-
rical structure.
Upon acquiring the mapping A, the matching problem
over Grassmannian manifolds is reduced to classification
in vector spaces. More precisely, for any query image
set X
q
, a vector representation using the kernel function
and the mapping A is acquired, ie., V
q
= A
T
K
q
, where
K
q
= (hX
1
, X
q
i , hX
2
, X
q
i , · · · , hX
N
, X
q
i)
T
. Similarly,
gallery points X
i
are represented by r dimensional vectors
V
i
= A
T
K
i
and classification methods such as Nearest-
Neighbour or Support Vector Machines [6] can be em-
ployed to label X
q
.
4. Grassmannian Kernels
The similarity between two points on a Grassmannian
manifold, eg., X
i
and X
j
∈ R
D×m
, can be measured using
kernels such as the projection kernel:
k
[proj]
i,j
=
‚
‚
‚
X
T
i
X
j
‚
‚
‚
2
F
(13)
One of the first attempts to solve the problem of image set
matching was based on the notion of principal angles. More
precisely, Yamaguchi et al. [29] used the largest canonical
correlation value (the cosine of principal angles) to mea-
sure the similarity between two image sets. In Section 4.1
we show that the largest canonical correlation between sub-
spaces is a kernel on Grassmannian manifolds. We then
show in Section 4.2 that a more complex kernel, created
through linearly combining existing Grassmannian kernels,
is also a Grassmannian kernel.
We will later demonstrate that combining the projection
kernel with the proposed canonical correlation kernel can
lead to considerable improvements in discrimination accu-
racy, in the context of the proposed graph-embedding dis-
criminant analysis.
4.1. Canonical Correlation Kernel
Given subspaces X
i
and X
j
, we define the canonical
correlation kernel as:
k
[CC]
i,j
= max
a
p
∈span(X
i
)
max
b
q
∈span(X
j
)
a
T
p
b
q
(14)
subject to a
T
p
a
p
= b
T
p
b
p
= 1 and a
T
p
a
q
= b
T
p
b
q
= 0, p 6= q.
For k
[CC]
to be a Grassmannian kernel [14], it must be
(i) positive definite, and (ii) well defined, meaning it
is invariant to various representations of the subspaces,
ie., k(X
1
, X
2
) = k(X
1
R
1
, X
2
R
2
), ∀ R
1
, R
2
∈ Q(m),
where Q(m) indicates orthonormal matrices of order m.
Since the singular values of X
T
1
X
2
are equal to
R
T
1
X
T
1
X
2
R
2
, the canonical correlation kernel is well-
defined. To show that the kernel matrix [K]
ij
= k
[CC]
i,j
is pos-
itive definite, it suffices to show that z
T
Kz > 0 for ∀z ∈ R
n
:
z
T
Kz =
0
B
B
B
@
z
1
z
2
.
.
.
z
n
1
C
C
C
A
T
0
B
B
B
B
@
k
[CC]
1,1
k
[CC]
1,2
. . . k
[CC]
1,n
k
[CC]
2,1
k
[CC]
2,2
. . . k
[CC]
2,n
.
.
.
.
.
.
.
.
.
k
[CC]
n,1
k
[CC]
n,2
. . . k
[CC]
n,n
1
C
C
C
C
A
0
B
B
B
@
z
1
z
2
.
.
.
z
n
1
C
C
C
A
= z
2
1
k
[CC]
1,1
+ z
2
2
k
[CC]
2,2
+ . . . + z
2
n
k
[CC]
n,n
+2
“
z
1
z
2
k
[CC]
1,2
+ z
1
z
3
k
[CC]
1,3
+ . . . + z
1
z
n
k
[CC]
1,n
”
+2
“
z
2
z
3
k
[CC]
2,3
+ z
2
z
4
k
[CC]
2,4
+ . . . + z
2
z
n
k
[CC]
2,n
”
+ . . . + 2z
n−1
z
n
k
[CC]
n−1,n
(15)
In Eqn. (15) we have used the fact that k
[CC]
i,j
= k
[CC]
j,i
. Since
the principal angle between X
i
to itself is zero, k
[CC]
i,i
= 1.
Hence Eqn. (15) can be further simplified to:
z
T
Kz =
n
X
i=1
z
i
!
2
− 2
n
X
i=1
X
j6=i
z
i
z
j
+ 2
n
X
i=1
X
j6=i
z
i
z
j
k
[CC]
i,j
=
n
X
i=1
z
i
!
2
+ 2
n
X
i=1
X
j6=i
z
i
z
j
“
k
[CC]
i,j
− 1
”
(16)
Note that min
“
z
i
z
j
“
k
[CC]
i,j
-1
””
= − z
i
z
j
, since k
[CC]
i,j
∈ [0, 1].
Consequently:
min
“
z
T
Kz
”
=
n
X
i=1
z
i
!
2
− 2
n
X
i=1
X
j6=i
z
i
z
j
(17)
As the right-hand side of Eqn. (17) is always positive for
z
i
6= 0, K is a positive-definite matrix.
Input:
• Training set X = {(X
i
, l
i
)}
N
i=1
from the underlying Grass-
mannian manifold, where X
i
∈ R
D×m
is a subspace
(obtained for example via SVD over an image-set) and
l
i
∈ {1, 2, · · · , C}, with C denoting the number of classes
• A kernel function k
ij
, for measuring the similarity between two
points on the Grassmannian manifold
Processing:
1. Compute the Gram matrix [K]
ij
for all X
i
, X
j
2. Compute the within-class and between-class graph similarity
matrices, W
w
, W
b
, respectively, the between Laplacian ma-
trix L
b
and the diagonal within matrix D
w
3. To obtain A, solve the maximisation problem in Eqn. (11) by
eigen decomposition; A is equal to the r largest eigenvectors
of the Rayleigh quotient
KD
w
K
T
K{L
b
+βW
w
}K
T
Output:
• The projection matrix A = [A
1
|A
2
| · · · |A
r
], where each A
i
is an eigenvector found in step 3 above; the eigenvectors are
sorted in a descending manner according to their corresponding
eigenvalues
Figure 2. Pseudocode for training Grassmannian graph-
embedding discriminant analysis.
![Table 1. Average correct recognition rate for image set matching using geodesic distance, GDA [13], KGDA [26], as well as the proposed algorithm in conjunction with the projection kernel. The standard deviation is shown in brackets. The suffix in the dataset name indicates the number of images per set (eg. the 6 in BANCA-6).](/figures/table-1-average-correct-recognition-rate-for-image-set-25c3yfa1.webp)
![Table 2. Average correct recognition rate for image set matching using Kernel AHISD [9], as well as the proposed algorithm in conjunction with various kernels. The standard deviation is shown in brackets.](/figures/table-2-average-correct-recognition-rate-for-image-set-2e7geztd.webp)
![Figure 5. Performance on CMU-PIE for various values of γ[CC], used for combining k[proj] and k[CC].](/figures/figure-5-performance-on-cmu-pie-for-various-values-of-g-cc-15auupdq.webp)
