BIROn - Birkbeck Institutional Research Online
Tao, D. and Tang, X. and Li, Xuelong and Rui, Y. (2006) Direct kernel
biased discriminant analysis: a new content-based image retrieval relevance
feedback algorithm. IEEE Transactions on Multimedia 8 (4), pp. 716-727.
ISSN 1520-9210.
Downloaded from:
Usage Guidelines:
Please refer to usage guidelines at or alternatively
contact lib-eprints@bbk.ac.uk.
Birkbeck ePrints: an open access repository of the
research output of Birkbeck College
http://eprints.bbk.ac.uk
Tao, Dacheng; Tang, Xiaoou; Li, Xuelong and Rui,
Yong (2006). Direct kernel biased discriminant
analysis: a new content-based image retrieval
relevance feedback algorithm. IEEE Transactions
on Multimedia 8 (4) 716-727.
Copyright © 2006 IEEE. Reprinted from IEEE Transactions on Multimedia
(ISSN 1520-9210). This material is posted here with permission of the IEEE.
Such permission of the IEEE does not in any way imply IEEE endorsement of
any of Birkbeck’s products or services. Internal or personal use of this
material is permitted. However, permission to reprint/republish this material for
advertising or promotional purposes or for creating new collective works for
resale or redistribution must be obtained from the IEEE by writing to pubs-
permissions@ieee.org.
By choosing to view this document, you agree to all provisions of the
copyright laws protecting it.
Citation for this copy:
Tao, Dacheng; Tang, Xiaoou; Li, Xuelong and Rui, Yong (2006). Direct kernel
biased discriminant analysis: a new content-based image retrieval relevance
feedback algorithm. London: Birkbeck ePrints. Available at:
http://eprints.bbk.ac.uk/archive/00000452
Citation as published:
Tao, Dacheng; Tang, Xiaoou; Li, Xuelong and Rui, Yong (2006). Direct kernel
biased discriminant analysis: a new content-based image retrieval relevance
feedback algorithm. IEEE Transactions on Multimedia 8 (4) 716-727.
http://eprints.bbk.ac.uk
Contact Birkbeck ePrints at lib-eprints@bbk.ac.uk
716 IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 8, NO. 4, AUGUST 2006
Direct Kernel Biased Discriminant Analysis:
A New Content-Based Image Retrieval
Relevance Feedback Algorithm
Dacheng Tao, Student Member, IEEE, Xiaoou Tang, Senior Member, IEEE, Xuelong Li, Member, IEEE, and
Yong Rui, Senior Member, IEEE
Abstract—In recent years, a variety of relevance feedback (RF)
schemes have been developed to improve the performance of con-
tent-based image retrieval (CBIR). Given user feedback informa-
tion, the key to a RF scheme is how to select a subset of image fea-
tures to construct a suitable dissimilarity measure. Among various
RF schemes, biased discriminant analysis (BDA) based RF is one of
the most promising. It is based on the observation that all positive
samples are alike, while in general each negative sample is negative
in its own way. However, to use BDA, the small sample size (SSS)
problem is a big challenge, as users tend to give a small number
of feedback samples. To explore solutions to this issue, this paper
proposes a direct kernel BDA (DKBDA), which is less sensitive to
SSS. An incremental DKBDA (IDKBDA) is also developed to speed
up the analysis. Experimental results are reported on a real-world
image collection to demonstrate that the proposed methods outper-
form the traditional kernel BDA (KBDA) and the support vector
machine (SVM) based RF algorithms.
Index Terms—Biased discriminant analysis (BDA), content-
based image retrieval (CBIR), direct kernel biased discriminant
analysis (DKBDA), incremental direct kernel biased discriminant
analysis (IDKBDA), kernel biased discriminant analysis (KBDA),
relevance feedback (RF).
I. INTRODUCTION
W
ITH the explosive growth in image records and the rapid
increase of computer power, retrieving images from a
large-scale image database becomes one of the most active re-
search fields [1], [2]. To give all images text annotations manu-
ally is tedious and impractical and to automatically annotate an
image is beyond current technology.
Content-based image retrieval (CBIR) is a technique to re-
trieve images semantically relevant to the user’s query from an
image database. It is based on automatically extracted visual
features from an image, such as color [3], [4], [10]–[12], texture
[5]–[10], [12], and shape [11]–[13]. However, the gap between
these low-level visual features and high-level semantic mean-
ings usually leads to poor performance.
Manuscript received October 31, 2004; revised May 31, 2005. The work of
D. Tao and X. Tang was supported by grants from the Research Grants Council
of the Hong Kong (SAR). The associate editor coordinating the review of this
manuscript and approving it for publication was Dr. Jiebo Luo.
D. Tao and X. Li are with the School of Computer Science and Information
Systems, Birkbeck, University of London, London WC1E 7HX, U.K. (e-mail:
dacheng@dcs.bbk.ac.uk; xuelong@dcs.bbk.ac.uk).
X. Tang is with the Department of Information Engineering, The Chinese
University of Hong Kong, Shatin, N.T., Hong Kong (e-mail: xtang@ie.cuhk.
edu.hk).
Y. Rui is with the Microsoft Research, Redmond, WA 98052 USA (e–mail:
yongrui@microsoft.com).
Digital Object Identifier 10.1109/TMM.2005.861375
Relevance feedback (RF) is a way to bridge this gap and to
scale the performance in CBIR systems [14]–[17]. RF focuses
on the interactions between the user and the search engine by
letting the user label semantically positive or negative samples.
RF is different from the traditional classification problem be-
cause the user is not likely to label a large number of images.
As a result, small sample learning methods, where the number
of the training samples is much smaller than the dimension of
the descriptive features, are important in CBIR RF. Discrimi-
nant analysis [18]–[26] and the support vector machine (SVM)
method [27]–[31] are two small sample learning methods used
in recent years to obtain
state-of-the-art performances.
Discriminant analysis [18] is one of the most popular solu-
tions for the small sample learning problem. In the last 20 years,
Fisher linear discriminant analysis (LDA) has been successfully
used in face recognition [19]–[23], [26]. LDA was first used
in CBIR for feature selection and extracts the most discrimi-
nant subset feature for image retrieval. The remaining images
in the database were then projected onto the subspace and fi-
nally, some similarity or dissimilarity measures were used to
sort these images. However, with LDA all negative feedbacks
are deemed equivalent, and this is a severe limitation of the
method because all positive examples are alike and each neg-
ative example is negative in its own way. With this observation,
biased discriminant analysis (BDA) [24], [25] was developed
by Zhou and Huang to scale the performance of CBIR and ob-
tained a more satisfactory result. In the BDA model, the nega-
tive feedbacks are required to stay away from the center of the
positive feedbacks. Motivated by the kernel trick successfully
used in pattern recognition [32], Zhou et al. also generalized
the BDA to the kernel feature space as the kernel biased dis-
criminant analysis (KBDA). KBDA performs much better than
BDA [24], [25]. Just like LDA, BDA and KBDA also lead to
the small sample size (SSS) problem [33] because the number
of the sample is much smaller than the dimension of the repre-
sentative features of images. Traditionally, the SSS problem is
solved by the regularization method [33], [24], [25].
However, the regularization method to solve the SSS problem
is not a good choice for LDA, as is pointed out by many papers
on face recognition [20]–[23], [26]. We aim to significantly im-
prove the performance of CBIR RF and utilize the direct idea
to the BDA algorithm in the kernel feature space. We name the
approach as the direct kernel BDA (DKBDA) [34]. DKBDA is
motivated by a) direct LDA (DLDA) [23], [26], which has been
successfully applied to face recognition; (b) unlike face recogni-
tion, image retrieval deals with diverse images, so the nonlinear
1520-9210/$20.00 © 2006 IEEE
TAO et al.: DIRECT KERNEL BIASED DISCRIMINANT ANALYSIS 717
properties of image features should be considered because of
the success of kernel algorithms in pattern recognition.
The DKBDA algorithm can be regarded as an enhanced
KBDA. According to the kernel trick idea, the original input
space is first nonlinearly mapped to an arbitrarily high dimen-
sion feature space, in which the distribution of the images’
patterns is linearized. Then, the DLDA idea [23], [26] is used to
obtain a set of optimal discriminant basis vectors in the kernel
feature space. The BDA criterion is modified as in Liu
et al.
[20], so that a robust result can be gained.
The following section describes the related previous work:
BDA, KBDA, and DLDA; DKBDA is then proposed in Sec-
tion III; in Section IV, an image retrieval system is introduced;
in Section V, a large number of experiments validate the effec-
tiveness and efficiency of DKBDA on a large real world image
database; possible future work is briefly described in Section VI;
finally, Section VII draws conclusions. Detailed deduction of
DKBDA is given in Appendix A; Appendix B provides full de-
duction of the incremental DKBDA (IDKBDA).
II. P
REVIOUS WORK
In this section, previous work including Direct Linear Dis-
criminant Analysis (DLDA), Biased Discriminant Analysis
(BDA), and Kernel Biased Discriminant Analysis (KBDA) are
introduced.
A. Direct Linear Discriminant Analysis (DLDA)
Before describing DLDA [23], we first describe linear dis-
criminant analysis (LDA) [18].
LDA tries to find the best discriminating subspace for dif-
ferent classes. It is spanned by a set of vectors
, which aims
at maximizing the ratio between
and , the within-class
scatter matrix and the between-class scatter matrix:
(1)
Assume the training set contains
individual classes and each
class
has samples. Then and are defined as
(2)
where
. is the mean vector
of the total training set.
is the mean
vector for the individual class
. is the sample belongs to
class
. Therefore, can be computed from the eigenvectors
of
.Given equals 2, LDA changes to Fisher discrimi-
nant analysis (FDA); otherwise, multiple discriminant analysis
(MDA).
LDA has the SSS problem when the number of the training
samples is smaller than the dimension of the low-level visual
features, which is almost always true for CBIR RF.
Yu et al. [23] propose a DLDA method. It accepts high-
dimensional data as input, and optimizes Fisher’s criterion
directly without any feature extraction or dimension reduction
steps. So, it takes advantage of all the information within and
outside of the null space of
. In this approach, is first
diagonalized, then the null space of
is removed:
(3)
where
comprises eigenvectors and comprises the corre-
sponding nonzero eigenvalues of
. is transformed to
(4)
where
is diagonalized by eigenanalysis:
(5)
The LDA transformation matrix is defined as
(6)
In DLDA, the null space of
is removed, and the discrimi-
nant vectors are restricted in the subspace spanned by class cen-
ters. It is assumed that the null space of
contains no discrim-
inative information at all.
B. Biased Discriminant Analysis (BDA)
Zhou et al. [24], [25] developed BDA, which defines the
-class classification problem. This means there is an unknown
number of classes but the user is only interested in one class.
BDA tries to find the subspace to discriminate the positive
samples (the only class of concern to the user) and negative sam-
ples (unknown number of classes). It is spanned by a set of vec-
tors
maximizing the ratio between the biased matrix and
the positive covariance matrix
:
(7)
Assume the training set contains
positive and negative
samples.
and can be defined as (8):
(8)
where
denotes the positive samples, denotes the negative
samples, and
is the mean vector of the
positive samples.
can be computed from the eigenvectors
of
. Firstly, BDA minimizes the variance of the positive
samples. Then it maximizes the distance between the two cen-
ters of the positive feedbacks and all negative feedbacks.
C. Kernel Biased Discriminant Analysis (KBDA)
The data is in a nonlinear space, in which the kernel method is
successfullyused.Therefore,BDAisgeneralized toitskernelver-
sion, named as KBDA. To obtain the nonlinear generalization, the
linear input space is mapped to a nonlinear kernel feature space:
(9)
(10)
718 IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 8, NO. 4, AUGUST 2006
The data is mapped from into a po-
tentially much higher dimensional feature space
. Now, given
a learning problem, one can consider the BDA in
instead of
. In other words, the idea behind KBDA is to perform the
BDA in the feature space
instead of the input space .
Let
and be the positive within-class scatter and the
negative scatter with respect to positive centroid matrices in the
feature space
. They can be respectively expressed as follows:
(11)
(12)
where
is the centroid of positive
samples,
is the positive samples’ number, and is the neg-
ative samples’ number. KBDA determines a set of optimal dis-
criminant basis vectors
, which, according to
eigenvectors of
, can be obtained to solve the following
eigenvalue problem:
(13)
The dimension of the feature space
is arbitrarily high, and
possibly infinite. Fortunately, there is no need to use the exact
to calculate , because the kernel method can be utilized
to avoid mapping the feature point from the linear input space
to a nonlinear kernel feature space. This mapping is based on
replacing the dot product with a kernel function in
.
In KBDA based RF, the number of feedback samples is much
smaller than the dimension of the low-level visual feature. This
leads to a degenerated
, i.e., the SSS problem or the matrix
singular problem. Zhou et al. [24], [25] solve the SSS problem
by the regularized version
and , which adds small quanti-
ties to the diagonal of the scatter matrices. However, this is not
an optimal solution and sometimes it may lead to an ill-posed
problem, which limits the performance of their method.
III. D
IRECT KERNEL BIASED DISCRIMINANT ANALYSIS
(DKBDA) AND ITS INCREMENTAL VERSION
The regularization method to solve the SSS problem is not a
good choice for LDA, as is pointed out by many papers on face
recognition [20]–[23], and [26]. We aim to significantly improve
the performance of CBIR RF and utilize the direct idea to the
BDA algorithm in the kernel feature space. This direct method
is proposed based on all positive examples are alike and each
negative example is negative in its own way [24], [25]. We name
the approach as the direct kernel BDA (DKBDA).
DKBDA is motivated by a) the fact that direct LDA (DLDA)
[23], [26], recently developed for face recognition, has made
some advances; and b) unlike face recognition, image retrieval
deals with diverse images, so the nonlinear properties of image
features should be considered because of the success of kernel
algorithms in pattern recognition.
DKBDA can be regarded as an enhanced KBDA. According
to the kernel trick idea, the original input space is first nonlin-
early mapped to an arbitrarily high dimension feature space,
in which the distribution of the images’ patterns is linearized.
Then, the DLDA idea [23], [26] is used to obtain a set of op-
timal discriminant basis vectors in the kernel feature space. The
BDA criterion is modified as in Liu et al. [20], so that a robust
result can be gained. First of all, the kernel matrix
is intro-
duced:
(14)
where
where stands for positive feedback samples, and is the
number of positive feedback samples;
stands for negative
feedback samples, and
is the number of negative feedback
samples.
is the kernel function. Some typical kernel func-
tions can be employed, such as Polynomial, Gaussian, or Sig-
moid based kernel functions.
DKBDA begins from the analysis of the negative scatter with
respect to positive centroid matrix (11). Since the dimension of
could be arbitrarily infinitive, it is impossible to calculate
directly and implement eigen analysis with .
Fortunately, this can be avoided through the following analysis:
(15)
The dimension of
is the number of negative RF sam-
ples. The next problem is to obtain the matrix:
(16)
, , , and
should then be calculated. The detailed deductions can be seen
from Appendix A and the results are given by the following
formulations:
(17)
(18)
(19)
where
is an by 1 column vector (all terms equal to 1).
