Max-Planck-Institut
für biologische Kybernetik
Spemannstraße 38 • 72076 Tübingen • Germany
Arbeitsgruppe Bülthoff
Technical Report No. 44 December 1996
Nonlinear Comp onent Analysis as a Kernel
Eigenvalue Problem
Bernhard Scholkopf, Alexander Smola, and Klaus{Rob ert M uller
Abstract
We describ e a new metho d for performing a nonlinear form of Principal ComponentAnal-
ysis. By the use of integral op erator kernel functions, we can eciently compute principal
components in high{dimensional feature spaces, related to input space by some nonlinear
map for instance the space of all p ossible 5{pixel products in 16
16 images. We givethe
derivation of the method, along with a discussion of other techniques whichcanbemade
nonlinear with the kernel approach and present rst exp erimental results on nonlinear fea-
ture extraction for pattern recognition.
AS and KRM are with GMD First (Forschungszentrum Informationstechnik), Rudower Chaussee 5, 12489
Berlin. AS and BS were supp orted by grants from the Studienstiftung des deutschen Volkes. BS thanks
the GMD First for hospitality during two visits. AS and BS thank V. Vapnik for introducing them
to kernel representations of dot pro ducts during jointwork on Supp ort Vector machines. This work
proted from discussions with V. Blanz, L. Bottou, C. Burges, H. B ultho, K. Gegenfurtner, P. Haner,
N. Murata, P. Simard, S. Solla, V. Vapnik, and T. Vetter. We are grateful to V. Blanz, C. Burges, and
S. Solla for reading a preliminary version of the manuscript.
This do cumentisavailable as
/pub/mpi-memos/TR-044.ps
via anonymous ftp from
ftp.mpik-tueb.mpg.d
e
or
from the World Wide Web,
http://www.mpik-tueb.mpg.de/bu.html
.
1 Intro duction
Principal Comp onent Analysis (PCA) is a power-
ful technique for extracting structure from p ossi-
bly high{dimensional data sets. It is readily per-
formed by solving an Eigenvalue problem, or by
using iterative algorithms which estimate princi-
pal components for reviews of the existing liter-
ature, see Jollie (1986) and Diamataras & Kung
(1996). PCA is an orthogonal transformation of
the co ordinate system in whichwe describe our
data. The new coordinate values bywhichwe rep-
resent out data are called
principal components
.It
is often the case that a small number of principal
components is sucient to account for most of the
structure in the data. These are sometimes called
the
factors
or
latent variables
of the data.
The presentwork generalizes PCA to the case
where we are not interested in principal compo-
nents in input space, but rather in principal com-
ponents of variables, or
features
, which are non-
linearly related to the input variables. Among
these are for instance variables obtained by taking
higher{order correlations b etween input variables.
In the case of image analysis, this would amount
to nding principal components in the space of
products of input pixels.
To this end, we are using the method of ex-
pressing dot pro ducts in feature space in terms
of kernel functions in input space. Given
any
al-
gorithm which can be expressed solely in terms
of dot pro ducts, i.e. without explicit usage of the
variables themselves, this kernel metho d enables
us to construct dierent nonlinear versions of it
(Aizerman, Braverman, & Rozonoer, 1964 Boser,
Guyon,&V
apnik, 1992). Even though this gen-
eral fact was known (Burges, 1996), the machine
learning community has made little use of it, the
exception being Supp ort Vector machines (Vap-
nik, 1995).
In this pap er, wegive some examples of non-
linear metho ds constructed by this approach. For
one example, the case of a nonlinear form of prin-
cipal component analysis, we shall give details and
experimental results (Sections 2 { 6) for some
other cases, we shall briey sketch the algorithms
(Sec. 7).
In the next section, we will rst review the stan-
dard PCA algorithm. In order to be able to gener-
alize it to the nonlinear case, we shall then formu-
late it in a way which uses exclusively dot pro d-
ucts. In Sec. 3, we shall discuss the kernel metho d
for computing dot products in feature spaces. To-
gether, these two sections form the basis for Sec. 4,
which presents the proposed kernel{based algo-
rithm for nonlinear PCA. Following that, Sec. 5
will discuss some dierences b etween kernel{based
PCA and other generalizations of PCA. In Sec. 6,
we shall give some rst experimental results on
kernel{based feature extraction for pattern recog-
nition. After a discussion of other applications of
the kernel method (Sec. 7), we conclude with a dis-
cussion (Sec. 8). Finally, some technical material
which is not essential for the main thread of the
argument has been relegated into the appendix.
2 PCA in Feature Spaces
Given a set of
M
centered observations
x
k
,
k
=
1
:::M
,
x
k
2
R
N
,
P
M
k
=1
x
k
= 0, PCA diagonal-
izes the covariance matrix
1
C
=
1
M
M
X
j
=1
x
j
x
>
j
:
(1)
To do this, one has to solve the Eigenvalue equa-
tion
v
=
C
v
(2)
for Eigenvalues
0and
v
2
R
N
nf
0
g
.As
C
v
=
1
M
P
M
j
=1
(
x
j
v
)
x
j
, all solutions
v
must lie in the
span of
x
1
:::
x
M
, hence (2) is equivalentto
(
x
k
v
)=(
x
k
C
v
) for all
k
=1
:::M:
(3)
The remainder of this section is devoted to
a straightforward translation to a nonlinear sce-
nario, in order to prepare the ground for the
method proposed in the presentpaper. We shall
now describe this computation in another dot
product space
F
, which is related to the input
space by a p ossibly nonlinear map
:
R
N
!
F
x
7!
X
:
(4)
Note that
F
, whichwe will refer to as the
feature
space
, could have an arbitrarily large, possibly in-
nite, dimensionality. Here and in the following,
upper case characters are used for elements of
F
,
while lower case characters denote elements of
R
N
.
Again, wemake the assumption that we are
dealing with centered data, i.e.
P
M
k
=1
(
x
k
)=0
|we shall return to this point later. Using the
covariance matrix in
F
,
C
=
1
M
M
X
j
=1
(
x
j
)(
x
j
)
>
(5)
1
More precisely,thecovariance matrix is dened as
the expectation of
xx
>
for convenience, we shall use
the same term to refer to the maximum likeliho od
esti-
mate
(1) of the covariance matrix from a nite sample.
2
(if
F
is innite{dimensional, we think of
(
x
j
)(
x
j
)
>
as the linear operator which maps
X
2
F
to (
x
j
)((
x
j
)
X
)) wenowhavetond
Eigenvalues
0 and Eigenvectors
V
2
F
nf
0
g
satisfying
V
=
C
V
(6)
By the same argumentasabove, the solutions
V
lie in the span of (
x
1
)
:::
(
x
M
). For us, this
has two useful consequences: rst, we can consider
the equivalent equation
((
x
k
)
V
) = ((
x
k
)
C
V
)forall
k
=1
:::M
(7)
and second, there exist co ecien
ts
i
(
i
=
1
:::M
) suchthat
V
=
M
X
i
=1
i
(
x
i
)
:
(8)
Combining (7) and (8), weget
M
X
i
=1
i
((
x
k
)
(
x
i
)) =
1
M
M
X
i
=1
i
((
x
k
)
M
X
j
=1
(
x
j
))((
x
j
)
(
x
i
))
for all
k
=1
:::M:
(9)
Dening an
M
M
matrix
K
by
K
ij
:= ((
x
i
)
(
x
j
))
(10)
this reads
MK
=
K
2
(11)
where
denotes the column vector with entries
1
:::
M
.As
K
is symmetric, it has a set of
Eigenvectors which spans the whole space, thus
M
=
K
(12)
gives us all solutions
of Eq. (11). Note that
K
is p ositive semidenite, which can be seen by
noticing that it equals
((
x
1
)
:::
(
x
M
))
>
((
x
1
)
:::
(
x
M
))
(13)
which implies that for all
X
2
F
,
(
X
K
X
)=
k
((
x
1
)
:::
(
x
M
))
X
k
2
0
:
(14)
Consequently,
K
's Eigenvalues will be nonnega-
tive, and will exactly give the solutions
M
of
Eq. (11). We therefore only need to diagonalize
K
.Let
1
2
:::
M
denote the Eigenval-
ues, and
1
:::
M
the corresponding complete
set of Eigenvectors, with
p
being the rst nonzero
Eigenvalue.
2
We normalize
p
:::
M
b
y requir-
ing that the corresp onding vectors in
F
be nor-
malized, i.e.
(
V
k
V
k
) = 1 for all
k
=
p:::M:
(15)
By virtue of (8) and (12), this translates into a
normalization condition for
p
:::
M
:
1 =
M
X
ij
=1
k
i
k
j
((
x
i
)
(
x
j
))
=
M
X
ij
=1
k
i
k
j
K
ij
= (
k
K
k
)
=
k
(
k
k
) (16)
For the purpose of principal comp onent extrac-
tion, we need to compute pro jections on the Eigen-
vectors
V
k
in
F
(
k
=
p:::M
). Let
x
beatest
point, with an image (
x
)in
F
,then
(
V
k
(
x
)) =
M
X
i
=1
k
i
((
x
i
)
(
x
)) (17)
may b e called its nonlinear principal comp onents
corresponding to .
In summary, the following steps were necessary
to compute the principal components: rst, com-
pute the dot product matrix
K
dened by (10)
3
second, compute its Eigenvectors and normalize
them in
F
third, compute pro jections of a test
pointonto the Eigenvectors by(17).
For the sake of simplicity,wehaveabovemade
the assumption that the observations are centered.
This is easy to achieve in input space, but more
dicult in
F
,aswe cannot explicitly compute the
mean of the mapped observations in
F
. There is,
however,away to do it, and this leads to slightly
modied equations for kernel{based PCA (see Ap-
pendix A).
Before we pro ceed to the next section, which
more closely investigates the role of the map ,
the following observation is essential. The map-
ping used in the matrix computation can b e an
arbitrary nonlinear map into the possibly high{
dimensional space
F
. e.g. the space of all
n
th or-
der monomials in the entries of an input vector.
2
If we require that should not map all observa-
tions to zero, then sucha
p
will always exist.
3
Note that in our derivation we could haveused
the known result (e.g. Kirby & Sirovich, 1990) that
PCA can b e carried out on the dot pro duct matrix
(
x
i
x
j
)
ij
instead of (1), however, for the sakeofclarity
and extendability (in Appendix A, we shall consider
the case where the data must b e centered in
F
), we
gave a detailed derivation.
3
In that case, we need to compute dot pro ducts of
input vectors mapped by , with a possibly pro-
hibitive computational cost. The solution to this
problem, which will b e described in the following
section, builds on the fact that we
exclusively
need
to compute dot products b etween mapp ed pat-
terns (in (10) and (17))
we never need the mapped
patterns explicitly.
3 Computing Dot Pro ducts in
Feature Space
In order to compute dot pro ducts of the form
((
x
)
(
y
)), we use kernel representations of the
form
k
(
x
y
)=((
x
)
(
y
))
(18)
which allow us to compute the value of the dot
product in
F
without having to carry out the map
. This metho d was used by Boser, Guyon, &
Vapnik (1992) to extend the \Generalized Por-
trait" hyperplane classier of Vapnik & Chervo-
nenkis (1974) to nonlinear Support Vector ma-
chines. To this end, they substitute a priori chosen
kernel functions for all o ccurances of dot products.
This way, the p owerful results of Vapnik & Cher-
vonenkis (1974) for the Generalized Portrait carry
over to the nonlinear case. Aizerman, Braver-
man & Rozono er (1964) call
F
the \linearization
space", and use it in the context of the poten-
tial function classication metho d to express the
dot pro duct between elements of
F
in terms of ele-
ments of the input space. If
F
is high{dimensional,
wewould like to b e able to nd a closed form ex-
pression for
k
which can be eciently computed.
Aizerman et al. (1964) consider the p ossibilityof
choosing
k
a priori, without being directly con-
cerned with the corresp onding mapping into
F
.
A sp ecic choice of
k
might then corresp ond to a
dot pro duct b etween patterns mapp ed with a suit-
able . A particularly useful example, whichis a
direct generalization of a result proved byPoggio
(1975, Lemma 2.1) in the context of polynomial
approximation, is
(
x
y
)
d
=(
C
d
(
x
)
C
d
(
y
))
(19)
where
C
d
maps
x
to the vector
C
d
(
x
) whose entries
are all p ossible
n
-th degree ordered pro ducts of
the entries of
x
.For instance (Vapnik, 1995), if
x
=(
x
1
x
2
), then
C
2
(
x
)=(
x
2
1
x
2
2
x
1
x
2
x
2
x
1
),
or, yielding the same value of the dot product,
c
2
(
x
)=(
x
2
1
x
2
2
p
2
x
1
x
2
)
:
(20)
For this example, it is easy to verify that
;
(
x
1
x
2
)(
y
1
y
2
)
>
2
= (
x
2
1
x
2
2
p
2
x
1
x
2
)(
y
2
1
y
2
2
p
2
y
1
y
2
)
>
=
c
2
(
x
)
c
2
(
y
)
>
:
(21)
In general, the function
k
(
x
y
)=(
x
y
)
d
(22)
corresponds to a dot pro duct in the space of
d
-th order monomials of the input coordinates.
If
x
represents an image with the entries b eing
pixel values, we can thus easily work in the space
spanned by pro ducts of any
d
pixels | provided
that we are able to do our work solely in terms
of dot products, without any explicit usage of a
mapped pattern
c
d
(
x
). The latter lives in a pos-
sibly very high{dimensional space: even though
we will identify terms like
x
1
x
2
and
x
2
x
1
into one
coordinate of
F
as in (20), the dimensionalityof
F
, the image of
R
N
under
c
d
, still is
(
N
+
p
;
1)!
p
!(
N
;
1)!
and
thus grows like
N
p
.For instance, 16
16 input im-
ages and a polynomial degree
d
= 5 yield a dimen-
sionalityof10
10
.Thus, using kernels of the form
(22) is our only waytotakeinto account higher{
order statistics without a combinatorial explosion
of time complexity.
The general question which function
k
corre-
sponds to a dot pro duct in some space
F
has
been discussed by Boser, Guyon, & Vapnik (1992)
and Vapnik (1995): Mercer's theorem of functional
analysis states that if
k
is a continuous kernel of
a p ositiveintegral operator, we can construct a
mapping into a space where
k
acts as a dot prod-
uct (for details, see App endix B.
The application of (18) to our problem is
straightforward: we simply substitute an a priori
chosen kernel function
k
(
x
y
) for all occurances
of ((
x
)
(
y
)). This was the reason whywe had
to formulate the problem in Sec. 2 in a waywhich
only makes use of the values of dot products in
F
. The choice of
k
then
implicitly
determines the
mapping and the feature space
F
.
In App endix B, we give some examples of ker-
nels other than (22) whichmay b e used.
4 Kernel PCA
4.1 The Algorithm
To perform kernel{based PCA (Fig. 1), from now
on referred to as
kernel PCA
, the following steps
have to be carried out: rst, we compute the dot
product matrix (cf. Eq. (10))
K
ij
=(
k
(
x
i
x
j
))
ij
:
(23)
Next, we solve(12)by diagonalizing
K
, and nor-
malize the Eigenvector expansion co ecients
n
by requiring Eq. (16),
1=
n
(
n
n
)
:
4
R
2
l
inear PC
A
R
2
F
=R
2
0
Φ
k
ernel PC
A
k
Figure 1: The basic idea of kernel PCA. In some high{
dimensional feature space
F
(bottom right), weare
performing linear PCA, just as a PCA in input space
(top). Since
F
is nonlinearly related to input space
(via ), the contour lines of constant pro jections onto
the principal Eigenvector (drawn as an arrow) b ecome
nonlinear
in input space. Note that we cannot draw
a pre{image of the Eigenvector in input space, as it
may not even exist. Crucial to kernel PCA is the fact
that we do not actually perform the map into
F
,but
instead p erform all necessary computations by the use
ofakernel function
k
in input space (here:
R
2
).
To extract the principal comp onents (correspond-
ing to the kernel
k
) of a test p oint
x
,we then
compute pro jections onto the Eigenvectors by (cf.
Eq. (17)),
(
k
PC)
n
(
x
)=(
V
n
(
x
)) =
M
X
i
=1
n
i
k
(
x
i
x
)
:
(24)
If we use a kernel as described in Sec. 3, we
know that this procedure exactly corresp onds to
standard PCA in some high{dimensional feature
space, except that we do not need to p erform ex-
pensiv
e computations in that space.
4.2 Properties of (Kernel{) PCA
If we use a kernel which satises the conditions
given in Sec. 3, we know that we are in fact doing
a standard PCA in
F
. Consequently, all math-
ematical and statistical prop erties of PCA (see
for instance Jollie, 1986) carry over to kernel{
based PCA, with the modications that they be-
come statements about a set of points (
x
i
)
i
=
1
:::M
,in
F
rather than in
R
N
.In
F
,we can
thus assert that PCA is the orthogonal basis trans-
formation with the following properties (assuming
that the Eigenvectors are sorted in ascending order
of the Eigenvalue size):
the rst
q
(
q
2f
1
:::M
g
) principal comp o-
nents, i.e. pro jections on Eigenvectors, carry
more variance than any other
q
orthogonal
directions
the mean{squared approximation error in
representing the observations by the rst
q
principal comp onents is minimal
the principal components are uncorrelated
the representation entropy is minimized
the rst
q
principal components havemaxi-
mal mutual information with respect to the
inputs
For more details, see Diamantaras & Kung (1996).
To translate these properties of PCA in
F
into
statements ab out the data in input space, they
need to b e investigated for sp ecic choices of a
kernels. Weshallnotgointo detail on that mat-
ter, but rather proceed in our discussion of kernel
PCA.
4.3 Dimensionality Reduction and
Feature Extraction
Unlike linear PCA, the prop osed metho d allows
the extraction of a number of principal compo-
nents which
can
exceed the input dimensionality.
Suppose that the number of observations
M
ex-
ceeds the input dimensionality
N
. Linear PCA,
even when it is based on the
M
M
dot product
matrix, can nd at most
N
nonzero Eigenvalues
| they are identical to the nonzero Eigenvalues
of the
N
N
covariance matrix. In contrast, ker-
nel PCA can nd up to
M
nonzero Eigenvalues
4
| a fact that illustrates that it is imp ossible to
perform kernel PCA based on an
N
N
covari-
ance matrix.
4.4 Computational Complexity
As mentioned in Sec. 3, a fth order p olynomial
kernel on a 256{dimensional input space yields a
10
10
{dimensional space. It would seem that lo ok-
ing for principal components in his space should
pose intractable computational problems. How-
ever, as wehave explained ab ove, this is not the
case. First, as pointed out in Sect. 2 wedo not
need to look for Eigenvectors in the full space
F
,
but just in the subspace spanned by the images
of our observations
x
k
in
F
. Second, wedo not
need to compute dot products explicitly b etween
vectors in
F
,as we know that in our case this can
be done directly in the input space, using kernel
4
If we use one kernel | of course, we could extract
features with several kernels, to get even more.
5