Image Classification using Super-Vector Coding
of Local Image Descriptors
Xi Zhou
†
, Kai Yu
‡
, Tong Zhang
∗
, and Thomas S. Huang
†
†
Dept. of ECE, University of Illinois at Urbana-Champaign, Illinois
‡
NEC Laboratories America, California
∗
Department of Statistics, Rutgers University, New Jersey
Abstract. This paper introduces a new framework for image classifi-
cation using local visual descriptors. The pipeline first performs a non-
linear feature transformation on descriptors, then aggregates the results
together to form image-level representations, and finally applies a clas-
sification model. For all the three steps we suggest novel solutions which
make our approach appealing in theory, more scalable in computation,
and transparent in classification. Our experiments demonstrate that the
proposed classification method achieves state-of-the-art accuracy on the
well-known PASCAL benchmarks.
1 Introduction
Image classification, including object recognition and scene classification, re-
mains to be a major challenge to the computer vision community. Perhaps one
of the most significant developments in the last decade is the application of lo-
cal features to image classification, including the introduction of “bag-of-visual-
words” representation that inspires and initiates a lot of research efforts [1].
A large body of work investigates probabilistic generative models, with the
objective towards understanding the semantic content of images. Typically those
models extend the famous topic models on bag-of-word representation by further
considering the spatial information of visual words [2][3].
This paper follows another line of research on building discriminative models
for classification. The previous work includes SVMs using pyramid matching ker-
nels [4], biologically-inspired models [5][6], and KNN methods [7][8][9]. Over the
past years, the nonlinear SVM method using spatial pyramid matching (SPM)
kernels [4][10] seems to be dominant among the top performers in various im-
age classification benchmarks, including Caltech-101 [11], PASCAL [12], and
TRECVID. The recent improvements were often achieved by combining differ-
ent types of local descriptors [10][13][14], without any fundamental change of the
underlying classification method. In addition to the demand for more accurate
classifiers, one has to develop more practical methods. Nonlinear SVMs scale
at least quadratically to the size of training data, which makes it nontrivial to
handle large-scale training data. It is thus necessary to design algorithms that
are computationally more efficient.
2 ECCV-10 submission ID 453
1.1 Overview of Our Approach
Our work represents each image by a set of local descriptors with their spatial
coordinates. The descriptor can be SIFT, or any other local features, computed
from image patches at locations on a 2D grid. Our image classification method
consists of three computational steps:
1. Descriptor coding:
Each descriptor of an image is nonlinearly mapped to form a high-dimensional
sparse vector. We propose a novel nonlinear coding method called Super-
Vector coding, which is algorithmically a simple extension of Vector Quan-
tization (VQ) coding;
2. Spatial pooling:
For each local region, the codes of all the descriptors in it are aggregated
to form a single vector, then vectors of different regions are concatenated to
form the image-level feature vector. Our pooling is base on a novel proba-
bility kernel incorporating the similarity metric of local descriptors;
3. Image classification:
The image-level feature vector is normalized and fed into a classifier. We
choose linear SVMs, which scale linearly to the size of training data.
We note that the coding-pooling-classification pipeline is the de facto frame-
work for image scene classification. One notable example is the SPM kernel ap-
proach [4], which applies average pooling on top of VQ coding, plus a nonlinear
SVM classifier using Chi-square or intersection kernels.
In this paper, we propose novel methods for each of the three steps and
formalize their underlying mathematical principles. The work stresses the im-
portance of learning good coding of local descriptors in the context of image
classification, and makes the first attempt to formally incorporate the metric of
local descriptors into distribution kernels. Putting all these together, the over-
all image classification framework enjoys a linear training complexity, and also
a great interpretability that is missing in conventional models (see details in
Sec. 2.3). The most importantly, our method demonstrates state-of-the-art per-
formances on the challenging PASCAL07 and PASCAL09 image classification
benchmarks.
2 The Method
In the following we will describe all the three steps of our image classification
pipeline in detail.
2.1 Descriptor Coding
We introduce a novel coding method, which enjoys appealing theoretical prop-
erties. Suppose we are interested in learning a smooth nonlinear function f(x)
ECCV-10 submission ID 453 3
defined on a high dimensional space R
d
. The question is, how to derive a good
coding scheme (or nonlinear mapping) φ(x) such that f(x) can be well approxi-
mated by a linear function on it, namely w
>
φ(x). Our only assumption here is
that f(x) should be sufficiently smooth.
Let us consider a general unsupervised learning setting, where a set of bases
C ⊂ R
d
, called codebook or dictionary, is employed to approximate any x,
namely,
x ≈
X
v∈C
γ
v
(x)v,
where γ(x) = [γ
v
(x)]
v∈C
is the coefficients, and sometimes
P
v
γ
v
(x) = 1. By
restricting the cardinality of nonzeros of γ(x) to be 1 and γ
v
(x) ≥ 0, we obtain
the Vector Quantization (VQ) method
v
∗
(x) = arg min
v∈C
kx − vk,
where k · k is the Euclidean norm (2-norm). The VQ method uses the coding
γ
v
(x) = 1 if v = v
∗
(x) and γ
v
(x) = 0 otherwise. We say that f(x) is β Lipschitz
derivative smooth if for all x, x
0
∈ R
d
:
|f(x) − f(x
0
) − ∇f(x
0
)
>
(x − x
0
)| ≤
β
2
kx − x
0
k
2
.
It immediately implies the following simple function approximation bound via
VQ coding: for all x ∈ R
d
:
f(x) − f
v
∗
(x)
− ∇f
v
∗
(x)
>
x − v
∗
(x)
≤
β
2
kx − v
∗
(x)k
2
. (1)
This bounds simply states that one can approximate f (x) by f
v
∗
(x)
+
∇f
v
∗
(x)
>
x − v
∗
(x)
, and the approximation error is upper bounded by the
quality of VQ. It further suggests that the function approximation can be im-
proved by learning the codebook C to minimize this upper bound. One way is
the K-means algorithm
C = arg min
C
(
X
x
min
v∈C
kx − vk
2
)
.
Eq. (1) also suggests that the approximation to f(x) can be expressed as a linear
function on a nonlinear coding scheme
f(x) ≈ g(x) ≡ w
>
φ(x),
where φ(x) is called the Super-Vector (SV) coding of x, defined by
φ(x) =
sγ
v
(x), γ
v
(x)(x − v)
>
>
v∈C
(2)
where s is a nonnegative constant. It is not difficult to see that w = [
1
s
f(v), ∇f(v)]
v∈C
,
which can be regarded as unknown parameters to be estimated. Because γ
v
(x) =
4 ECCV-10 submission ID 453
1 if v = v
∗
(x), otherwise γ
v
(x) = 0, the obtained φ(x) a is highly sparse represen-
tation, with dimensionality |C|(d+1). For example, if |C| = 3 and γ(x) = [0, 1, 0],
then
φ(x) =
0, . . . , 0
| {z }
d+1 dim.
, s, (x − v)
>
| {z }
d+1 dim.
, 0, . . . , 0
| {z }
d+1 dim.
>
(3)
(1) (2) (3)
Fig. 1. Function f (x) approximated by w
>
φ(x)
As illustrated in Figure 1, w
>
φ(x) provides a piece-wise linear function to
approximate a nonlinear function f(x), as shown in Figure 1-(2), while with
VQ coding φ(x) = [γ
v
(x)]
>
v∈C
, the same formulation w
>
φ(x) gives a piece-wise
constant approximation, as shown in Figure 1-(3). This intuitively suggests that
SV coding may achieve a lower function approximation error than VQ coding.
We note that the popular bag-of-features image classification method essentially
employs VQ to obtain histogram representations. The proposed SV coding is a
simple extension of VQ, and may lead to a better approach to image classifica-
tion.
2.2 Spatial Pooling
Pooling Let each image be represented as a set of descriptor vectors x that fol-
lows an image-specific distribution, represented as a probability density function
p(x) with respect to an image independent back-ground measure dµ(x). Let’s
first ignore the spacial locations of x, and address the spacial pooling later. A
kernel-based method for image classification is based on a kernel on the proba-
bility distributions over x ∈ Ω, K : P × P 7→ R. A well-known example is the
Bhattacharyya kernel [15]:
K
b
(p, q) =
Z
Ω
p(x)
1
2
q(x)
1
2
dµ(x).
Here p(x) and q(x) represent two images as distributions over local descriptor
vectors, and µ(x) is the image independent background measure. Bhattacharyya
ECCV-10 submission ID 453 5
kernel is closely associated with Hellinger distance, defined as D
h
(p, q) = 2 −
K
b
(p, q), which can be seen as a principled symmetric approximation of the
Kullback Leibler (KL) divergence [15]. Despite the popular application of both
Bhattacharyya kernel and KL divergence, a significant drawback is the ignorance
of the underlying similarity metric of x, as illustrated in Figure 2. In order to
avoid this problem, one has to work with very smooth distribution families that
are inconvenient to work with in practice. In this paper, we propose a novel
formulation that explicitly takes the similarity of x into account:
K
s
(p, q) =
Z
Ω
Z
Ω
p(x)
1
2
q(x
0
)
1
2
κ(x, x
0
)dµ(x)dµ(x
0
)
=
Z
Ω
Z
Ω
p(x)
−
1
2
q(x
0
)
−
1
2
κ(x, x
0
)p(x)q(x
0
)dµ(x)dµ(x
0
)
where κ(x, x
0
) is a RKHS kernel on Ω that reflects the similarity structure of x.
In the extreme case where κ(x, x
0
) = δ(x −x
0
) is the delta-function with respect
to µ(·), then the above kernel reduces to the Bhattacharyya kernel.
(1) (2)
Fig. 2. Illustration of the drawback of Bhattacharyya kernel: in both cases their
density kernels K
b
(p, q) remain to be the same, equal to 0.
In reality we cannot directly observe p(x) from any image, but a set X of
local descriptors. Therefore, based on the empirical approximation to K
s
(p, q),
we define a kernel between sets of vectors:
K(X, X
0
) =
1
NN
0
X
x∈X
X
x
0
∈X
0
p(x)
−
1
2
q(x
0
)
−
1
2
κ(x, x
0
) (4)
where N and N
0
are the sizes of the descriptor sets from two images.
Let κ(x, x
0
) = hφ(x), φ(x
0
)i, where φ(x) is the SV coding defined in the
previous section. It is easy to see that κ(x, x
0
) = 0 if x and x
0
fall into different
clusters. Then we have
K(X, X
0
) =
1
NN
0
|C|
X
k=1
X
x∈X
k
X
x
0
∈X
0
k
p(x)
−
1
2
q(x
0
)
−
1
2
κ(x, x
0
)
