ClassCut for Unsupervised Class Segmentation
Bogdan Alexe, Thomas Deselaers, and Vittorio Ferrari
Computer Vision Laboratory, ETH Zurich, Zurich, Switzerland
{bogdan,deselaers,ferrari}@vision.ee.ethz.ch
Abstract. We propose a novel method for unsupervised class segmen-
tation on a set of images. It alternates between segmenting object in-
stances and learning a class model. The method is based on a segmen-
tation energy defined over all images at the same time, which can be
optimized efficiently by techniques used before in interactive segmenta-
tion. Over iterations, our method progressively learns a class model by
integrating observations over all images. In addition to appearance, this
model captures the location and shape of the class with respect to an
automatically determined coordinate frame common across images. This
frame allows us to build stronger shape and location models, similar to
those used in object class detection. Our method is inspired by inter-
active segmentation methods [1], but it is fully automatic and learns
models characteristic for the object class rather than specific to one par-
ticular object/image. We experimentally demonstrate on the Caltech4,
Caltech101, and Weizmann horses datasets that our method (a) trans-
fers class knowledge across images and this improves results compared
to segmenting every image independently; (b) outperforms Grabcut [1]
for the task of unsupervised segmentation; (c) offers competitive per-
formance compared to the state-of-the-art in unsupervised segmentation
and in particular it outperforms the topic model [2].
1 Introduction
Image segmentation is a fundamental problem in computer vision. Over the past
years methods that use graph-cut to minimize binary pairwise energy functions
have become the de-facto standard for segmenting specific objects in individual
images [1, 3, 4]. These methods employ appearance models for the foreground
and background which are estimated through user interactions [1, 3, 4].
On the one hand, analog approaches have been presented for object class
segmentation where the appearance models are learned from a set of training
images with ground-truth segmentations [5–7]. However, obtaining ground-truth
segmentations is cumbersome and error-prone.
On the other hand, approaches to unsupervised class segmentation have also
been proposed [2, 8–10, 12, 13]. In unsupervised segmentation a set of images
depicting different instances of an object class is given, but without information
about the appearance and shape of the objects to be segmented. The aim of an
algorithm is to automatically segment the object instance in each image.
K. Daniilidis, P. Maragos, N. Paragios (Eds.): ECCV 2010, Part V, LNCS 6315, pp. 380–393, 2010.
c
Springer-Verlag Berlin Heidelberg 2010
ClassCut for Unsupervised Class Segmentation 381
Interestingly, most previous approaches to unsupervised segmentation do not
use energy functions similar to those in interactive and supervised segmentation,
but instead use topic models [2] or other specialized generative models [10, 12]
to find recurring patterns in the images.
We propose ClassCut, a novel method for unsupervised segmentation based on
a binary pairwise energy function similar to those used in interactive/supervised
segmentation. As opposed to those, our energy function is defined over a set
of images rather than on one image [1, 3–5]. Inspired by GrabCut [1], where
the two stages of learning the foreground/background appearance models and
segmenting the image are alternated, our method alternates between learning a
class model and segmenting the objects in all images jointly. The class model is
learned from all images at the same time, so as to capture knowledge about the
class rather than specific to one image [1]. Therefore, it helps the next segmen-
tation iteration, as it transfers between images knowledge about the appearance
and shape of the class. Thanks to the nature of our energy function, we can
segment all images jointly using existing efficient algorithms used in interactive
segmentation approaches [1, 3, 14, 15].
Inspired by representations successfully used in supervised object class detec-
tion [16, 17], our approach anchors the object class in a reference coordinate
frame common across images. This enables modeling the spatial structure and
shape of the class, as well as designing novel priors tailored to the unsupervised
segmentation task. We determine this reference frame automatically in every
image with a procedure based on a salient object detector [18].
At each iteration ClassCut updates the class model, which captures the ap-
pearance, shape, and location distribution of the class within the reference frame.
The final output of the method are images with segmented object instances as
well as the class model.
In the experiments, we demonstrate that our method (a) transfers knowledge
between images and this improves the performance over segmenting each image
independently; (b) outperforms the original GrabCut [1], which is the main inspi-
ration behind it and turns out to be a very competitive baseline for unsupervised
segmentation; (c) offers competitive performance compared to the state-of-the-
art in unsupervised segmentation; (d) learns meaningful, intuitive class models.
Source code for ClassCut is available at http://www.vision.ee.ethz.ch/˜calvin.
Related Work. We discussed in the introduction that our method employs
energy minimization techniques used in interactive segmentation [1, 3, 4, 14, 15],
and how it is related to supervised [5, 7, 19] as well as to unsupervised [2, 10–12]
class segmentation methods.
A different task is object discovery, which aims at finding multiple object
classes from a mixed set of unlabeled images [11, 29]. In our work instead, all
images contain instances of one class.
The two closest work to ours are [8, 9], which have a procedure iterating be-
tween updating a model and segmenting the images. In [8] the model is given
a set of class and non-class images and then it iteratively improves the fore-
ground/background labeling of image fragments based on their class likelihoods.
382 B. Alexe, T. Deselaers, and V. Ferrari
Their method learns local segmentations masks for image fragments, while our
method learns a more complete class model, including appearance, shape and
location in a global reference frame.
Arora et al. [9] learn a template consistent over all images using variational
inference. Their template model is very different from our class model, and closer
to a constellation model [20]. Moreover, their method optimizes the segmentation
of the images individually rather than jointly.
Finally, our approach is also related to co-segmentation [21] where the goal is
to segment a specific object from two images at the same time. Here we try to go
a step further and co-segment a set of images showing different object instances
of an unknown class.
2OverviewofOurMethod
The goal is to jointly segment objects of an unknown class from a set of images.
Analog to the scheme of GrabCut [1], ClassCut alternates two stages: (1) learn-
ing/updating a class model given the current segmentations (sec. 4); (2) jointly
segmenting the objects in all images given the current class model (sec. 3). It
converges when the segmentation is unchanged in two consecutive iterations.
Our segmentation model for stage (2) is a binary pairwise energy function,
which can be optimized efficiently by techniques used in interactive segmenta-
tion [1, 3, 22], but jointly over all images rather than on a single image [1]
(sec. 3).
In stage (1), learning the class model over all images at once enables cap-
turing knowledge characteristic for the class rather than specific to a particular
image [1]. As the class model is used in the next segmentation iteration it trans-
fers knowledge across images, typically from easier images to more difficult ones,
aiding their segmentation. For example, the model might learn in the first itera-
tion that airplanes are typically grayish and the background is often blue (fig. 1).
In the next iteration, this will help in images where the airplane is difficult to
segment (e.g. because of low contrast).
The class model we propose (sec. 3.2) consists of several components modeling
different class characteristics: appearance, location, and shape. In addition to a
color component also used in GrabCut [1], the appearance model includes a bag-
of-words [23] of SURF descriptors [24], which is well suited for modeling class
appearance. Moreover, we model the location (sec. 3.2) and shape (sec. 3.2)
of the object class w.r.t. a reference coordinate frame common across images
(sec. 5). Overall, our model focuses on knowledge at the class level rather than
at the level of one object as in the works it is inspired from [1, 4].
In addition to the class model, the segmentation energy include priors tailored
for segmenting classes (sec. 3.1). The priors are defined on superpixels [25], which
act as grouping units for homogeneous areas. Superpixels bring two advantages:
(i) they provide additional structure, i.e. the set of possible segmentations is
reduced to those aligning well with image boundaries; (ii) they reduce the com-
putational complexity of segmentation. We formulate four class segmentation
priors over superpixels and multiple images (sec. 3.1).
ClassCut for Unsupervised Class Segmentation 383
Fig. 1. Overview of our method. The top row shows the input images, the auto-
matically determine reference frames and the initial location and shape models. The
bottom row shows how the segmentations evolve over the iterations as well as the final
location and shape models.
If a common reference frame on the objects is available, our method exploits
it to anchor the location and shape models to it and to improve the effectiveness
of some of the priors. We apply a salient object detector [18] to determine this
reference frame automatically (sec. 5). In sec. 6 we show how this detector im-
proves segmentation results compared to using the whole image as a reference
frame. Fig. 1 shows an overview of the entire method.
3Segmentation
In the set of images I = {I
1
,...,I
N
} each image I
n
(given either as a full
image or as automatically determined reference frame) consists of superpixels
{S
1
n
,...,S
K
n
n
}.WesearchforthelabelingL
∗
=
(l
1
1
,...,l
K
1
1
),...,(l
1
n
,...,l
K
n
n
),
...,(l
1
N
,...,l
K
N
N
)
that sets l
k
n
= 1 for all superpixels S
k
n
on the foreground and
l
j
n
= 0 for all superpixels S
j
n
on the background.
To determine L
∗
, we minimize
L
∗
=argmin
L
{E
Θ
(L, I)} with E
Θ
(L, I)=Φ
Θ
(L, I)+Ψ
Θ
(L, I)(1)
where Φ is the segmentation prior (sec. 3.1) and Ψ is the class model (sec. 3.2).
In sec. 3.3 we describe how to minimize eq. (1). Θ are the parameters of the
model.
3.1 Prior Φ
Θ
(L, I )
The prior Φ consists of four terms
Φ
Θ
(L, I)=w
Λ
Λ(L, I)+w
χ
χ(L, I)+w
Γ
Γ (L, I)+w
Δ
Δ(L, I)(2)
The scalars w are part of the model parameters Θ and weight the terms. Below
we describe the terms in detail.
384 B. Alexe, T. Deselaers, and V. Ferrari
(a)
(b)
(c)
Fig. 2. Priors. (a) The smoothness prior between two superpixels is weighted inversely
to the sum over the gradients along their boundary (shown in yellow and blue for
two pairs of superpixels). (b) The between image smoothness prior is weighted by
the overlap (yellow) of superpixels (shown for two pairs of superpixels (red/green) in
two images). (c) The border penalty assigns high values to superpixels touching the
reference frame boundary (dark=low values, bright=high values).
The Within Image Smoothness Λ is a smoothness prior for superpixels
which generalizes the pixel-based smoothness priors typically used in interactive
segmentation [1]. It penalizes neighboring superpixels having different labels.
Λ(L, I)=
n
j,k
δ(l
j
n
= l
k
n
)exp(−grad(S
j
n
,S
k
n
)) (3)
where j, k are the indices of neighboring superpixels S
j
n
, S
k
n
within image I
n
.
δ(l
j
n
= l
k
n
) = 1 if the labels l
j
n
, l
k
n
are different and 0 otherwise. The gradient
grad(S
j
n
,S
k
n
) between S
j
n
and S
k
n
is computed by summing the gradient mag-
nitudes [26] along the boundary between S
j
n
, S
k
n
(fig. 2a) normalized w.r.t. the
length of the boundary. Thus, the penalty is smaller if the two superpixels are
separated by high gradients. This term encourages segmentations aligned with
the image gradients.
The Between Image Smoothness χ operates on superpixels across images.
It encourages superpixels in different images but with similar location w.r.t. the
reference frame to have the same label:
χ(L, I)=
n,m
j,k
δ(l
j
n
= l
k
m
)
|S
j
n
∩ S
k
m
|
|S
j
n
∪ S
k
m
|
(4)
where n, m are two images and j, k superpixels, one in I
n
, the other in I
m
.
This penalty grows with the overlap of the superpixels (measured as area of
intersection over area of union). Therefore only overlapping superpixels interact
(fig. 2b). This term encourages similar segmentations across all images (w.r.t.
the reference frame).
The Border Penalty Γ prefers superpixels at the image boundary to be
labeled background. Objects rarely touch the boundary of the reference frame.
Notice how the object would touch even a tight bounding-box around itself only
in a few points (e.g. fig. 2a). The border penalty
Γ (L, I)=
n
k
l
k
n
border(S
k
n
)
perimeter(S
k
n
)
(5)