The International Journal of Computer Vision, Kluwer 2005. To appear.
A Multiphase Dynamic Labeling Model for
Variational Recognition-driven Image Segmentation
Daniel Cremers
Department of Computer Science
University of California, Los Angeles, USA
http://www.cs.ucla.edu/∼cremers
Nir Sochen
Department of Applied Mathematics
Tel Aviv University, Israel
http://www.math.tau.ac.il/∼sochen
Christoph Schn¨orr
Department of Mathematics and Computer Science
University of Mannheim, Germany
http://www.cvgpr.uni-mannheim.de
Abstract. We propose a variational framework for the integration of multiple
comp eting shape priors into level set based segmentation schemes. By optimizing
an appropriate cost functional with respect to both a level set function and a
(vector-valued) lab eling function, we jointly generate a segmentation (by the level set
function) and a recognition-driven partition of the image domain (by the labeling
function) which indicates where to enforce certain shape priors. Our framework
fundamentally extends previous work on shape priors in level set segmentation by
directly addressing the central question of where to apply which prior. It allows
for the seamless integration of numerous shape priors such that – while segmenting
both multiple known and unknown objects – the level set process may selectively use
specific shape knowledge for simultaneously enhancing segmentation and recognizing
shape.
Keywords: Image segmentation, shape priors, variational methods, level set meth-
ods, dynamic labeling
1. Introduction
Image segmentation and object recognition in vision are driven both
by low-level cues such as intensities, color or te xture properties, and
by prior knowledge about objects in our environment. Modeling the
interaction between data-driven and model-based processes has become
the focus of current research on image segmentation in the field of
computer vision. In this work, we consider prior knowledge given by
the shapes associated with a set of objects and focus on the problem of
how to exploit such knowledge for images containing multiple objects,
some of which may be familiar, while others may be unfamiliar.
c
2005 Kluwer Academic Publishers. Printed in the Netherlands.
recognition_ijcv.tex; 30/03/2005; 12:07; p.1
2
Following their introduction as a means of front propagation [24]
1
,
level set based contour representations have become a popular frame-
work for image segmentation [2, 22]. They permit to elegantly model
topological changes of the implicitly represented boundary, which make s
them well suited for segmenting image s containing multiple objects.
Level set s egm entation schemes can be formulated to exploit various
low level image properties such as edge information [22, 3, 20], intensity
homogeneity [5, 33], texture [25, 30, 19, 1] or motion information [13].
In recent years, there has been much effort in trying to integrate
prior shape knowledge into level set based segmentation. This was
shown to make the segmentation process robust to misleading low-level
information caused by noise, background clutter or partial occlusion of
an object of interest (cf. [21, 32, 7, 27]).
All of these approaches were designed to segment a single known
object in a given image. Yet, in general a given image will contain
several familiar or unfamiliar objects. A key problem in this context is
therefore to ensure that prior knowledge is selectively applied at image
locations only where image data indicate a familiar object. Conversely,
lack of any evidence for the presence of some familiar object should
result in a purely data-driven segmentation process.
Clearly, any use of shape priors consistent with the philosophy of
the level set method should retain the capacity of the resulting seg-
mentation scheme to deal with multiple independent objects, no matter
whether they are familiar or not. One may instead suggest to iteratively
apply the segmentation scheme with a different prior at each time
and thereby successively segment the respective objects. We believe,
however, that such a sequential processing mode will not scale up to
large databases of objects and that – even more importantly – the
parallel use of competing priors is essential for modeling the chicken-egg
relationship between segmentation and recognition.
In this paper, we propose a variational framework for image seg-
mentation which allows the integration of multiple competing shape
priors into a segmentation process which can simultaneously handle
multiple known and unknown objects in a given im age. To this end, we
propose to introduce a labeling or decision function in order to restrict
the effect of given priors to specific domains of the image plane. Learnt
shape information is thereby applied in regions where the image data
indicates the presence of a familiar object. For a recent variant of the
labeling approach, we refer to [4]. During optimization, this labeling
function evolves so as to select image regions where given shape models
are applied. The resulting process segments scenes containing corrupted
1
Precursors containing key ideas of the level set method appeared in [17, 18].
recognition_ijcv.tex; 30/03/2005; 12:07; p.2
3
versions of known objects in a way which does not affect the correct
segmentation of other unfamiliar objects. A smoothness constraint on
the labeling function induces the process to distinguish between occlu-
sions (which are close to the familiar object) and separate independent
2
objects (assumed to be sufficiently far from the object of interest).
In this work, the term shape prior refers to fixed templates with
variable 2D pose and location. However, the proposed framework of
selective shape priors c ould be extended to statistical shape models
which would additionally allow certain deformation modes of each tem-
plate. For promising advances regarding level set based statistical shape
representations, we refer to [6].
This paper comprises and extends work which was presented on two
conferences [14, 15]. The outline of the paper is as follows: In Section
2, we briefly review the level set formulation of the piecewise constant
Mumford-Shah functional proposed in [5]. In Section 3, we augment this
variational framework by a labeling function which selectively imposes
a single shape prior in certain image regions. In Section 4, we enhance
this prior by explicit transformation parameters for pose and location
and demonstrate the effect of simultaneous optimization of pose and
location in an image for which the exact transformation parameters of
the familiar object are unknown. The resulting segmentation process
not only selects appropriate regions where to apply the prior, it also
selects appropriate pose and translation parameters associated with a
given prior. In Section 5, we extend the labeling approach from the case
of one known object and background to that of two independent know n
objects. In Section 6, we introduce the concept of multiphase dynamic
labeling which allows the generalization of the labeling approach to an
arbitrary number of known and unknown objects by means of a vector-
valued labeling function. In Section 7, we derive the gradient descent
equations which minimize the proposed functional. In subsequent sec-
tions we present numerical results to illuminate various properties of
our approach: We demonstrate that the segmentation scheme is capable
of reconstructing corrupted versions of multiple known objects dis-
played in a scene containing other unknown objects. The segmentation
of multiple partially occluded objects moving independently in image
sequences illuminates how the evolution of the labeling or decision func-
tions is driven by the input data. This evolution can be interpreted in
the sense that different shape models compete for areas of influence. In
the context of mutual occlusion, we show that the segmentation process
2
In this paper, we assume that objects share the same Gaussian intensity model,
by “independent” we mean that their pose and location are independent. Extensions
which allow each object to have its own intensity model are conceivable but they
are be yond the scope of this paper.
recognition_ijcv.tex; 30/03/2005; 12:07; p.3
4
is forced to decide for one or the other shape model. The experimental
results demonstrate that our variational framework couples the input
intensity data, the shape models and the labeling or decision functions
in a recognition-driven segmentation process. We end with a discussion
of limitations and open problems.
2. Data-Driven Level Set Segmentation
Level set representations of moving interfaces [24, 17] have become a
popular framework for image segmentation. A contour C is represe nted
as the zero level set function φ : Ω → IR on the image domain Ω ⊂ IR
2
:
C = {x ∈ Ω | φ(x) = 0}. (1)
During the segmentation process, this contour is propagated implicitly
by evolving the embedding function φ. In contrast to explicit parame-
terizations, one avoids the issues of control point regridding. Moreover,
the implicitly represented contour can undergo topological changes such
as splitting and merging during the evolution of the embedding func-
tion. This makes the level set formalism well suited for the segmentation
of multiple objects. In this work, we revert to a region-based level set
scheme introduced by Chan and Vese [5]. However, other data-driven
level set schemes could be employed.
In [5] Chan and Vese introduce a level set formulation of the piece-
wise constant Mumford-Shah functional [23]. In particular, they pro-
pose to generate a segmentation of an input image f(x) with two gray
value constants µ
1
and µ
2
by minimizing the functional
E
CV
({µ
i
}, φ) =
Z
Ω
(f − µ
1
)
2
Hφ + (f − µ
2
)
2
(1−Hφ)dx +ν
Z
Ω
|∇Hφ|, (2)
with respect to the scalar variables µ
1
and µ
2
and the embedding level
set function φ. Here H denotes the Heaviside function
Hφ ≡ H(φ(x)) =
1, φ(x) ≥ 0
0, else
. (3)
The last term in (2) measures the length of the zero-crossing of φ.
The Euler-Lagrange equation for this functional is implemented by
gradient descent:
∂φ
∂t
= δ(φ)
ν div
∇φ
|∇φ|
− (f − µ
1
)
2
+ (f − µ
2
)
2
, (4)
recognition_ijcv.tex; 30/03/2005; 12:07; p.4
5
Figure 1. Purely intensity-based segmentation. Contour evolution generated
by minimizing the Chan-Vese model (2) [5]. The central figure is partially corrupted,
i.e. one leg and two arms are missing.
where µ
1
and µ
2
are updated in alternation with the level set evolution
to take on the mean gray value of the input image f in the regions
defined by φ > 0 and φ < 0, respectively:
µ
1
=
R
f(x)Hφdx
R
Hφdx
, µ
2
=
R
f(x)(1 − Hφ)dx
R
(1 − Hφ)dx
. (5)
Figure 1 shows a representative contour evolution obtained for an
image containing three figures, the middle one being partially cor-
rupted.
3. Selective Shape Priors by Dynamic Labeling
The evolution in Figure 1 demonstrates the well-known fact that the
level set based segmentation process can cope with multiple objects
in a given scene. However, if the low-level segmentation criterion is
violated due to unfavorable lighting conditions, background clutter or
partial occlusion of the objects of interest, then the purely image-based
segmentation scheme will fail to converge to the desired segmentation
(see Figure 8, top row).
To cope with such degraded low-level information, it was proposed
to introduce prior shape knowledge into the level set scheme (cf. [21,
32, 27]). The basic idea is to extend the image- based cost functional by
a shape energy which favors certain contour formations:
E
total
(φ) = E
CV
(µ
1
, µ
2
, φ) + α E
shape
(φ) (α > 0). (6)
In ge neral, the prop os ed shape constraints affect the embedding
surface φ globally (i.e. on the entire domain Ω). In the simplest case,
such a prior has the form:
E
shape
(φ) =
Z
Ω
(φ(x) − φ
0
(x))
2
dx, (7)
recognition_ijcv.tex; 30/03/2005; 12:07; p.5
