Learning Must-Link Constraints for Video
Segmentation based on Spectral Clustering
Anna Khoreva
1
, Fabio Galasso
1
, Matthias Hein
2
, and Bernt Schiele
1
1
Max Planck Institute for Informatics, Saarbr¨ucken, Germany
{khoreva,galasso,schiele}@mpi-inf.mpg.de
2
Saarland University, Saarbr¨ucken, Germany
hein@cs.uni-saarland.de
Abstract. In recent years it has been shown that clustering and seg-
mentation methods can greatly benefit from the integration of prior in-
formation in terms of must-link constraints. Very recently the use of such
constraints has been integrated in a rigorous manner also in graph-based
methods such as normalized cut. On the other hand spectral cluster-
ing as relaxation of the normalized cut has been shown to be among
the best methods for video segmentation. In this paper we merge these
two developments and propose to learn must-link constraints for video
segmentation with spectral clustering. We show that the integration of
learned must-link constraints not only improves the segmentation result
but also significantly reduces the required runtime, making the use of
costly spectral methods possible for today’s high quality video.
1 Introduction
Video segmentation is an open problem in computer vision, which has recently
attracted increasing attention. The problem is of high interest due to its poten-
tial applications in action recognition, scene classification, 3D reconstruction and
video indexing, among others. The literature on the topic has become prolific [7,
43, 2, 28, 27, 19, 11, 10, 4, 29] and a number of techniques have become available,
e.g. generative layered models [25, 26], graph-based models [20, 46, 36] and spec-
tral techniques [39, 8, 15, 18, 32, 35, 16].
Spectral methods, stemming from the seminal work of [39, 34], have received
much attention from the theoretical viewpoint [31, 9, 21], and currently provide
state-of-the-art segmentation performance [3, 40, 18, 41, 35, 42, 32, 16]. Spectral
clustering, as a relaxation of the NP-hard normalized cut problem, is suitable
due to its ability to include long-range affinities [18, 40] and its global view on
the problem [14], providing balanced solutions.
In this paper, we focus on two important limitations of spectral techniques:
the excessive resource requirements and the lack of exploiting available training
data. The large demands of spectral techniques [40, 18] are particularly clear
in the case of high-quality video datasets [17], limiting their current large-scale
applicability. While often a labeled dataset is available, a systematic learning
of the affinities used to build the graph for spectral clustering is very difficult.
2 A. Khoreva, F. Galasso, M. Hein, B. Schiele
a. Video sequence b. SPX c. Proposed M SPX d. Video segm.
Fig. 1. Video segmentation [18] employs fine superpixels (b), resulting in large resource
requirements, esp. when using spectral methods. We propose learned must-links to
merge superpixels into fewer must-link-constrained M superpixels (c). This reduces
runtime and memory consumption and maintains or improves the segmentation (d).
In particular, as the normalized cut itself is a NP-hard problem and even the
spectral relaxation is non-convex, the optimization of the minimizer which yields
the segmentation is out of reach. Thus in practice one typically validates a few
model parameters [8, 18, 32], refraining spectral methods to make use of recently
available large training data [17].
We propose to learn must-link constraints to overcome both limitations. Re-
cent spectral theory [38, 16] has shown that the integration of must-links (i.e.
forcing two vertices to be in the same cluster) allows to reduce the size of the
problem, while preserving the original optimization objective for all partitions
satisfying the must-links. On the other hand by learning must-link constraints
we can leverage the available training data in order to guide spectral clustering
towards a desired segmentation. Figure 1 illustrates the advantages of learning
must-links: superpixel-based techniques [18] build spectral graphs on fine super-
pixels, Figure 1(b); by contrast, we propose to build graphs merging superpixels
based on learned must-link constraints, Figure 1(c). In particular, specifically
training a classifier to minimize the number of false positives allows conservative
superpixel merging, which: i. reduces the problem size significantly; ii. preserves
the original optimization problem; and iii. improves the video segmentation,
Figure 1(d), because correct must-links avoid undesired solutions (cf. Section 3).
In the following, we present the integration and learning of must-link con-
straints in Section 3 and validate them experimentally under various setups in
Section 4 on two recent video segmentation datasets [8, 17].
2 Related Work
The usage of must-link constraints, first introduced in [44], is an active area
of research in machine learning known as constrained clustering (see [5] for an
overview). The goal of integrating must-link constraints into spectral clustering
has been tried via: i. modifying the value of affinities (cf. [24], which first con-
sidered constrained spectral clustering); ii. modifying the spectral embedding
[30]; or iii. adding constraints in a post-processing step [49, 13, 48, 45, 33]. In-
terestingly, none of these methods can guarantee that the must-link constraints
Learning Must-Link Constraints for Video Segmentation based on SC 3
are actually satisfied in the final clustering. By contrast, we employ must-link
constraints to reduce the original graph to one of smaller size, thus enforcing the
constraints while additionally benefiting runtime and memory consumption.
In particular, [38, 16] have shown that must-link constraints can be used
to reduce the graph, based on the corresponding point groupings, and proved
equivalence between the reduced and the original graph, respectively in terms
of NCut [38] and SC [16], for any clustering satisfying the must-link constraints.
We employ these recent advances and propose to learn the must-link constraints
in a data-driven discriminative fashion for video segmentation.
Other related work in segmentation have looked at merging superpixels with
equivalence [1], but using hand-designed affinities, or learned pair-wise relations
between superpixels [23], disregarding equivalence in the agglomerative merging
process. This work brings together learning affinities and merging with equiva-
lence guarantees for the first time.
3 Learning sp ectral must-link constraints
We provide here the steps of a video segmentation framework based on the
normalized cut [39, 34, 22] and review the integration of must-link constraints by
graph reductions as proposed in [38, 16]. While the idea of learning must-link
constraints applies to any segmentation problem, we discuss in detail learning
and inference in the specific case of the video segmentation features of [18].
3.1 Segmentation and Must-link Constraints
We represent a video sequence as a graph G = (V, E): nodes i ∈ V represent su-
perpixels, extracted at each frame of the video sequence with an image segmen-
tation algorithm [3]; edges e
ij
∈ E between superpixels i and j take non-negative
weights w
ij
and express the similarity (affinity) between the superpixels.
A video segmentation can be defined as a partition S = {S
1
, S
2
, . . . , S
K
}
of the (superpixel) vertex set V, i.e. ∪
k
S
k
= V, S
k
∩ S
m
= ∅ ∀ k 6= m.
Given S the set of all partitions, we look for an optimal video segmentation
S
∗
= {S
∗
1
, S
∗
2
, . . . , S
∗
N
} ∈ S (where N is the number of visual objects), minimizer
of an objective function, implicit [20, 47, 37] or explicit [39, 34, 43, 10].
Must-link constraints alter the video segmentation by reducing the set of
feasible partitions S. Given correct
1
must-links, a video segmentation algorithm
generally improves in performance, since the solver is constrained to disregard
non-optimal segmentations wrt S
∗
. Moreover, the integration of must-links leads
to reduced runtime and memory load as the recent work [38, 16] suggests.
We are interested in learning a must-link grouping function M, which groups
certain
2
superpixels in the graph, while respecting S
∗
. M should conservatively
1
correct refers to the desired ground truth segmentation, which ideally corresponds
with the optimal segmentation S
∗
2
certain groupings are the conservative grouping decisions which we propose to learn
4 A. Khoreva, F. Galasso, M. Hein, B. Schiele
associate each node i with a point grouping I
k
⊆ S
∗
l
(in most uncertain cases a
point grouping may only include a single node). More formally:
M : V 7→ P, i 7→ I
k
(1)
s.t. I
k
⊆ S
∗
l
⊆ V , ∪
k
I
k
= V , I
k
∩ I
m
= ∅ ∀ k 6= m ,
where P is the set of possible partitions of V.
3.2 Framework
Here we tailor the general theory to a video segmentation framework based on
the normalized cut, solved either via the spectral [39, 34] or 1-spectral [9, 21]
relaxation. Further, we discuss the integration of learned must-link constraints
via graph reduction techniques [38, 16] and learning and inference strategies.
Video segmentation setup. We build upon Galasso et al. [18]. Their con-
structed graph G = (V, E) uses superpixels extracted from the lowest level (level
1) of a hierarchical image segmentation [3]. Edges connect superpixels from spa-
tial and temporal neighbors and are weighted by their pair-wise affinities, com-
puted from motion, appearance and shape features.
We consider six pairwise affinities: spatio-temporal appearance (STA), based
on the median CIE Lab color distance; spatio-temporal motion (STM), based
on median optical flow distance; across boundary appearance (ABA) and mo-
tion (ABM), computed across the common boundary of superpixels; short-term-
temporal (STT), measuring shape similarity by the spatial overlap of optical
flow-propagated superpixels; long-term-temporal (LTT), given by the fraction
of common trajectories between the superpixels. Additionally we consider the
number of common intersecting trajectories (IT). We distinguish four types of
affinities, depending on whether the related superpixels: i. lie within the same
frame (STA,STM,ABA,ABM); ii. lie on adjacent frames (STA,STM,STT); iii-
iv. lie on frames at a distance of 2 (STT,LTT,IT) or more frames (LTT,IT)
respectively.
Video segmentation objective function. Given a partition of V into N sets
S
1
, . . . , S
N
, the normalized cut (NCut) is defined [31] as:
NCut(S
1
, . . . , S
N
) =
N
X
k=1
cut(S
k
, V\S
k
)
vol(S
k
)
, (2)
where cut(S
k
, V\S
k
) =
P
i∈S
k
,j∈V\S
k
w
ij
and vol(S
k
) =
P
i∈S
k
,j∈V
w
ij
. The
balancing factor prevents trivial solutions and is ideal when unary terms cannot
be defined, but is also the reason why minimization of the NCut is NP-Hard.
Learning Must-Link Constraints for Video Segmentation based on SC 5
Spectral relaxations. The most widely adopted relaxation of NCut is spectral
clustering (SC) [39, 34, 31], where the solution of the relaxed problem is given by
representing the data points with the first few eigenvectors and then clustering
them with k-means.
While widely adopted [16, 32, 3, 8, 40, 18, 41], the SC relaxation is known to
be loose. We therefore additionally consider the 1-spectral clustering (1-SC) [21,
22] - a tight relaxation based on the 1-Laplacian. However, the relaxation is only
tight for bi-partitioning, for multi-way partitioning recursive splitting is used as
greedy heuristic.
Reducing the original graph size with learned must-link constraints allows
to experiment with 1-SC on state-of-the-art video segmentation benchmarks [8,
17], notwithstanding the increased computational costs.
Graph reduction schemes. Given must-link constraints provided as point
groupings {I
1
, I
2
, . . . , I
q
} on the original vertex set I
k
⊆ V, recent work [38, 16]
shows how to integrate such constraints into the original problem with respec-
tively preserving the NCut and the spectral clustering objective function.
In more detail, integration proceeds by reducing the original graph G to one
of smaller size G
M
= (V
M
, E
M
), whereby the vertex set is given by the point
grouping V
M
= {I
1
, I
2
, . . . , I
q
}, the edge set E
M
preserves the original node
connectivity and weights w
M
IJ
are estimated so as to preserve the original video
segmentation problem in terms of the NCut or spectral clustering objective. In
particular, the NCut reduction is given by
w
M
IJ
=
X
i∈I
X
j∈J
w
ij
(3)
while the spectral clustering reduction is defined as
w
M
IJ
=
X
i∈I
X
j∈J
w
ij
if I 6= J
1
|I|
X
i∈I
X
j∈J
w
ij
−
(|I| − 1)
|I|
X
i∈I
X
j∈V\I
w
ij
if I = J,
(4)
provided equal affinities of elements of G constrained in G
M
, cf. [16].
3.3 Learning
An ideal must-link constraining function M (Eq. 1) should only merge super-
pixels which are correct, i.e. belong to the same set in the optimal segmentation.
From an implementation viewpoint, it is convenient to consider instead M
pw
,
defined over the set of edges E of the graph G representing the video sequence:
M
pw
: E 7→ {0, 1} (5)
M
pw
casts the must-link constraining problem as a binary classification one,
where a true output for an input edge e
ij
means that i and j belong to the
same point grouping, in the must-link constrained graph G
M
.
![Fig. 3. Sample superpixels (SPX) and segmentation results of [18], compared with the proposed learned must-link variants, both when employing SC and 1-SC (cf. Section 4 for details). The proposed superpixels (M SPX) respect the video segmentation output while reducing the problem size. Additionally, M SPX improve results, esp. for 1-SC.](/figures/fig-3-sample-superpixels-spx-and-segmentation-results-of-18-hg57zn68.webp){kind=tracking}
![Fig. 2. Comparison of state-of-the-art video segmentation algorithms with the learned must-links, on BMDS (restricted to first 30 frames) [8]. The plots and table show BPR and VPR, aggregate measures ODS, OSS and AP, and length statistics (mean µ, std. δ, no. clusters NCL) [17].](/figures/fig-2-comparison-of-state-of-the-art-video-segmentation-qzl6ofwm.webp)
![Fig. 1. Video segmentation [18] employs fine superpixels (b), resulting in large resource requirements, esp. when using spectral methods. We propose learned must-links to merge superpixels into fewer must-link-constrained M superpixels (c). This reduces runtime and memory consumption and maintains or improves the segmentation (d).](/figures/fig-1-video-segmentation-18-employs-fine-superpixels-b-2k538ubt.webp){kind=tracking}
![Fig. 4. Comparison of state-of-the-art video segmentation algorithms with our proposed method based on the learned must-links, on VSB100 [17] (cf. Section 4 for details).](/figures/fig-4-comparison-of-state-of-the-art-video-segmentation-1zxq74ez.webp)