Volume 31 (2012), Number 8 pp. 2516–2526 COMPUTER GRAPHICS forum
Smart Scribbles for Sketch Segmentation
G. Noris
†1,2
, D. Sýkora
3
, A. Shamir
4
, S. Coros
1
, B. Whited
5
, M. Simmons
5
, A. Hornung
1
, M. Gross
1,2
, and R. Sumner
1
1
Disney Research Zürich,
2
ETH Zürich, CGL,
3
CTU in Prague, FEE,
4
The Interdisciplinary Center,
5
Walt Disney Animation Studios
Figure 1: Sketch segmentation: For each example pair, Scribbles on the left produce the segmentation on the right.
Abstract
We present Smart Scribbles—a new scribble-based interface for user-guided segmentation of digital sketchy draw-
ings. In contrast to previous approaches based on simple selection strategies, Smart Scribbles exploits richer geo-
metric and temporal information, resulting in a more intuitive segmentation interface. We introduce a novel energy
minimization formulation in which both geometric and temporal information from digital input devices is used to
define stroke-to-stroke and scribble-to-stroke relationships. Although the minimization of this energy is, in gen-
eral, a NP-hard problem, we use a simple heuristic that leads to a good approximation and permits an interactive
system able to produce accurate labelings even for cluttered sketchy drawings. We demonstrate the power of our
technique in several practical scenarios such as sketch editing, as-rigid-as-possible deformation and registration,
and on-the-fly labeling based on pre-classified guidelines.
Categories and Subject Descriptors (according to ACM CCS): Computer Graphics [I.3.4]: Graphics Utilities—
Graphics editors, Computer Graphics [I.3.6]: Methodology and Techniques—Interaction techniques, Image Pro-
cessing and Computer Vision [I.5.3]: Clustering—Similarity measures
Keywords: digital sketches, interactive segmentation,
scribble-based user interface, energy minimization
1. Introduction
Sketchy drawings are prevalent across a wide range of ap-
plications and domains. In early development phases, rough
drawings are used, for example, for concept art in product
†
chino@disneyresearch.com
design, and for storyboards in animation environments, and
are favored both for the speed of generation, and the expres-
siveness of the results. A sketchy style also has a place in fin-
ished art – providing a level of visual richness not found in
“clean” line representations, i.e. drawings constructed from
crisp, distinct outlines and minimal interior detail.
Modern digital devices and graphics software solutions
offer powerful stylization, deformation, morphing, and an-
imation capabilities for 2D drawings. However, in order to
perform these high-level tasks, a certain degree of under-
c
2012 The Author(s)
Computer Graphics Forum
c
2012 The Eurographics Association and Blackwell Publish-
ing Ltd. Published by Blackwell Publishing, 9600 Garsington Road, Oxford OX4 2DQ,
UK and 350 Main Street, Malden, MA 02148, USA.
Noris et al. / Smart Scribbles for Sketch Segmentation
standing of the content of the drawing is required. This is a
challenging problem due to the significant gap between the
ability of a human to discern structure in a drawing and the
capability of an algorithm to derive it from low level stroke
information. This is true even for clean line drawings, and
most existing approaches rely on the presence of a human
user to provide sufficient information to guide the task.
The problem of extracting structure from drawings be-
comes substantially more difficult for sketchy input, and this
is one reason it is far less common to find a consistently
sketchy style in full-length animations or automatic support
for sketchy input in high-level editing packages. One impor-
tant category of drawing abstraction is segmenting the draw-
ing into logical parts. To-date, there is no efficient method
available for automatic segmentation in this domain. In con-
texts where a breakdown of the drawing is required, seg-
mentation is typically achieved by design: the drawings are
created in different layers, one for each logical component.
This approach is too limiting in practice: it requires a priori
knowledge of the use of the drawing, is cumbersome (espe-
cially when different tasks require different segmentations),
and is an error-prone process, even for experienced artists.
We seek a semi-automated solution to segmenting sketchy
drawings that is fast enough for interactive use, but also pre-
dictable and easy to use – making it accessible to even the
most novice user.
To this end, we propose the concept of Smart Scribbles as
an accurate and simple way for the user to specify seman-
tically meaningful stroke clusters within a drawing. In con-
trast to previous methods that use scribbles as positional con-
straints for various image editing tasks [BJ01,LLW04,AP08,
SDC09b], our formulation considers more detailed geomet-
ric (position, orientation, curvature) and temporal informa-
tion (time of creation) when analyzing stroke-to-stroke and
Scribble-to-stroke relationships. In addition, we introduce
the concept of locality control as a way of conveniently trad-
ing off the Scribbles’ areas of influence for accuracy. This
allows our system to produce desired results with minimal
user intervention even for cluttered sketches.
We evaluate our approach on a collection of digitally
drawn sketches of varying complexity, and demonstrate its
application to various tasks including sketch editing and as-
rigid-as-possible (ARAP) deformation and registration. As
our solution is fast to compute, our method enables tight in-
tegration of these tasks within an interactive digital drawing
session.
2. Related Work
Relevant prior art can be divided into three main categories:
sketch labeling interfaces, scribble-based image segmenta-
tion, and classification of vector fields in scientific visualiza-
tion.
User-guided labeling of strokes in hand-drawn images
plays a central role in many sketch-based editing systems.
In Lank et al. [LS05], the authors present an approach for
inferring user intent from the local velocities, accelerations
and curvatures of the selection lasso. More recently, Wolin
et al. [WSA07] presented a technique for labeling groups of
strokes from a vectorized sketch where the system attempts
to automatically fragment continuous strokes into logical
pieces to assist the user. Both of these techniques ultimately
utilize a region-based selection approach. ScanScribe, a sys-
tem developed by Saund and colleagues [SFLM04], presents
the user with an intuitive selection paradigm that allows for
the creation of objects from collections of pixels and sup-
ports further grouping into composite objects. The system
is able to automatically segment the image into basic prim-
itives, such as linear curve fragments, and then group them
into more complex objects, such as rectangles, using a frag-
ment alignment metric (or by finding perceptually closed
paths as proposed in [Sau03]). Two limitations of this au-
tomatic technique are 1) limited complexity of objects de-
tected by the system and 2) the inability to handle sketchy
overlapping curve fragments, thus requiring more traditional
and tedious lasso/selection-box methods for more complex
drawings.
The approach presented in this paper leverages previous
works on interactive image segmentation in order to opti-
mize the labeling process based on user scribbles. Boykov
et al. [BJ01] developed such an approach based on graph
cuts for segmenting images and finding optimal boundaries
between objects. In [LLW04], Levin and colleagues present
a similar framework based on a least-squares optimization
for colorizing gray-scale images by roughly labeling re-
gions with colored scribbles. More recently, An and Pel-
lacini [AP08] developed an interactive energy minimiza-
tion framework for propagating color edits to similar regions
throughout the image. Our approach is most similar to Lazy-
Brush [SDC09b], a graph cut based system for the selec-
tion of regions in sketchy drawings. The main difference is
that this system cannot provide the labeling of the strokes
that bound each painted region. From this point of view, our
framework can be seen as a generalization of LazyBrush,
since it extracts meaningful boundaries first, and then builds
regions inferred from those boundaries. Because this process
removes clutter from the input drawing, it greatly improves
the accuracy of selection and reduces the amount of user in-
teraction needed to obtain clean results.
Our approach also bears some resemblance to sketch-
based clustering of vector fields in scientific visualiza-
tion [WWYM10]. Here the aim is to allow the user to sketch
2D curves and use them as a query to retrieve 3D field lines
whose view-dependent 2D projection is most similar to the
input sketch. The curvature along the sketched input is used
to measure the similarity between the input and projected
curves using the edit distance [WF74]. In our approach, cur-
vature is also used to distinguish between different shapes.
However, the main advantage of our work is that we formu-
c
2012 The Author(s)
c
2012 The Eurographics Association and Blackwell Publishing Ltd.
Noris et al. / Smart Scribbles for Sketch Segmentation
late an energy minimization problem where, in addition to
shape similarity, we also take proximity, orientation, tempo-
ral information, and smoothness of the final labeling into ac-
count. As a result, our system can produce reasonable clus-
tering even in cases when the shape of the input sketch is
very rough or incomplete.
3. Method
The method we present allows users to intuitively segment
digital sketches into semantically meaningful regions. The
input to our framework consists of a digitally hand-drawn
sketch and a small set of rough Scribbles. The input sketch
is composed of a set of strokes, which are piecewise linear
curves represented by sets of 2D vertices recorded from a
digital input device such as a tablet. For each vertex of a
stroke, we additionally store its time of creation. This helps
us to differentiate strokes which are spatially close but are
drawn at different moments in time.
The input Scribbles are special strokes that indicate the
user’s intent to segment a particular portion of the drawing.
Two criteria related to the Scribble primitives are critical in
order to ensure a useful and intuitive system. First, Scribbles
should not have to closely follow the target region. However,
if desired, the user should be able to precisely select local-
ized regions. We call this property locality control. The sec-
ond criterion specifies that the time of creation of the Scrib-
ble should not influence the segmentation results.
We observe that generally speaking, processing strokes as
a whole is very difficult. A single stroke can be arbitrarily
complex: it can cross or overlap with itself multiple times,
and/or it can densely cover an area the artist wished to fill in.
For this reason, we break strokes and Scribbles into linear
segments by densely resampling the input. Any property de-
fined locally over the stroke can easily be transferred to the
segments.
The remainder of this section describes in detail each of
the steps used by our method
We formulate the task of sketch clustering as an optimiza-
tion problem, where the goal is to label each stroke in a way
that minimizes an energy function. The concept of our de-
sign is depicted in Fig. 2 and the remainder of this section
describes in detail each of the steps used by our method.
The energy function is defined in Section 3.1. It relies on
a smoothness and data term which are described in Sec-
tions 3.1.1 and 3.1.2, respectively. In Section 3.2 we discuss
the minimization method used to compute the final solution
to the stroke labeling.
3.1. Energy function
The input to our method consists of a set of stroke segments
S and a set of Scribble segments R associated with a set of
labels L. The goal is to find a labeling, i.e., an assignment φ
of the labels in L to every segment in S, that minimizes the
following energy function E:
E(φ) =
∑
i, j∈S
V
i, j
(φ
i
,φ
j
) + λ
∑
i∈S
D
i
(φ
i
) (1)
where V
i, j
is a smoothness term that captures the cost of the
labeling with respect to the similarity between two stroke
segments i and j. The data term D
i
measures the affinity be-
tween Scribbles and strokes. The parameter λ controls the
relative influence of the smoothness and data terms.
Smoothness Term
Input
Output
Data Term
1
2
Scribble
Stroke
Figure 2: Energy definition overview. The input consists of
a set of strokes (black) and Scribbles (red and blue dotted
lines). The output consists of a labeling of all strokes (the
labeling is indicated here by the red/blue color assignment
to the strokes in the output). Smoothness Term: For a seg-
ment i and a neighbor segment j, V
i, j
expresses the energy
of assigning a different label to i and j, based on how simi-
lar they are. Data Term: Given a labeling φ
i
= l
∗
(assigning
label l
∗
to segment i), D
i
(φ
i
) expresses the energy of the la-
beling, which is a function of the similarity of segment i to
all Scribbles associated with l
∗
.
3.1.1. Smoothness Term
The smoothness term is defined as:
V
i, j
(φ
i
,φ
j
) =
∏
g∈G
δ(g(i, j),σ
g
) (2)
when φ
i
6= φ
j
, otherwise it is zero. G is a set of similarity
terms:
prox(i, j) = ||~p
j
− ~p
i
||
dir(i, j) = 1 − |
~
d
i
·
~
d
j
|
curv(i, j) = 1 − min(c
i
,c
j
)/max(c
i
,c
j
)
time(i, j) = |t
j
−t
i
|
c
2012 The Author(s)
c
2012 The Eurographics Association and Blackwell Publishing Ltd.
Noris et al. / Smart Scribbles for Sketch Segmentation
where i and j are two segments, and p, d, c and t are the
position, direction, radius of curvature, and time of creation
associated with each segment. The fall-off function δ is de-
fined as:
δ(g(i, j),σ
g
) = exp
−
g(i, j)
2
σ
2
g
!
(3)
3.1.2. Data Term
The data term is defined as:
D
i
(φ
i
) = 1 − max
r∈R(φ
i
)
A(i,r) (4)
where R(φ
i
) denotes a set of Scribble segments r with label
φ
i
. The affinity A(i,r) is defined as:
A(i,r) =
∏
g∈G
dat a
δ(g(i,r),σ
g
) (5)
Here, as with the smoothness term, we measure the similar-
ity between segments rather than strokes. However, as Scrib-
bles have no associated time information, we reduce the set
of similarity terms to G
data
= {prox,dir,curv} ⊂ G. Addi-
tionally, we alter the definition of curvature to become ori-
ented: curv(i, j) = ||~c
i
− ~c
j
||. This allows extra control in
separating curves with the same curvature but different ori-
entation (e.g. the tangled lines in Fig. 5).
One of our main goals is to allow users, if desired, to
have precise local control over the strokes that get affected
by each Scribble. To illustrate this, we consider a scenario
where the user draws a single Scribble, as shown in Fig. 3a.
In this case, because no concurrent label exists, all strokes
are selected. This behavior, though reasonable, is not in line
with a user’s expectations of having local control.
To address this, we introduce an artificial background la-
bel b ∈ L, in addition to the labels prescribed by the user.
This new label has a constant influence on each stroke seg-
ment i regardless of the existence of any particular user-
defined Scribble, i.e., A(i, b) = B, where B is a threshold to
override the influence of distant Scribbles. The background
label therefore serves as a lower bound for computing the
max component in the data term (4).
Furthermore, we control the locality of each Scribble r by
modifying its proximity fall-off δ (3) as follows:
δ(prox(i,r),σ
prox
) =
1
σ
prox
exp
−
prox(i,r)
2
σ
2
prox
!
. (6)
Here σ
prox
follows the desired locality (i.e., is large for
global influence and small for local influence) and the nor-
malization term 1/σ
prox
ensures the integral over the fall-off
function stays equal for different values of σ
prox
(i.e., ampli-
tude is high for small values and low for large ones). In other
words, the overall energy remains constant, while its spatial
spread is controlled. When σ
prox
becomes very low, the re-
sponse of the fall-off function (6) for distant stroke segments
also becomes very low and can therefore be easily overrid-
den when computing the max value in (4) as illustrated in
Fig. 3b-f.
There are several possible ways to control the parameter
σ
prox
. One natural way is to use the speed of the Scribble
based on the experimentally demonstrated linear relation-
ship between speed and perceived locality [AZ97]:
W =
β · L
T − α
. (7)
Here W is the selection radius, L is length of the Scribble, T
is time spent on drawing it, and α and β are empirically mea-
sured constants. This rule was used to control the selection
locality in systems having limited modality [LS05]. Since
the spatial spread of the fall-off function (6) grows linearly
with the increasing σ
prox
we can set σ
prox
= W /2. Alter-
natively, one could consider the use of pen-pressure, or—in
the case of binary modality—a simple key toggle to switch
between two locality values.
3.2. Optimization method
As shown in [BVZ98], minimizing the energy function de-
fined in Equation 1 is equivalent to solving a multi-way
cut on a specific weighted graph G = {V,E}, where V =
{S,L} is a set of vertices and E = {E
s
,E
l
} is a set of edges
(See Fig. 4). The graph vertices V consist of stroke seg-
ments S and label terminals L. Each stroke segment i ∈ S
is connected to all other stroke segments j ∈ S − {i} via
edges E
i, j
having weight w
i, j
equal to the smoothness term
V
i, j
when φ
i
6= φ
j
. In addition, auxiliary edges E
i,l
connect
stroke segments i ∈ S to label terminals l ∈ L. Each E
i,l
has
weight w
i,l
= λ(1−D
i
(l)), where λ is the parameter defined
in Equation 1.
1
2
1
2
Figure 4: Graph Construction. Stroke segments are shown
as black circles. Terminal labels (in this example l
1
and l
2
)
are shown as colored squares. The graph edges w
i, j
reflect
the smoothness terms V
i, j
between the stroke segments i, j ∈
S, while the data terms D
i
(l) for stroke segment i ∈ S and
label l ∈ L are captured by the weights w
i,l
.
c
2012 The Author(s)
c
2012 The Eurographics Association and Blackwell Publishing Ltd.
Noris et al. / Smart Scribbles for Sketch Segmentation
= 3px= 30px= 60px= 90px= 120px= 150px
a b c d e f
Figure 3: The effect of the locality control by varying σ
prox
: A blue Scribble is drawn on the foot (circled in red). On the right,
the value of σ
prox
is progressively decreased. Notice how the selection becomes progressively more local as the influence of the
blue label gets overruled by the background label (shown in black).
The multi-way cut problem with two terminals is equiv-
alent to a max-flow/min-cut problem for which efficient
polynomial algorithms exist [BK04]. However, for three or
more terminals the problem is NP-hard [DJP
∗
92]. To ob-
tain a good approximate solution we use a simple divide-
and-conquer heuristic previously proposed in [SDC09b] to
gradually simplify the N-terminal problem into a sequence
of N − 1 binary max-flow/min-cut sub-problems. This ap-
proach provides results similar to more advanced techniques
(such as α-expansion or α/β-swap [BVZ01]), but is signifi-
cantly faster and therefore better suited for interactive appli-
cations.
4. Results
We demonstrate the effectiveness of our algorithm on a va-
riety of input sketches. All results were generated using the
parameters in Table 1.
Fig. 5 shows a collection of simple input sketches and
Scribbles, together with the color-coded stroke labeling out-
put by our system. These results show that desirable sketch
segmentations can be obtained using very different scrib-
bling strategies. We note that the input Scribbles do not have
to closely match the sketch in order for our algorithm to
work well—approximate similarity in terms of position, ori-
entation and curvature is sufficient.
Figs. 1 and 6 show results from more complex input
sketches. To correctly segment these images, users typically
start with rough, fast strokes, and then refine the output lo-
cally using slower, more accurate strokes. Our method ro-
bustly handles scenarios where strokes that are close to-
gether and almost parallel belong semantically to different
regions (as shown on the waiter’s legs and snake and pole
example in Fig. 6). In these cases, the time metric plays an
important role in the labeling process.
Our framework does not require artists to draw the input
sketches in any particular manner. It is possible that strokes
representing the same region can be drawn at very different
moments in time. This happens, for instance, when artists
first draw silhouettes for the whole scene, and then proceed
to refine the drawing. This can diminish the advantage of
Figure 5: Results for simple sketches: several different in-
puts produce the same segmentation.
taking timing into account in the similarity metric. Correct
segmentations can still be obtained, but more Scribbles may
be required. Alternatively the similarity metric can be ad-
justed to apply a smaller weight to the time parameter, or it
can be removed as is done for the Scribble metric.
4.1. User Study
In order to test the efficiency and ease of use of our method,
we conducted a user study comparing Smart Scribbles to our
implementation of several commonly-used selection tools,
c
2012 The Author(s)
c
2012 The Eurographics Association and Blackwell Publishing Ltd.