TO APPEAR IN THE PROCEEDINGS OF THE IEEE INTERNATIONAL CONFERENCE ON ROBOTICS AND AUTOMATION (ICRA), 2011.
Bringing Clothing into Desired Configurations with Limited Perception
Marco Cusumano-Towner, Arjun Singh, Stephen Miller, James F. O’Brien, Pieter Abbeel
Fig. 1: The PR2 with a pair of pants in a crumpled initial
configuration.
Abstract— We consider the problem of autonomously bring-
ing an article of clothing into a desired configuration using
a general-purpose two-armed robot. We propose a hidden
Markov model (HMM) for estimating the identity of the
article and tracking the article’s configuration throughout a
specific sequence of manipulations and observations. At the
end of this sequence, the article’s configuration is known,
though not necessarily desired. The estimated identity and
configuration of the article are then used to plan a second
sequence of manipulations that brings the article into the
desired configuration. We propose a relaxation of a strain-
limiting finite element model for cloth simulation that can
be solved via convex optimization; this serves as the basis
of the transition and observation models of the HMM. The
observation model uses simple perceptual cues consisting of
the height of the article when held by a single gripper and
the silhouette of the article when held by two grippers. The
model accurately estimates the identity and configuration of
clothing articles, enabling our procedure to autonomously bring
a variety of articles into desired configurations that are useful
for other tasks, such as folding.
I. INTRODUCTION
Due to their inherently high-dimensional configuration
spaces, non-rigid objects pose a number of difficult chal-
lenges. This difficulty is exemplified by the state of the art
in robotic laundry folding, where existing methods are far
from being able to perform the task with general purpose
manipulators. Perhaps the biggest challenge facing robotic
laundry manipulation is how to bring a clothing article into
a known configuration from an arbitrary initial state.
The authors are with the Department of Electrical Engineering and
Computer Sciences, UC Berkeley, CA 94720, U.S.A. Contact Email:
marcoct@berkeley.edu.
Fig. 2: The PR2 holding up the pair of pants after starting
with the configuration shown in Figure 1.
We present an approach that enables a general purpose
robot to bring a variety of clothing articles into desired
configurations. The core of our approach is a hidden Markov
model based on the behavior of cloth under certain simple
manipulation strategies, and how it is perceived using basic
computer vision primitives. Key contributions of this paper
are:
• We propose a method for identifying an article of cloth-
ing and estimating its configuration with only simple
manipulation and limited perception.
• We propose a convex relaxation of the isotropic strain-
limiting model of Wang et. al [16] for cloth simulation.
• We present a planning algorithm for bringing an article
from a known configuration into a desired configuration.
• We describe our implementation of an end-to-end sys-
tem on the Willow Garage PR2 robotic platform. The
system starts with crumpled articles and brings them
into a desired configuration. We successfully tested our
implementation on seven articles of various types. All of
the parameters in our system were trained on a separate
set of articles.
Videos of our experimental results are available at:
http://rll.berkeley.edu/ICRA_2011
II. RELATED WORK
Extensive work has been done on enabling specialized and
general-purpose robots to manipulate laundry. To the best
of our knowledge, however, with the exception of the towel
folding capability demonstrated by Maitin-Shepard et al. [8],
no prior work has reported successful completion of the full
1
end-to-end task of picking up an arbitrarily placed clothing
article and bringing it into a neatly folded state. In this paper
we focus on a key part of this end-to-end task; namely,
bringing a clothing article from an unknown configuration
into a desired configuration.
The work of Osawa et al. [12] and Kita et al. [3]–[6] is
the most closely related to our approach. Osawa et al. use
the idea of iteratively grasping the lowest-hanging point.
They describe how this procedure leads to a relatively small
number of fixed points. Once their perception unit recognizes
that a corner has been grasped, their procedure compares
the shape observed while pulling taut with pre-recorded
template images. They reported recognition rates on seven
different clothing categories. In contrast to our work, they
require template images of the articles, they have a one-
shot decision making process (rather than a probabilistic
estimation framework), and their procedure only performs
“lowest-hanging point” re-grasps. As a consequence of only
re-grasping lowest points, their final configuration is not
necessarily spread out. While they do not report success
rates, they show successful manipulation of a long-sleeved
shirt with a final configuration having it held by the ends
of the two sleeves. Kita et al. consider a mass-spring model
to simulate how clothing will hang. Their work shows the
ability to use fits of these models to silhouettes and 3-D
point clouds to extract the configuration of a clothing article
held up by a single point with a good success rate. Their
later work [4]–[6] shows the ability to identify and grasp a
desired point with the other gripper. None of this prior work
demonstrates the ability to generalize to previously unseen
articles of clothing.
There is also a body of work on recognizing categories of
clothing; some of this work includes manipulation to assist
in the categorization. For example, Osawa and collaborators
[12] and Hamajima and Kakikura [2] present approaches to
spread out a piece of clothing using two robot arms and then
categorize the clothing.
Some prior work assumes a known, spread-out, or partially
spread-out configuration, and focuses on folding or complet-
ing other tasks. The work of Miller et al. [9], building on the
work of van den Berg et al. [15], has demonstrated reliable
folding of a wide range of articles. Paraschidis et al. [1] de-
scribe the isolated executions of grasping a laid-out material,
folding a laid-out material, laying out a piece of material that
was already being held, and flattening wrinkles. Yamakazi
and Inaba [17] present an algorithm that recognizes wrinkles
in images, which in turn enables them to detect clothes laying
around. Kobori et al. [7] have extended this work towards
flattening and spreading clothing. They successfully spread
out a towel.
III. OVERVIEW
A. Problem Definition
The problem we examine is defined as follows: we are
presented with an unknown article of clothing and wish to
identify it, work it into an identifiable configuration, and
subsequently bring it into a desired configuration.
B. Notation
We consider articles of different types and sizes. These
potential articles make up the set of articles (A) under
consideration. For example, we may be considering two
different pairs of pants and a t-shirt; in this case, we have
A = {pants
1
, pants
2
, t-shirt}. We assume that each type of
clothing can have its own desired configuration; for example,
we choose to hold up pants by the waist.
We represent each potential article a (from the set A) via
a triangulated mesh. For concreteness, let the mesh contain
N points {v
1
, . . . , v
N
}. We work under the assumption that
the robot may be grasping a single point or a pair of points
on the mesh. Let g
t
be the grasp state of the cloth at time t,
where g
t
= (g
l
t
, g
r
t
) consists of the mesh point of the cloth in
the robot’s left and right gripper respectively. More precisely,
we have g
t
= (g
l
t
, g
r
t
) ∈ G = {∅, v
1
, . . . , v
N
}
2
, where ∅
denotes that the gripper does not contain any mesh point. The
set G contains all possible grasp states of the cloth. The 3D
coordinates of the left and right gripper at time t are denoted
by x
l
t
and x
r
t
respectively. We denote the 3D coordinates of
the N mesh points at time t as X
t
= {x
1
t
, . . . , x
N
t
}.
C. Outline of Our Approach
Our approach consists of two phases, as shown in Figure
3.
First, we use a probabilistic model to determine the identity
of the clothing article while bringing it into a known configu-
ration through a sequence of manipulations and observations,
which we refer to as the disambiguation phase. Second,
we bring the article into the desired configuration through
another sequence of manipulations and observations, which
we call the reconfiguration phase.
Repeat lowest-
hanging point
procedure
Take
observations
Choose most
likely article and
configuration
Plan sequence
to desired
configuration
Initial
configuration
Arbitrary known
configuration
Desired
configuration
Disambiguation Phase Reconfiguration Phase
Fig. 3: Block diagram outlining our procedure. The t-shirt starts out in a crumpled state. We manipulate it with the lowest-
hanging point procedure and take observations. We choose the most likely configuration and article and plan a sequence of
manipulations to the desired configuration. The robot executes the sequence and grasps the t-shirt by the shoulders.
2
...
A
G
1
G
2
G
t
E
1
E
2
E
t
Fig. 4: Graphical representation of the hidden Markov model.
These phases use three major components: the hidden
Markov model, our cloth simulator, and our algorithm for
planning the manipulations for the reconfiguration phase.
1. Hidden Markov model. The hidden state of our model
consists of the grasp state of the cloth (g
t
) and the article
which the robot is currently grasping (a). The HMM operates
in the disambiguation phase, where the robot executes a
sequence of manipulations consisting of repeatedly holding
up the clothing article by one gripper under the influence
of gravity and grasping the lowest-hanging point with the
other gripper. The transition model of the HMM encodes
how the grasped points change when the robot manipulates
the cloth. This sequence quickly reduces uncertainty in the
hidden state. After this manipulation sequence the HMM
uses two observations, the height of the article when held by
the last point in the sequence and the contour of the article
when held by two points.
2. Cloth simulator. We simulate articles using triangulated
meshes in which each triangle element is strain-limited and
bending energy and collisions are ignored. This model has
a unique minimum-energy configuration (X
t
) when some
points in the mesh are fixed at the locations (g
l
t
, g
r
t
).
3. Planning algorithm. To generate the plan for the
reconfiguration phase, our planning algorithm generates a
sequence of manipulations in which the robot repeatedly
grasps two points on the cloth, places the cloth onto the
table, and grasps two other points. The planning algorithm
assumes that the most likely model and state reported by the
HMM in the disambiguation phase are correct.
IV. HIDDEN MARKOV MODEL
We have the discrete random variables A and G
t
, where
G
t
takes on values from the set G of all possible grasp
states of the cloth (as defined in Section III-B). The model
estimates the probability P (A = a, G
t
= g
t
|E
1:t
= e
1:t
),
where E
1:t
is the set of all observations through time t.
The robot’s gripper locations (x
l
t
and x
r
t
) are assumed to
be deterministic. The graphical model for our problem is
shown in Figure 4.
As described above, the 3D coordinates of the mesh points
at time t (X
t
) are uniquely determined from the article, grasp
state of the cloth, and the locations of the grippers (a, g
t
,
x
l
t
, and x
r
t
, respectively). Using G
t
as the state rather than
X
t
reduces the state space to a tractable size on the order
of N
2
, where N is the number points in a mesh.
Without loss of generality, we assume that the robot first
picks up the article with its left gripper. Intuitively, the initial
probability distribution over models and grasp states should
be zero for any state in which the right gripper is grasping
the cloth and uniform over all states in which the left gripper
is grasping the cloth. Therefore the initial distribution is:
P (a, g
0
) =
(
1
N|A|
g
r
0
= ∅, g
l
0
6= ∅
0 otherwise.
Recall that the disambiguation phase consists of repeatedly
holding up the article (under the influence of gravity) by one
gripper and grasping the lowest-hanging point with the other
gripper. Let g
t−1
be the grasp state of the robot at the end
of this process; the robot is only holding the article in one
gripper. Next, we take an observation of the height (h
t−1
)
of the article in this grasp state (g
t−1
). Afterwards, we have
the free gripper grasp the lowest-hanging point, bringing us
into the grasp state g
t
, in which both grippers are grasping
the article. We then move the grippers such that they are at
an equal height and separated by a distance of roughly h
t−1
.
Therefore the gripper locations are now
x
l
t
=
x,
h
t−1
2
, z
, x
r
t
=
x,
−h
t−1
2
, z
where the exact values of x and z are unimportant. We then
take an observation of the contour of the article against the
background.
Together, the lowest-hanging point sequence along with
the two observations compose all of the information obtained
about the article during the disambiguation sequence; the
details of the probabilistic updates for the transitions and
observations are explained below.
A. Transition Model
A transition in the HMM occurs after holding the cloth up
with one gripper, grasping the lowest-hanging point with the
free gripper, and then releasing the topmost point. Without
loss of generality, let the cloth be grasped by only the
left gripper at time t. Specifically, the grasp state g
t
is
(g
l
t
, ∅). This implies g
t+1
= (∅, g
r
t+1
), where g
r
t+1
is the
lowest-hanging point at time t. The transition model gives
the probability that each mesh point hangs lowest and is
therefore grasped by the right gripper. In particular,
P (g
t+1
|a, g
t
) =
P (g
r
t+1
is lowest|a, g
l
t
is held) if g
l
t+1
= ∅
0 if g
l
t+1
6= ∅.
The transition model assumes the robot has successfully
grasped a point with the right gripper. When we simulate an
article held at a single point v
i
, the resulting configuration
X is a straight line down from v
i
; the probability of point
v
j
hanging lowest when the cloth is held by v
i
depends on
X. Let d
ij
be the vertical distance from v
i
to v
j
in this
configuration. The probability of a point hanging lowest is
3
based on d
ij
:
P (v
j
is lowest | a, v
i
is held) =
e
λd
ij
P
N
k=1
e
λd
ik
.
This expression is a soft-max function, resulting in a dis-
tribution in which points that hang lower in the simulated
configuration are more likely to be the lowest-hanging point
in reality. The parameter λ expresses how well the simulated
configuration reflects reality.
Repeated application of the lowest-hanging point primitive
causes the grasp state to converge to one of a small number of
regions on the clothing article. For example, if a pair of pants
is initially grasped by any point and held up under gravity, the
lowest-hanging point will likely be on the rim of a pant-leg.
Once the rim of the pant-leg is grasped and held up, the rim
of the other pant-leg will likely contain the lowest-hanging
point. Continuing this procedure typically cycles between
the two pant-leg rims. In practice, the clothing articles we
consider converge to a small number of points within two or
three repetitions of the lowest-hanging point primitive. The
HMM transition model captures this converging behavior,
thereby significantly reducing uncertainty in the estimate of
the article’s grasp state (g
t
). Note that this transition model
does not change the marginal probabilities of the articles
(P (a|e
1:t
)).
B. Height Observation
When the article is held up by a single gripper, the
minimum-energy configuration provided by our cloth sim-
ulator is a straight line, as described in Section V. Although
this provides no silhouette to compare against, the length of
this line (h
sim
) is a good approximation to the article’s actual
height (h
t
). The uncertainty in this height is modeled with a
normal distribution:
P (h
t
|g
t
, a) ∼ N(h
sim
+ µ, σ
2
)
where µ is the mean difference between the actual height
and the simulated height.
This update makes configurations of incorrect sizes less
likely. For example, if we measure that the article has a
height of 70 cm, it is highly unlikely that the article is in a
configuration with a simulated height of 40 cm.
C. Contour Observation
When the cloth is held up by two grippers, the contour of
the simulated configuration is a good approximation to the
actual contour, as seen in Figure 5. The predicted contours
for each pair of grasp states and articles (g
t
, a) are computed
from the mesh coordinates (X
t
) generated by the cloth
simulator, which is detailed in Section V. Next, the dynamic
time warping algorithm is used to find the best alignment
of each predicted contour to the actual contour. The score
associated with each alignment is then used to update the
belief p(g
t
, a|e
1:t
).
Although the general shape of the simulated contour and
the actual cloth contour are similar, the amount of overlap
between them can vary greatly between different trials due
Fig. 5: The simulated contour (pink) is overlaid on the actual
cloth image.
to inconsistency in grasping and other unmodeled factors. To
account for this, we use a dynamic programming algorithm
known as dynamic time warping (DTW) in the speech-
recognition literature [13] and the Needleman-Wunsch al-
gorithm in the biological sequence alignment literature [11].
Dynamic time warping is generally used to align two se-
quences and/or to calculate a similarity metric for sequences.
In order to closely match the key features of clothing
articles, such as corners, collars, etc., we choose a cost
function of the form
φ(p
i
, p
j
) = kθ(p
i
) − θ(p
j
)k
where θ extracts the weighted features of each pixel p
i
. Our
features are the (x, y) pixel coordinates of the contour points
and the first and second derivatives with respect to the arc
length s, (
dx
ds
,
dy
ds
) and (
d
2
x
ds
2
,
d
2
y
ds
2
). The derivative terms force
corners and other salient points to align to each other. An
example where the amount of overlap is a poor measure of
similarity but DTW returns a reasonable alignment is shown
in Figure 6.
Let the dynamic time warping cost for each article and
grasp state pair (a, g
t
) be denoted c
a,g
t
. We found that using
an exponential distribution with the maximum-likelihood
estimate was too generous to costs associated with incorrect
configurations. Based on inspection of empirically collected
Fig. 6: An example of a challenging alignment where a sim-
ple overlap metric would perform poorly. The dynamic time
warping algorithm matches the salient features of simulated
contour (yellow) to the actual contour (blue) well.
4
DTW data, we propose the following distribution for the
dynamic-time-warping costs:
P (c
a,g
t
|a, g
t
) =
(
1
f+
1
d
if c
a,g
t
< f
1
f+
1
d
e
−dc
a,g
t
if c
a,g
t
≥ f.
This distribution, shown in Figure 7, is uniform for costs
below a certain threshold and quickly drops off as the cost
increases past the threshold.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7
0
1
2
3
4
5
6
c
a,g
t
P(c
a,g
t
|a, g
t
)
Fig. 7: Our probability distribution over DTW costs for the
correct grasp state and article.
V. CLOTH SIMULATOR
To go from the grasp state (g
t
) and article (a) to the
simulated 3D coordinates of each mesh point (X
sim
=
{x
1
sim
, . . . , x
N
sim
}), we minimize the gravitational potential
energy of all mesh points subject to two sets of constraints.
Our choice of constraints and energy function make this a
convex optimization problem with a unique solution.
The first set of constraints represents the cloth’s grasp state
as equality constraints on the simulated mesh configuration
X
sim
. If the grasp state (g
l
t
, g
r
t
) = (v
a
, v
b
), then the equality
constraints are x
a
sim
= x
l
t
and x
b
sim
= x
r
t
.
The second set of constraints limits the extensibility of the
cloth. We use the isotropic strain-limiting model introduced
by Wang et al. [16]. This model limits the strain of each trian-
gle element e ∈ E ⊂ {v
1
. . . v
N
}
3
by restricting the singular
values of the deformation gradient F
e
1
. Wang et al. restrict
the minimum and maximum strain of each triangle element.
We restrict only the maximum strain
2
; therefore our con-
straints are expressed as
maxSingularValue(F
e
(X
sim
)) ≤ σ for all e ∈ E,
where σ is the extensibility of the mesh surface, with σ =
1 indicating the cloth cannot be stretched at all
3
. This can
1
See Wang et al. for details [16].
2
The minimum strain constraint is non-convex. We also ignore bending
energy and collisions because they too are non-convex.
3
We found that σ = 1.03 works well.
also be expressed as the following semidefinite programming
(SDP) constraint:
σ
2
I
3
F
e
(X
sim
)
F
>
e
(X
sim
) I
2
0 for all e ∈ E. (1)
In summary, our optimization problem becomes:
min
X
sim
U(X
sim
) =
N
X
i=1
z
i
s.t. x
a
sim
= x
l
t
, x
b
sim
= x
r
t
X
sim
satisfies Equation (1)
where z
i
is the z-coordinate of the ith mesh point. We feed
this problem into the SDP solver SDPA [18] to simulate the
article’s configuration
4
.
Despite the strong assumptions, this model predicts config-
urations whose contours are realistic for the case of two fixed
points. The predicted contours are used in the observation
update described in Section IV-C.
When the robot is only grasping the cloth with one gripper,
the simulated configuration is a straight line down from the
fixed point. Although this predicted configuration is visually
unrealistic, the resulting height of each point is a good
approximation. These height values are used by the HMM in
the transition model and the height observation as described
in Sections IV-A and IV-B respectively.
VI. PLANNING ALGORITHM
Our planning algorithm generates the sequence of manip-
ulations to be carried out during the reconfiguration phase.
We assume that the disambiguation phase correctly identifies
the article and grasp state of the cloth.
For each type of clothing (e.g., shirts, pants, etc.), the user
specifies a desired configuration, determined by the pair of
points (v
i
, v
j
) that the robot should grasp. This determines
a desired grasp state g
d
= (v
i
, v
j
). For example, the user
could select that the robot hold a t-shirt by the shoulders or
a pair of pants by the hips.
Our algorithm plans a sequence of grasp states, where each
state has both grippers holding the cloth, to get from the ini-
tial grasp state g
i
(obtained from the disambiguation phase)
to the desired grasp state g
d
. The sequence of manipulations
to get from one grasp state to the next consists of laying the
cloth on the table, opening both grippers, and picking up the
cloth by a new pair of points.
The appropriate sequence of grasp states is generated by
building the directed graspability graph, which indicates
which other grasp states can be reached from each grasp
state. To build this graph the article is simulated for all
grasp states, and the resulting configurations are analyzed
for graspability. To ensure that our assumption that the robot
fixes a single point with each gripper is reasonable, we say
that a point v
i
is graspable in a given configuration X when
4
On a dual-core 2.0 GHz processor, SDPA can run roughly four simula-
tions per second when we use about 300 triangle elements per mesh. We
find that increasing the number of elements past this point does not make
the simulations significantly more realistic.
5
