University of Pennsylvania University of Pennsylvania
ScholarlyCommons ScholarlyCommons
Departmental Papers (MEAM)
Department of Mechanical Engineering &
Applied Mechanics
October 2002
A Vision-Based Formation Control Framework A Vision-Based Formation Control Framework
Aveek K. Das
University of Pennsylvania
Rafael Fierro
Oklahoma State University
R. Vijay Kumar
University of Pennsylvania
, kumar@grasp.upenn.edu
James P. Ostrowski
University of Pennsylvania
John Spletzer
University of Pennsylvania
See next page for additional authors
Follow this and additional works at: https://repository.upenn.edu/meam_papers
Recommended Citation Recommended Citation
Das, Aveek K.; Fierro, Rafael ; Kumar, R. Vijay; Ostrowski, James P.; Spletzer, John; and Taylor, Camillo J., "A
Vision-Based Formation Control Framework" (2002).
Departmental Papers (MEAM)
. 9.
https://repository.upenn.edu/meam_papers/9
Copyright 2002 IEEE. Reprinted from
IEEE Transactions on Robotics and Automation
, Volume 18, Issue 5, October
2002, pages 813-825.
Publisher URL: http://ieeexplore.ieee.org/xpl/tocresult.jsp?isNumber=22928&puNumber=70
This material is posted here with permission of the IEEE. Such permission of the IEEE does not in any way imply
IEEE endorsement of any of the University of Pennsylvania's products or services. Internal or personal use of this
material is permitted. However, permission to reprint/republish this material for advertising or promotional
purposes or for creating new collective works for resale or redistribution must be obtained from the IEEE by writing
to pubs-permissions@ieee.org. By choosing to view this document, you agree to all provisions of the copyright laws
protecting it.
This paper is posted at ScholarlyCommons. https://repository.upenn.edu/meam_papers/9
For more information, please contact repository@pobox.upenn.edu.
A Vision-Based Formation Control Framework A Vision-Based Formation Control Framework
Abstract Abstract
We describe a framework for cooperative control of a group of nonholonomic mobile robots that allows
us to build complex systems from simple controllers and estimators. The resultant modular approach is
attractive because of the potential for reusability. Our approach to composition also guarantees stability
and convergence in a wide range of tasks. There are two key features in our approach: 1) a paradigm for
switching between simple decentralized controllers that allows for changes in formation; 2) the use of
information from a single type of sensor, an omnidirectional camera, for all our controllers. We describe
estimators that abstract the sensory information at different levels, enabling both decentralized and
centralized cooperative control. Our results include numerical simulations and experiments using a
testbed consisting of three nonholonomic robots.
Keywords Keywords
Cooperative localization, formation control, hybrid control, nonholonomic mobile robots
Comments Comments
Copyright 2002 IEEE. Reprinted from
IEEE Transactions on Robotics and Automation
, Volume 18, Issue 5,
October 2002, pages 813-825.
Publisher URL: http://ieeexplore.ieee.org/xpl/tocresult.jsp?isNumber=22928&puNumber=70
This material is posted here with permission of the IEEE. Such permission of the IEEE does not in any way
imply IEEE endorsement of any of the University of Pennsylvania's products or services. Internal or
personal use of this material is permitted. However, permission to reprint/republish this material for
advertising or promotional purposes or for creating new collective works for resale or redistribution must
be obtained from the IEEE by writing to pubs-permissions@ieee.org. By choosing to view this document,
you agree to all provisions of the copyright laws protecting it.
Author(s) Author(s)
Aveek K. Das, Rafael Fierro, R. Vijay Kumar, James P. Ostrowski, John Spletzer, and Camillo J. Taylor
This journal article is available at ScholarlyCommons: https://repository.upenn.edu/meam_papers/9
IEEE TRANSACTIONS ON ROBOTICS AND AUTOMATION, VOL. 18, NO. 5, OCTOBER 2002 813
A Vision-Based Formation Control Framework
Aveek K. Das, Student Member, IEEE, Rafael Fierro, Member, IEEE, Vijay Kumar, Senior Member, IEEE,
James P. Ostrowski, Member, IEEE, John Spletzer, and Camillo J. Taylor, Member, IEEE
Abstract—We describe a framework for cooperative control of
a group of nonholonomic mobile robots that allows us to build
complex systems from simple controllers and estimators. The re-
sultant modular approach is attractive because of the potential
for reusability. Our approach to composition also guarantees sta-
bility and convergence in a wide range of tasks. There are two key
features in our approach: 1) a paradigm for switching between
simple decentralized controllers that allows for changes in forma-
tion; 2) the use of information from a single type of sensor, an
omnidirectional camera, for all our controllers. We describe es-
timators that abstract the sensory information at different levels,
enabling both decentralized and centralized cooperative control.
Our results include numerical simulations and experiments using
a testbed consisting of three nonholonomic robots.
Index Terms—Cooperative localization, formation control, hy-
brid control, nonholonomic mobile robots.
I. INTRODUCTION
T
HE LAST FEW years haveseenactive research in the field
of control and coordination for multiple mobile robots,
with applications including tasks such as exploration [1],
surveillance [2], search and rescue [3], mapping of unknown
or partially known environments, distributed manipulation
[4], [5], and transportation of large objects [6], [7]. While
robot control is considered to be a well-understood problem
area [8], [9], most of the current success stories in multirobot
coordination do not rely on or build on the results available
in the control theory and dynamical systems literature. This is
because traditional control theory primarily enables the design
of controllers in a single mode of operation, in which the task
and the model of the system are fixed [10]. When operating
in unstructured or dynamic environments with many different
sources of uncertainty, it is very difficult if not impossible to
design controllers that will guarantee performance even in a
local sense. In contrast, we know that one can readily design
Manuscript received March 22, 2002. This paper was recommended for
publication by Associate Editor T. Arai and Editor S. Hutchinson upon
evaluation of the reviewers’ comments. This work was supported by the
Defense Advanced Research Projects Agency ITO MARS Program under
Grant 130-1303-4-534328-xxxx-2000-0000, the Air Force Office of Scientific
Research under Grant F49620-01-1-0382, and the National Science Foundation
under Grant CDS-97-03220 and Grant IIS 987301. This paper was presented
in part at the IEEE International Conference on Robotics and Automation,
Seoul, Korea, May, 2001.
A. K. Das, V. Kumar, J. P. Ostrowski, J. Spletzer, and C. J. Taylor are with the
GRASP Laboratory, University of Pennsylvania, Philadelphia, PA 19104-6228
USA (e-mail: aveek@grasp.cis.upenn.edu; kumar@grasp.cis.upenn.edu;
jpo@grasp.cis.upenn.edu; spletzer@grasp.cis.upenn.edu; cjtaylor@grasp.cis.
upenn.edu).
R. Fierro is with the MARHES Laboratory, School of Electrical and Com-
puter Engineering, Oklahoma StateUniversity, Stillwater, OK74078-5032USA
(e-mail: rfierro@okstate.edu).
Digital Object Identifier 10.1109/TRA.2002.803463
reactive controllers or behaviors that react to simple stimuli
or commands from the environment. Successful applications
of this idea are found in subsumption architectures [11],
behavior-based robotics [12], and other works [13].
In this paper, we address the development of intelligent robot
systems by composing simple building blocks in a bottom-up
approach. The building blocks consist of controllers and esti-
mators, and the framework for composition allows for tightly
coupled perception-action loops. While this philosophy is sim-
ilar in spirit to a behavior-based control paradigm [12], we differ
in the more formal, control-theoretic approach in developing the
basic components and their composition.
The goal of this paper is to develop a framework for composi-
tion of simple controllers and estimators to control theformation
of a group of robots. By formation control, we simply mean the
problem of controlling the relative positions and orientations of
robots in a group, while allowing the group to move as a whole.
Problems in formation control that have been investigated in-
clude assignment of feasible formations [14], [15], moving into
formation [16], maintenance of formation shape [17], [18], and
switching between formations [19], [20]. Approaches to mod-
eling and solving these problems have been diverse, ranging
from paradigms based on combining reactive behaviors [12],
[21] to those based on leader-follower graphs [17], [19] and vir-
tual structures [22], [23].
We are particularlyinterested in applications like cooperative
manipulation, where a semirigid formation may be necessary to
transport a grasped object to a prescribed location, and coop-
erative mapping, where the formation may be defined by a set
of sensor constraints. We consider situations in which there is
no global positioning system and the main sensing modality is
vision. Our platform of interest is a car-like robot with a single
physical sensor, an omnidirectional camera.
Our contributions in this paper are two-fold. First, we de-
velop a control-theoretic bottom-up approach to building and
composing controllers and estimators. These include simple de-
centralized, reactivecontrollers for obstacle avoidance, collision
recovery, and pursuing targets, and more complex controllers
for maintaining formation. These controllers can be either cen-
tralized or decentralized and are derived from input–output lin-
earization [10]. Our second contribution is a framework for mul-
tirobot coordination that allows robots to maintain or change
formation while following a specified trajectory and to perform
cooperative manipulation tasks. Our framework involves a se-
quential composition of controllers, or modes, and we show that
the dynamics of the resulting switched system are stable and
converge to the desired formation.
The paper is organized as follows. In Section II, we state the
assumptions of our control framework and present details on our
1042-296X/02$17.00 © 2002 IEEE
814 IEEE TRANSACTIONS ON ROBOTICS AND AUTOMATION, VOL. 18, NO. 5, OCTOBER 2002
controllers for formation control. We discuss the assignment of
formations, changes in formations, and stable switching strate-
gies in Section III using a group of three robots as an example.
Section IV addresses our sensing and estimation schemes for
formation control. Hardware details and experimental results il-
lustrating the application of our multirobot coordination frame-
work are in Section V. Finally, in Section VI, we draw conclu-
sions and suggest future work.
II. C
ONTROL ALGORITHMS
Before describing the individual components of our control
framework, we list several important assumptions concerning
the group of robots and the formation. We assume, as in [17],
the robots are labeled and one of the robots, designated as
,
is the lead (or reference) robot. The lead robot’s motion de-
fines the bulk motion of the group. The motion of individual
members within the formation is then described in reference to
the lead robot. As in [17] and [19], the relationship between a
robot and its neighboring robots is described by a control graph.
The control graph is an acyclic, directed graph with robots as
nodes,
as the parent node, and edges directed from nodes
with smaller integer label values to those with with larger in-
teger values. Each edge denotes a constraint between the robots
connected by the edge and a controller that tries to maintain the
constraint. We present more details on control graphs in the fol-
lowing sections.
In this section, we describe control algorithms that specify
the interactions between each robot and its neighbor(s) or the
environment. The robots are velocity-controlled nonholonomic
car-like platforms and have two independent inputs. The control
laws are motivated by ideas from the well-established area of
input–output feedback linearization [10]. This means we can
regulate two outputs. The kinematics of the
th robot can be
abstracted as a unicycle model (other models can be adapted
to this framework)
(1)
where we let
, and and are the
linear and angular velocities, respectively.
A. Basic Leader-Following Control
We start with a simple leader-follower configuration (see
Fig. 1) (denoted
), in which robot follows with
a desired Separation
and desired relative Bearing . Note
that this relative bearing describes the heading direction of the
follower with respect to the leader. The two-robot system is
transformed into a new set of coordinates where the state of the
leader is treated as an exogenous input. Thus, the kinematic
equations are given by
(2)
where
is the system output,
is the relative orientation, is the input for ,
is ’s input, and
(a)
(b)
Fig. 1. Two robots using (a) basic leader-following controller and (b) the
leader-obstacle controller.
with . By applying input–output feedback lin-
earization, the control velocities for the follower are given by
(3)
where
is the offset to an off-axis reference point on the
robot and
is an auxiliary control input given by
and are the user-selected controller gains. The
closed-loop linearized system is simply given by
(4)
In the following, we prove that under suitable assumptions
on the motion of the lead robot, the closed-loop system is stable.
Since we are using input–output feedback linearization [10], the
output vector
will converge to the desired value arbi-
trarily fast. However, a complete stability analysis requires the
study of the internal dynamics of the robot, i.e., the relative ori-
entation
.
Theorem 1: Assume that the lead vehicle’s linear velocity
along the path
is lower bounded, i.e., , its
angular velocity is bounded, i.e.,
, and the initial
relative heading is bounded away from
, i.e., ,
for some
. If the control input (3) is applied to , then
the system described by (2) is stable and the output
in (4)
converges exponentially to the desired value
.
Proof: Let the system error
be de-
fined as
(5)
By looking at (4), we have that
and converge to zero ex-
ponentially. Then, we need to show that the internal dynamics
DAS et al.: A VISION-BASED FORMATION CONTROL FRAMEWORK 815
of are stable, which is equivalent to showing that the orien-
tation error
is bounded. Thus, we have
and, after some algebraic simplification, we obtain
(6)
where
The nominal system, i.e., is given by
(7)
which is (locally) exponentially stable if the velocity of the lead
robot
and . Since is bounded, one can
show that
. Using stability theory of perturbed
systems [10] and the condition
, gives [20]
for some finite time and positive number .
Remark 1: The above theorem shows that, under some rea-
sonable assumptions, the two-robot system is stable, i.e., there
exists a Lyapunov function
, where
and , such that .
We can study some particular formations of practical interest.
For example,if the leader travels in a straight line, i.e.,
,it
can be shown that the system is (locally) asymptotically stable,
i.e.,
as , provided that and
.If is constant (circular motion), then is bounded. It is
well known that an optimal nonholonomic path can be planned
by joining linear and circular trajectory segments. Hence, any
trajectory generated by such a planner for the leader will ensure
stable leader-follower dynamics using the above controller.
Remark 2: This result can be extended to
robots in an
inline, convoy-like formation where
follows under
. If each successive leader’s trajectory satisfies the
assumptions of Theorem 1, then the convoy-like system can
be shown to be stable. We will provide some more insight into
stabilizing
robot formations at the end of this section.
B. Leader-Obstacle Control
This controller (denoted
) allows the follower to avoid
obstacles while following a leader with a desired separation.
Thus, the outputs of interest are the separation
and the dis-
tance
between the reference point on the follower, and the
closest point
on the object. We define a virtual robot as
shownin Fig. 1 (right), which moveson the obstacle’s boundary.
We define
as the heading of the virtual robot, which is defined
locally by the tangent to the obstacle’s boundary. Our previous
estimation strategies for wall following [24] can be adapted to
recover the relative orientation to the closest sensed section of
the object’s boundary. For this case, the kinematic equations are
given by
(8)
where
is the system output, is
the input for
, and , . By applying
Fig. 2. Three-robot formation controller.
input–output feedback linearization as above, but replacing the
auxiliary control input,
, with , given by
( , are controller gains), the closed-loop linearized
system is given by
(9)
Remark 3: It is worth noting that feedback input–output lin-
earization is possible as long as
, i.e., the
controller is not defined if
. This occurs
when vectors
and are collinear, which should never
happen in practice.
Remark 4: By using this controller, a follower robot will
avoid the nearest obstacle within its field of view while keeping
a desired distance from the leader. This is a reasonable assump-
tion for many outdoor environments of practical interest. While
there are obvious limitations to this scheme in maze-like envi-
ronments, it is not difficult to characterize the set of obstacles
and leader trajectories for which this scheme will work.
C. Three-Robot Shape Control
Consider a formation of three nonholonomic robots labeled
, , and (see Fig. 2). There are several possible ap-
proaches to controlling the formation. For example, one could
use two basic lead-follower controllers: either
with
,or with . Another approach that is
more robust to noise is to use a three-robot formation shape
controller (denoted
), that has robot follow both
and with desired separations and , respectively,
while
follows with . Again, the kinematic
equations are given by
(10)
where
is the system output,
is the input vector, and