IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, VOL. 22, NO. 1, JANUARY 2004 1
Tapestry: A Resilient Global-scale Overlay for
Service Deployment
Ben Y. Zhao, Ling Huang, Jeremy Stribling, Sean C. Rhea, Anthony D. Joseph, Member, IEEE, and
John D. Kubiatowicz, Member, IEEE
Abstract— We present Tapestry, a peer-to-peer overlay
routing infrastructure offering efficient, scalable, location-
independent routing of messages directly to nearby copies
of an object or service using only localized resources.
Tapestry supports a generic Decentralized Object Location
and Routing (DOLR) API using a self-repairing, soft-
state based routing layer. This paper presents the Tapestry
architecture, algorithms, and implementation. It explores
the behavior of a Tapestry deployment on PlanetLab, a
global testbed of approximately 100 machines. Experimen-
tal results show that Tapestry exhibits stable behavior and
performance as an overlay, despite the instability of the
underlying network layers. Several widely-distributed ap-
plications have been implemented on Tapestry, illustrating
its utility as a deployment infrastructure.
Index Terms— Overlay networks, peer-to-peer (P2P),
service deployment, Tapestry.
I. INTRODUCTION
Internet developers are constantly proposing new and
visionary distributed applications. These new applica-
tions have a variety of requirements for availability,
durability, and performance. One technique for achieving
these properties is to adapt to failures or changes in load
through migration and replication of data and services.
Unfortunately, the ability to place replicas or the fre-
quency with which they may be moved is limited by
underlying infrastructure. The traditional way to deploy
new applications is to adapt them somehow to existing
infrastructures (often an imperfect match) or to standard-
ize new Internet protocols (encountering significant iner-
tia to deployment). A flexible but standardized substrate
on which to develop new applications is needed.
In this article, we present Tapestry [1], [2], an extensi-
ble infrastructure that provides Decentralized Object Lo-
cation and Routing (DOLR) [3]. The DOLR interface fo-
cuses on routing of messages to endpoints such as nodes
This paper was supported in part by the National Science Foundation
(NSF) under Career Award #ANI-9985129 and Career Award #ANI-
9985250, in part by the NSF Information Technology Research (ITR)
under Award 5710001344, in part by the California Micro Fund under
Award 02-032 and Award 02-035, and in part by Grants from IBM and
Sprint. B. Y. Zhao, L. Huang, S. C. Rhea, A. D. Joseph, and J. D. Kubi-
atowicz are with the University of California, Berkeley, CA 94720 USA
(e-mail:
ravenben, hling, srhea, adj, kubitron
@eecs.berkeley.edu). J.
Stribling is with Massachusetts Institute of Technology, Cambridge,
MA 02139 USA (e-mail: strib@mit.edu).
or object replicas. DOLR virtualizes resources, since
endpoints are named by opaque identifiers encoding
nothing about physical location. Properly implemented,
this virtualization enables message delivery to mobile
or replicated endpoints in the presence of instability
in the underlying infrastructure. As a result, a DOLR
network provides a simple platform on which to im-
plement distributed applications—developers can ignore
the dynamics of the network except as an optimization.
Already, Tapestry has enabled the deployment of global-
scale storage applications such as OceanStore [4] and
multicast distribution systems such as Bayeux [5]; we
return to this in Section VI.
Tapestry is a peer-to-peer overlay network that
provides high-performance, scalable, and location-
independent routing of messages to close-by endpoints,
using only localized resources. The focus on routing
brings with it a desire for efficiency: minimizing message
latency and maximizing message throughput. Thus, for
instance, Tapestry exploits locality in routing messages
to mobile endpoints such as object replicas; this behavior
is in contrast to other structured peer-to-peer overlay
networks [6]–[11].
Tapestry uses adaptive algorithms with soft-state to
maintain fault-tolerance in the face of changing node
membership and network faults. Its architecture is mod-
ular, consisting of an extensible upcall facility wrapped
around a simple, high-performance router. This Applica-
tions Programming Interface (API) enables developers to
develop and extend overlay functionality when the basic
DOLR functionality is insufficient.
In the following pages, we describe a Java-based
implementation of Tapestry, and present both micro-
and macro-benchmarks from an actual, deployed sys-
tem. During normal operation, the relative delay penalty
(RDP)
1
to locate mobile endpoints is two or less in the
wide area. Simulations show that Tapestry operations
succeed nearly 100% of the time under both constant
network changes and massive failures or joins, with
small periods of degraded performance during self-
1
RDP, or stretch, is the ratio between the distance traveled by a
message to an endpoint and the minimal distance from the source to
that endpoint.
2 IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, VOL. 22, NO. 1, JANUARY 2004
repair. These results demonstrate Tapestry’s feasibility
as a long running service on dynamic, failure-prone
networks such as the wide-area Internet.
The following section discusses related work. Then,
Tapestry’s core algorithms appear in Section III, with
details of the architecture and implementation in Sec-
tion IV. Section V evaluates Tapestry’s performance. We
then discuss the use of Tapestry as an application infras-
tructure in Section VI and conclude with Section VII.
II. R
ELATED WORK
The first generation of peer-to-peer (P2P) systems
included file-sharing and storage applications: Napster,
Gnutella, MojoNation, and Freenet. Napster uses central
directory servers to locate files. Gnutella provides a
similar, but distributed service using scoped broadcast
queries, limiting scalability. MojoNation [12] uses an on-
line economic model to encourage sharing of resources.
Freenet [13] is a file-sharing network designed to resist
censorship. Neither Gnutella nor Freenet guarantee that
files can be located—even in a functioning network.
The second generation of peer-to-peer systems are
structured peer-to-peer overlay networks, including
Tapestry [1], [2], Chord [8], Pastry [7], and CAN [6].
These overlays implement a basic Key-Based Routing
(KBR) interface, that supports deterministic routing of
messages to a live node that has responsibility for the
destination key. They can also support higher level
interfaces such as a distributed hash table (DHT) or a de-
centralized object location and routing (DOLR) layer [3].
These systems scale well, and guarantee that queries find
existing objects under non-failure conditions.
One differentiating property between these systems
is that neither CAN nor Chord take network distances
into account when constructing their routing overlay;
thus, a given overlay hop may span the diameter of the
network. Both protocols route on the shortest overlay
hops available, and use runtime heuristics to assist. In
contrast, Tapestry and Pastry construct locally optimal
routing tables from initialization, and maintain them in
order to reduce routing stretch.
While some systems fix the number and location
of object replicas by providing a distributed hash ta-
ble (DHT) interface, Tapestry allows applications to
place objects according to their needs. Tapestry “pub-
lishes” location pointers throughout the network to fa-
cilitate efficient routing to those objects with low net-
work stretch. This technique makes Tapestry locality-
aware [14]: queries for nearby objects are generally
satisfied in time proportional to the distance between the
query source and a nearby object replica.
Both Pastry and Tapestry share similarities to the
work of Plaxton, Rajamaran, and Richa [15] for a static
network. Others [16], [17] explore distributed object
location schemes with provably low search overhead,
but they require precomputation, and so are not suitable
for dynamic networks. Recent works include systems
such as Kademlia [9], which uses XOR for overlay
routing, and Viceroy [10], which provides logarithmic
hops through nodes with constant degree routing tables.
SkipNet [11] uses a multi-dimensional skip-list data
structure to support overlay routing, maintaining both
a DNS-based namespace for operational locality and
a randomized namespace for network locality. Other
overlay proposals [18], [19] attain lower bounds on local
routing state. Finally, proposals such as Brocade [20]
differentiate between local and inter-domain routing to
reduce wide-area traffic and routing latency.
A new generation of applications have been pro-
posed on top of these P2P systems, validating them
as novel application infrastructures. Several systems
have application level multicast: CAN-MC [21] (CAN),
Scribe [22] (Pastry), and Bayeux [5] (Tapestry). In
addition, several decentralized file systems have been
proposed: CFS [23] (Chord), Mnemosyne [24] (Chord,
Tapestry), OceanStore [4] (Tapestry), and PAST [25]
(Pastry). Structured P2P overlays also support novel
applications (e.g., attack resistant networks [26], network
indirection layers [27], and similarity searching [28]).
III. T
APESTRY ALGORITHMS
This section details Tapestry’s algorithms for routing
and object location, and describes how network integrity
is maintained under dynamic network conditions.
A. The DOLR Networking API
Tapestry provides a datagram-like communications
interface, with additional mechanisms for manipulating
the locations of objects. Before describing the API, we
start with a couple of definitions.
Tapestry nodes participate in the overlay and are
assigned nodeIDs uniformly at random from a large
identifier space. More than one node may be hosted
by one physical host. Application-specific endpoints
are assigned Globally Unique IDentifiers (GUIDs), se-
lected from the same identifier space. Tapestry currently
uses an identifier space of 160-bit values with a glob-
ally defined radix (e.g., hexadecimal, yielding 40-digit
identifiers). Tapestry assumes nodeIDs and GUIDs are
roughly evenly distributed in the namespace, which can
be achieved by using a secure hashing algorithm like
SHA-1 [29]. We say that node N has nodeID N
, and
an object O has GUID O
.
Since the efficiency of Tapestry generally improves
with network size, it is advantageous for multiple appli-
cations to share a single large Tapestry overlay network.
ZHAO et al.: TAPESTRY: A RESILIENT GLOBAL-SCALE OVERLAY FOR SERVICE DEPLOYMENT 3
L4
42A2
1D76
27AB
4228
4227
44AF
6F43
43C9
51E5
L1
L1
L1
L1
L2
L2
L3
Fig. 1. Tapestry routing mesh from the perspective of a
single node. Outgoing neighbor links point to nodes with
a common matching prefix. Higher-level entries match more
digits. Together, these links form the local routing table.
L4
L2
L1
L3
L4
L4
L3
L2
L3
L2
5230
L1
AC78
4227
42A2
42AD
4629
400F
42A7
4112
42A9
4211
42E0
Fig. 2. Path of a message. The path taken by a message
originating from node 5230 destined for node 42AD in a
Tapestry mesh.
To enable application coexistence, every message con-
tains an application-specific identifier, A
, which is used
to select a process, or application for message delivery at
the destination (similar to the role of a port in TCP/IP),
or an upcall handler where appropriate.
Given the above definitions, we state the four-part
DOLR networking API as follows:
1) P
UBLISHOBJECT(O
, A
): Publish, or make
available, object
on the local node. This call
is best effort, and receives no confirmation.
2) U
NPUBLISHOBJECT(O
, A
): Best-effort attempt
to remove location mappings for
.
3) R
OUTETOOBJECT(O
, A
): Routes message to
location of an object with GUID O
.
4) R
OUTETONODE(N, A
, Exact): Route message
to application A
on node N. “Exact” specifies
whether destination ID needs to be matched ex-
actly to deliver payload.
B. Routing and Object Location
Tapestry dynamically maps each identifier
to a
unique live node, called the identifier’s root or
.If
a node
exists with N
=
, then this node is the root
of
. To deliver messages, each node maintains a routing
table consisting of nodeIDs and IP addresses of the nodes
with which it communicates. We refer to these nodes as
NEXTHOP (
1 if
MAXHOP
then
2 return self
3 else
4
;
5 while
= nil do
6
7
8 endwhile
9 if
then
10 return
NEXTHOP (
,
)
11 else
12 return
13 endif
14 endif
Fig. 3. Pseudocode for NEXTHOP(). This function locates the
next hop towards the root given the previous hop number,
,
and the destination GUID,
. Returns next hop or self if local
node is the root.
neighbors of the local node. When routing toward
,
messages are forwarded across neighbor links to nodes
whose nodeIDs are progressively closer (i.e., matching
larger prefixes) to
in the ID space.
1) Routing Mesh: Tapestry uses local tables at each
node, called neighbor maps, to route overlay messages to
the destination ID digit by digit (e.g., 4***
42**
42A*
42AD, where *’s represent wildcards).
This approach is similar to longest prefix routing used
by CIDR IP address allocation [30]. A node N has a
neighbor map with multiple levels, where each level
contains links to nodes matching a prefix up to a digit
position in the ID, and contains a number of entries equal
to the ID’s base. The primary
entry in the
level
is the ID and location of the closest node that begins
with prefix(
,
)+“
”(e.g., the
entry of the
level for node 325AE is the closest node with an ID
that begins with 3259. It is this prescription of “closest
node” that provides the locality properties of Tapestry.
Figure 1 shows some of the outgoing links of a node.
Figure 2 shows a path that a message might take
through the infrastructure. The router for the
hop
shares a prefix of length
with the destination ID;
thus, to route, Tapestry looks in its (
level map
for the entry matching the next digit in the destination
ID. This method guarantees that any existing node in the
system will be reached in at most
logical hops,
in a system with namespace size
, IDs of base
, and
assuming consistent neighbor maps. When a digit cannot
be matched, Tapestry looks for a “close” digit in the
routing table; we call this surrogate routing [1], where
each non-existent ID is mapped to some live node with
a similar ID. Figure 3 details the
NEXTHOP function for
chosing an outgoing link. It is this dynamic process that
maps every identifier
to a unique root node
.
4 IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, VOL. 22, NO. 1, JANUARY 2004
4228
4A6D
4361
43FE
Location Mappin
g
Publish Path
4377
437A
(4378)
Phil’s
Books
AA93
(4378)
Phil’s
Books
Tapestry Pointer
s
4664
4B4F
57EC
E791
4228
4A6D
4361
43FE
Location Mappin
g
4377
437A
(4378)
Phil’s
Books
AA93
(4378)
Phil’s
Books
Tapestry Pointer
s
4664
4B4F
57EC
E791
Query Path
Fig. 4. Tapestry object publish example. Two copies of an
object (4378) are published to their root node at 4377.
Publish messages route to root, depositing a location pointer
for the object at each hop encountered along the way.
Fig. 5. Tapestry route to object example. Several nodes send
messages to object 4378 from different points in the network.
The messages route towards the root node of 4378. When they
intersect the publish path, they follow the location pointer to
the nearest copy of the object.
The challenge in a dynamic network environment is to
continue to route reliably even when intermediate links
are changing or faulty. To help provide resilience, we
exploit network path diversity in the form of redundant
routing paths. Primary neighbor links shown in Figure 1
are augmented by backup links, each sharing the same
prefix
2
. At the
routing level, the
neighbor links
differ only on the
digit. There are
pointers
on a level, and the total size of the neighbor map is
. Each node also stores reverse refer-
ences (backpointers) to other nodes that point at it. The
expected total number of such entries is
.
2) Object Publication and Location: As shown
above, each identifier
has a unique root node
assigned by the routing process. Each such root node in-
herits a unique spanning tree for routing, with messages
from leaf nodes traversing intermediate nodes en route
to the root. We utilize this property to locate objects by
distributing soft-state directory information across nodes
(including the object’s root).
A server S, storing an object O (with GUID, O
,
and root, O
3
), periodically advertises or publishes this
object by routing a publish message toward O
(see
Figure 4). In general, the nodeID of O
is different from
O
; O
is the unique [2] node reached through surrogate
routing by successive calls to
NEXTHOP(*, O
). Each
node along the publication path stores a pointer mapping,
O
, S
, instead of a copy of the object itself. When
there are replicas of an object on separate servers, each
server publishes its copy. Tapestry nodes store location
mappings for object replicas in sorted order of network
latency from themselves.
A client locates O by routing a message to O
(see
2
Current implementations keep two additional backups.
3
Note that objects can be assigned multiple GUIDs mapped to
different root nodes for fault-tolerance.
Figure 5). Each node on the path checks whether it has
a location mapping for O. If so, it redirects the message
to S. Otherwise, it forwards the message onwards to O
(guaranteed to have a location mapping).
Each hop towards the root reduces the number of
nodes satisfying the next hop prefix constraint by a factor
of the identifier base. Messages sent to a destination
from two nearby nodes will generally cross paths quickly
because: each hop increases the length of the prefix
required for the next hop; the path to the root is a
function of the destination ID only, not of the source
nodeID (as in Chord); and neighbor hops are chosen
for network locality, which is (usually) transitive. Thus,
the closer (in network distance) a client is to an object,
the sooner its queries will likely cross paths with the
object’s publish path, and the faster they will reach the
object. Since nodes sort object pointers by distance to
themselves, queries are routed to nearby object replicas.
C. Dynamic Node Algorithms
Tapestry includes a number of mechanisms to main-
tain routing table consistency and ensure object availabil-
ity. In this section, we briefly explore these mechanisms.
See [2] for complete algorithms and proofs. The majority
of control messages described here require acknowledg-
ments, and are retransmitted where required.
1) Node Insertion: There are four components to
inserting a new node N into a Tapestry network:
a) Need-to-know nodes are notified of N, because N
fills a null entry in their routing tables.
b) N might become the new object root for existing
objects. References to those objects must be moved
to N to maintain object availability.
c) The algorithms must construct a near optimal
routing table for N.
ZHAO et al.: TAPESTRY: A RESILIENT GLOBAL-SCALE OVERLAY FOR SERVICE DEPLOYMENT 5
d) Nodes near N are notified and may consider using
N in their routing tables as an optimization.
Node insertion begins at N’s surrogate S (the “root” node
that N
maps to in the existing network). S finds
,
the length of the longest prefix its ID shares with N
.
S sends out an Acknowledged Multicast message that
reaches the set of all existing nodes sharing the same
prefix by traversing a tree based on their nodeIDs. As
nodes receive the message, they add N to their routing
tables and transfer references of locally rooted pointers
as necessary, completing items (a) and (b).
Nodes reached by the multicast contact N and become
an initial neighbor set used in its routing table con-
struction. N performs an iterative nearest neighbor search
beginning with routing level
. N uses the neighbor set to
fill routing level
, trims the list to the closest
nodes
4
,
and requests these
nodes send their backpointers (see
Section III-B) at that level. The resulting set contains all
nodes that point to any of the
nodes at the previous
routing level, and becomes the next neighbor set. N then
decrements
, and repeats the process until all levels are
filled. This completes item (c). Nodes contacted during
the iterative algorithm use N to optimize their routing
tables where applicable, completing item (d).
To ensure that nodes inserting into the network in
unison do not fail to notify each other about their
existence, every node
in the multicast keeps state on
every node
that is still multicasting down one of its
neighbors. This state is used to tell each node
with
in its multicast tree about
. Additionally, the multicast
message includes a list of holes in the new node’s routing
table. Nodes check their tables against the routing table
and notify the new node of entries to fill those holes.
2) Voluntary Node Deletion: If node N leaves
Tapestry voluntarily, it tells the set
of nodes in N’s
backpointers of its intention, along with a replacement
node for each routing level from its own routing table.
The notified nodes each send object republish traffic
to both N and its replacement. Meanwhile, N routes
references to locally rooted objects to their new roots,
and signals nodes in
when finished.
3) Involuntary Node Deletion: In a dynamic, failure-
prone network such as the wide-area Internet, nodes
generally exit the network far less gracefully due to node
and link failures or network partitions, and may enter and
leave many times in a short interval. Tapestry improves
object availability and routing in such an environment
by building redundancy into routing tables and object
location references (e.g., the
backup forwarding
pointers for each routing table entry). Ongoing work
4
is a knob for tuning the tradeoff between resources used and
optimality of the resulting routing table.
Management
Dynamic Node
Object Pointer Database
and
Routing Table
Router
Decentralized
File System
Multicast
Application−Level
Collaborative
Text Filtering
Application Interface / Upcall API
Neighbor Link Management
Transport Protocols
Single Tapestry Node
Fig. 6. Tapestry component architecture. Messages pass up
from physical network layers and down from application
layers. The Router is a central conduit for communication.
has shown Tapestry’s viability as a resilient routing
layer [31].
To maintain availability and redundancy, nodes use
periodic beacons to detect outgoing link and node fail-
ures. Such events trigger repair of the routing mesh and
initiate redistribution and replication of object location
references. Furthermore, the repair process is augmented
by soft-state republishing of object references. Tapestry
repair is highly effective, as shown in Section V-C. De-
spite continuous node turnover, Tapestry retains nearly
a 100% success rate at routing messages to nodes and
objects.
IV. T
APESTRY NODE ARCHITECTURE AND
IMPLEMENTATION
In this section, we present the architecture of a
Tapestry node, an API for Tapestry extension, details
of our current implementation, and an architecture for a
higher-performance implementation suitable for use on
network processors.
A. Component Architecture
Figure 6 illustrates the functional layering for a
Tapestry node. Shown on top are applications that in-
terface with the rest of the system through the Tapestry
API. Below this are the router and the dynamic node
management components. The former processes routing
and location messages, while the latter handles the
arrival and departure of nodes in the network. These two
components communicate through the routing table. At
the bottom are the transport and neighbor link layers,
which together provide a cross-node messaging layer.
We now describe several of these layers.
1) Transport: The transport layer provides the ab-
straction of communication channels from one overlay
node to another, and corresponds to layer 4 in the OSI
layering. Utilizing native Operating System (OS) func-
tionality, many channel implementations are possible.