PIC: Practical Internet Coordinates for Distance Estimation
Manuel Costa, Miguel Castro, Antony Rowstron, and Peter Key
Microsoft Research, Cambridge, CB3 0FB, UK
Abstract
This paper introduces PIC, a practical coordinate-based
mechanism to estimate Internet network distance (i.e.,
round-trip delay or network hops). Network distance es-
timation is important in many applications, for example,
network-aware overlay construction and server selec-
tion. There are several proposals for distance estima-
tion in the Internet but they all suffer from problems that
limit their benefit. Most rely on a small set of infrastruc-
ture nodes that are a single point of failure and limit
scalability. Others use sets of peers to compute coordi-
nates but these coordinates can be arbitrarily wrong if one
of these peers is malicious. While it may be reasonable to
secure a small set of infrastructure nodes, it is unreason-
able to secure all peers. PIC addresses these problems:
it does not rely on infrastructure nodes and it can com-
pute accurate coordinates even when some peers are mali-
cious. We present PIC’s design, experimental evaluation,
and an application to network-aware overlay construc-
tion and maintenance.
1. Introduction
An efficient mechanism to estimate distance in the Inter-
net could benefit many large scale distributed applications.
For example, the performance of overlay networks can be
greatly improved by exploiting information about the un-
derlying network but the cost of sending probe messages to
estimate distances can be significant (e.g., [14, 24, 30]).
This paper introduces PIC, a practical coordinate-based
mechanism to estimate distances in the Internet. PIC assigns
a point in a d-dimensional Euclidean space to each node and
uses the distance between two points in the space as an es-
timate of the network distance between the corresponding
nodes. Nodes compute their coordinates in the Euclidean
space when they join the system. Given the coordinates for
two nodes, any node can predict the distance between them.
In the last few years there has been much interest in this
area, but previous proposals for network distance estima-
tion suffer from problems that limit their practicality. Most
proposals [9, 17, 13, 22] rely on a small set of infrastruc-
ture nodes that are a single point of failure and can limit
scalability if they become communication bottlenecks. For
example, GNP [17], which pioneered coordinate-based dis-
tance estimation, uses a set of fixed landmark nodes that are
probed whenever a node joins the system.
Other proposals [28, 19, 8] use sets of peer nodes in the
system to compute each node’s coordinates but they are vul-
nerable to malicious peers that can cause coordinates to be
arbitrarily wrong. While it may be reasonable to secure a
small set of infrastructure nodes, it is unreasonable to as-
sume that no peers will behave maliciously.
PIC addresses these problems. It scales well because it
does not rely on infrastructure nodes; any node whose coor-
dinates have already been computed can act as a landmark.
Therefore, it can distribute communication and computa-
tion load evenly over all the nodes in a system. Addition-
ally, it computes coordinates efficiently and we describe a
technique for choosing landmark nodes that can predict dis-
tances as accurately as GNP. Finally, PIC can compute ac-
curate coordinates even when some peers are malicious.
The rest of the paper is organised as follows. Section 2
describes PIC and presents some results. Section 3 describes
how to implement PIC efficiently. Section 4 describes and
evaluates the PIC approach to security and Section 5 dis-
cusses an application of PIC in proximity-aware overlays.
Section 6 presents related work and Section 7 concludes.
2. PIC coordinate computation
PIC maps each node to a point in a d-dimensional Eu-
clidean space. When a node n joins the system, it com-
putes the coordinates of its corresponding point. It probes
the network distance to each element of a set of landmarks,
L, where L must have at least d + 1 members. Then it ob-
tains the coordinates of each landmark, and uses a multi-
dimensional global optimization algorithm (e.g., Simplex
Downhill [16]) to compute its coordinates such that the er-
rors in the |L| predicted distances between n and each node
in L are minimized. The errors are computed using the mea-
sured and estimated distances. The probe could use ICMP,
application-level round-trip time, or number of IP hops.
This is similar to GNP [17] but GNP uses a fixed set L for
all the nodes that join the system.
In PIC, the joining node can pick any node whose coor-
dinates have already been computed to be a landmark. Let
N be the set of nodes whose coordinates have already been
computed. When a node n joins the system, it can select
any set L that is a subset of N with size |L| > d. We exper-
imented with three different strategies to choose L:
• random: pick the elements of L randomly with uni-
form probability from N;
• closest: pick the elements of L to be the elements of N
closest to n in the network topology;
• hybrid: pick some elements as in random and others as
in closest.
We define L for a node n to be the union of two sets L
r
and L
c
. The elements in L
r
are chosen randomly from N,
whilst the elements in L
c
are the |L
c
| members of N which
are closest to node n in the network.
When bootstrapping the behaviour of the system is
slightly different. If |N| < |L|, n selects all the nodes in N.
Then it obtains the measured distances between all pairs of
nodes in N (a |N | × |N | matrix). In this case, the global op-
timization algorithm computes new coordinates for all the
nodes in N by minimizing the error in the predicted dis-
tances between all pairs of nodes in N ∪ {n}.
We use the Simplex Downhill [16] algorithm to compute
coordinates as in [17]. We experimented with several tar-
get error functions to minimize. The one that performed the
best was the sum of the squares of the relative errors:
|L|
X
i=1
(
d
m
i
− d
p
i
d
m
i
)
2
where, d
m
i
is the distance measured between node n and the
ith node in L and d
p
i
is the distance predicted between node
n and the ith node in L.
The intuition behind the different strategies to choose L
in PIC is the following. The closest strategy should pro-
vide the Simplex algorithm with better information to po-
sition the joining node correctly in the Euclidean space rel-
ative to nearby nodes in the network. The random strategy
should provide the Simplex algorithm with better informa-
tion to position the joining node correctly in the Euclidean
space relative to distant nodes in the network. Therefore,
the closest strategy should achieve lower relative errors
when predicting short distances whereas the random strat-
egy should achieve lower relative errors when predicting
long distances. The hybrid strategy should achieve some-
thing in the middle. The experimental results in the next
section confirm this intuition and show that the hybrid strat-
egy achieves lower relative errors than the other strategies.
The current version of Lighthouses [19] uses the random
strategy to select L. The closest strategy is similar to the ap-
proach used in Mithos [28].
The closest and hybrid strategies require a mechanism to
find the closest nodes to a node in the network. This can
be done in several ways, e.g., using expanding ring multi-
cast or the algorithms described in [15, 28, 2, 3, 12]. In Sec-
tion 3, we describe an efficient closest node discovery algo-
rithm. The algorithm to find the closest nodes in Mithos is
significantly more expensive than ours.
2.1. Experimental evaluation
We ran a number of experiments to evaluate the differ-
ent strategies to pick landmarks in PIC and to compare them
with GNP.
2.1.1. Experimental setup The experiments used net-
work distance data from the following three network
topologies. In each topology, there is a core set of routers
and we ran PIC on 40,000 end nodes that were ran-
domly assigned to routers in the core with uniform proba-
bility. Each end node was directly attached by a LAN link
with a 1ms delay to its assigned router.
GATech is a transit-stub topology generated with the Geor-
gia Tech [29] random graph generator. This network topol-
ogy has 5050 routers arranged hierarchically. There are 10
transit domains at the top level with an average of 5 routers
in each. Each transit router has an average of 10 stub do-
mains attached, and each stub has an average of 10 routers.
The network distance in this topology is round-trip delay.
The delay between routers is computed by the graph gen-
erator and routing is performed using the routing policy
weights of the graph generator. As in the real Internet, the
triangle inequality does not hold for all round trip times
among end nodes in this topology.
Mercator is a topology with 102,639 routers and it was ob-
tained from real measurements of the Internet using the
Mercator system [10]. The authors of [27] used real data
and some simple heuristics to assign an autonomous system
to each router. The resulting AS overlay has 2,662 nodes.
Routing is performed hierarchically as in the Internet. A
route follows the shortest path in the AS overlay between
the AS of the source and the AS of the destination. The
routes within each AS follow the shortest path to a router
in the next AS of the AS overlay path. The network dis-
tance in this topology is the number of hops in the route.
CorpNet is a topology with 298 routers and is generated us-
ing real measurements of the World-Wide Microsoft Cor-
porate network. The network distance in this topology is
the minimum round-trip delay.
We experimented with different values for the number
of landmarks and dimensions. Increasing the number of di-
mensions improves accuracy for GNP and all PIC strate-
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 50 100 150 200
Relative error (%)
Fraction of distances
GNP
random
closest
hybrid
Figure 1. Cumulative distribution of relative
errors over random distances in GATech.
gies but we did not observe any benefit above 12 dimen-
sions. Increasing |L| also improves accuracy for GNP and
the random strategy but has little effect for closest and hy-
brid. The experiments described here used d = 8 dimen-
sions and |L| = 16 landmarks.
GNP relies on a fixed set of landmarks so its accuracy
is very sensitive to their placement in the network. To pro-
vide a fair comparison between PIC and GNP, we ran an op-
timization procedure to determine the best landmark place-
ment for GNP on each topology. We ran 100 different GNP
experiments with 1,000 nodes, 1,000 test distances, and dif-
ferent randomly picked sets of landmarks L. The mean rela-
tive error in these experiments varied significantly: it varied
between 0.17 and 0.29 in GATech, between 0.17 and 0.23
in Mercator, and between 0.11 and 0.32 in CorpNet. The re-
sults that we present for GNP were obtained using the land-
marks that produced the minimum average relative error.
The average relative error obtained using PIC did not vary
significantly across these experiments, which is a desirable
property.
2.1.2. Results Figures 1, 2, and 3 show the cumulative
distribution of relative errors in 100,000 random test dis-
tances for the GATech, Mercator, and CorpNet topologies.
Each figure has lines for GNP and each of the PIC strategies.
The hybrid strategy used 4 nearby landmarks and 12 ran-
domly selected ones. The results show that PIC can match
the accuracy of GNP with the hybrid strategy but performs
significantly worse using either the random or closest strate-
gies. It is interesting to note that GNP is using an optimized
landmark placement; PIC with hybrid provided better accu-
racy than GNP before we optimized landmark placement.
Figure 4 provides some intuition to explain the previ-
ous results. It shows the cumulative distribution of rela-
tive errors for short test distances. For this experiment, we
randomly selected 2,000 nodes and for each node we gen-
erated test distances between the node and the 50 closest
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 50 100 150 200
Relative error (%)
Fraction of distances
GNP
random
closest
hybrid
Figure 2. Cumulative distribution of relative
errors over random distances in Mercator.
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 50 100 150 200
Relative error (%)
Fraction of distances
GNP
random
closest
hybrid
Figure 3. Cumulative distribution of relative
errors over random distances in CorpNet.
nodes in the network. This generated a test set of 100,000
test distances. The results show that closest performs signif-
icantly better than GNP in this range whereas random per-
forms badly. They support the intuition that picking land-
marks that are close on the network reduces relative errors
in the prediction of short distances. Since closest does worse
than GNP over random test distances, this also shows that
using distant random landmarks reduces relative errors in
the prediction of long distances. It is unclear why the GNP
curve has a discontinuity around 100%. We suspect that this
is due to the delay discontinuity between LAN and inter-
router links.
The results also show that hybrid performs similarly to
closest over short distances. Therefore, using 4 nearby land-
marks appears to be a good configuration; it achieves the
benefits of closest over short distances and matches the per-
formance of GNP over random test distances.
Table 1 shows summary metrics for the relative error dis-
tributions of 100,000 random test distances in all topologies
for GNP and all PIC strategies. The results are qualitatively
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 50 100 150 200
Relative error (%)
Fraction of distances
closest
hybrid
GNP
random
Figure 4. Cumulative distribution of relative
errors over short distances in GATech.
similar across all topologies. They are even quantitatively
very similar for the mean and 90-th percentile of GNP and
hybrid on the GATech and Mercator topologies.
GNP random closest hybrid
GATech Max 19.11 391.60 1.62 2.13
90-th 0.37 0.75 0.51 0.38
Mean 0.17 0.41 0.26 0.17
Mercator Max 2.50 8.63 1.55 1.64
90-th 0.37 0.58 0.57 0.37
Mean 0.17 0.28 0.36 0.17
CorpNet Max 13.34 404.92 1.00 1.33
90-th 0.23 0.60 0.47 0.23
Mean 0.11 0.61 0.20 0.09
Table 1. Relative error distribution sum-
maries.
The results also show that the closest and hybrid strate-
gies reduce the maximum relative error significantly rela-
tive to the random strategy and GNP, which is not surpris-
ing given that this is likely to correspond to a short distance
and they perform well over short distances.
3. Finding close nodes
The previous section shows that PIC performs best using
the hybrid strategy but this strategy requires some mecha-
nism to find the closest nodes in the network. In the pre-
vious section, we used an oracle to find the closest nodes
with global knowledge. This section describes algorithms
that can be used to implement this oracle efficiently in a dis-
tributed system. We also evaluate the impact on PIC’s accu-
racy of replacing the oracle by one of these algorithms.
PIC can replace the oracle by one of several algorithms
that have been proposed to find the closest node to a par-
ticular node in a network, for example, [15, 2]. These al-
gorithms all share a similar overlay structure. Each node in
the overlay maintains a set of pointers to other nodes in the
overlay that we call its neighbors. These algorithms pre-
scribe a particular mix of near and far away neighbors to
ensure that a node n can find the closest node in the over-
lay in O(logN ) steps provided the topology satisfies cer-
tain conditions. To find the closest node, n starts by setting
its estimate of the closest node in the overlay, c, to a random
overlay node. Then it probes the distance to all of c’s neigh-
bors and picks the closest neighbor. If this closest neigh-
bor is closest than the current value of c, c is updated to
point to this neighbor and the process is repeated. Other-
wise, the algorithm stops and c is an approximation to the
closest node to n in the overlay. These algorithms can also
find the k closest nodes to n by keeping track of the k clos-
est nodes visited.
PIC could use one of these algorithms but they require
a significant number of probes to estimate the distance be-
tween nodes.
We can reduce the overhead by using PIC to estimate
distances rather than using probes to measure the distances.
The problem is that we need to find the closest nodes to
a node that does not have coordinates yet. Our improved
algorithm solves this problem as follows. A joining node
starts by using PIC with the random strategy to generate a
rough estimate of its coordinates from a set of random over-
lay nodes. Then these rough coordinates are used to esti-
mate distances in the algorithm to find the closest k nodes.
After finding the closest k nodes, the joining node uses the
PIC hybrid strategy to refine its coordinates.
We implemented and evaluated two variants of this strat-
egy. The pseudo code for the first one is in Figure 5. In
this variant, the joining node, n, computes rough coordi-
nates from a set of random nodes and performs a series
of greedy walks in the overlay towards the closest node.
It starts each walk from a random node and uses the rough
coordinates to estimate distances to guide the walk. This
search stops when the number of walks reaches a maxi-
mum value or when the predicted distance to the closest
node found is below a threshold t. Our current implementa-
tion sets the threshold to the average distance to the closest
neighbor computed over the set of all nodes visited during
the search process. When the search stops, the node probes
the k predicted closest nodes and uses the measured dis-
tances and their coordinates to recompute its coordinates.
We use k = 4 because our previous results indicate that
PIC hybrid performs well with this value.
The second variant is in Figure 6. It incurs a higher over-
head but it achieves better accuracy because nodes refine
their coordinates at each step of the search process. More
precisely, the joining node n probes the m predicted clos-
probed = Probe(random nodes)
allVisited = probed
CalculateCoordinates(probed)
numWalks = 0
do
numWalks++
nearNode = PickRandom(allVisited)
visited = nearNode
do
currentClosest = nearNode
visited += GetNeighbors(nearNode)
nearNode = GetPredictedClosest(visited+{nearNode},1)
while(currentClosest != nearNode)
allVisited += visited
while (PredictedDistanceTo(nearNode) > t
and numWalks < maxWalks)
probed += Probe(GetPredictedClosest(allVisited, k))
CalculateCoordinates(probed)
Figure 5. Optimized close node discovery al-
gorithm.
probed = Probe(random nodes)
allVisited = probed
CalculateCoordinates(probed)
numWalks = 0
do
numWalks++
nearNode = PickRandom(allVisited)
visited = nearNode
probed += Probe(nearNode)
do
currentClosest = nearNode
visited += GetNeighbors(nearNode)
nodes = GetPredictedClosest(visited, m)
foreach node in nodes
probed += Probe(node)
nearNode = closerToMe(node,nearNode)
CalculateCoordinates(probed)
while (currentClosest != nearNode)
allVisited += visited
probed += Probe(GetPredictedClosest(allVisited,k))
CalculateCoordinates(probed)
while (DistanceTo(nearNode) > t
and numWalks < maxWalks)
Figure 6. Optimized close node discovery al-
gorithm with progressive coordinate refine-
ment.
est neighbors at each step and it uses the measured distance
to select the closest neighbor. n also recomputes its coordi-
nates using all nodes probed so far. Our current implemen-
tation uses m = 2. Additionally, n probes the k predicted
closest nodes at the end of each walk (if not probed yet) and
recomputes its coordinates.
It is also interesting to note that we can improve the ac-
curacy of the algorithms to find close nodes, and reduce
their associated cost, by multicasting discovery messages
in a range of one or two network hops. This simple mech-
anism will work very effectively when the density of PIC
nodes in the network is high. In the particular case where
all the nodes in the network are part of the PIC system, a
joining node can find the closest nodes simply by broad-
casting a discovery message over all of its physical inter-
faces.
3.1. Experimental evaluation
We ran some experiments to evaluate PIC’s accuracy
when using both algorithms described above. Our experi-
ments ran on MSPastry [1] (configured with l = 16 and
b = 4). Each MSPastry node keeps a set pointers to neigh-
bors. These pointers form the overlay that is used to locate
closest nodes. The results should be similar for the over-
lay described in [15]. The experiments ran on the GAT-
ech topology in the experimental setting described in Sec-
tion 2.1.1. We set the maximum number of walks to 5,
m = 2, and we use k = 4 to match the experiments pre-
sented in previous sections.
Figure 7 shows the cumulative distribution of relative er-
rors for PIC using the oracle and the two algorithms to find
close nodes. The line labeled PIC Simple corresponds to
the version of the algorithm in Figure 5, and the line labeled
PIC Refine Coordinates corresponds to the one in Figure 6.
The line labeled PIC Optimal corresponds to the version of
PIC that uses the oracle to find the closest node as in the pre-
vious section.
The results show that PIC’s accuracy is essentially the
same using the oracle or the algorithm with coordinate re-
finement but the accuracy drops when using the algorithm
without refinement. This drop in accuracy is explained by
the results in Figure 8. This figure shows the cumulative dis-
tribution of absolute errors when finding close nodes with
both algorithms. The algorithm with coordinate refinement
is significantly more effective at locating the closest node
or a close approximation, which explains the improved ac-
curacy. This increased accuracy comes at the cost of an in-
creased number of probes: the algorithm with refinement
probes 55.6 nodes on average and the algorithm without re-
finement only probes 21 nodes.
