Updates in Highly Unreliable, Replicated Peer-to-Peer Systems
Anwitaman Datta, Manfred Hauswirth, Karl Aberer
Swiss National Institute of Technology (EPFL), Lausanne, Switzerland
anwitaman.datta, manfred.hauswirth, karl.aberer
@epfl.ch
Abstract
This paper studies the problem of updates in decen-
tralised and self-organising P2P systems in which peers
have low online probabilities and only local knowledge.
The update strategy we propose for this environment is
based on a hybrid push/pull rumor spreading algorithm
and provides a fully decentralised, efficient and robust com-
munication scheme which offers probabilistic guarantees
rather than ensuring strict consistency. We describe a
generic analytical model to investigate the utility of our
hybrid update propagation scheme from the perspective of
communication overhead.
1. Introduction
In most peer-to-peer (P2P) systems data is assumed
to be rather static and updates occur very infrequently.
For application domains beyond mere file sharing, for ex-
ample trust management [2] or peer commerce, such as-
sumptions do not hold and updates in fact may occur fre-
quently. Other typical applications where new data items
are added, deleted, or updated frequently by multiple users
are bulletin-board systems, shared calendars or address
books, e-commerce catalogues, and project management in-
formation.
To improve fault-tolerance and response time data is
heavily replicated in most P2P systems and the system must
take into account that peers are autonomous and may be of-
fline frequently and that no global knowledge on the system
exists. This is especially relevantfor upcoming mobile envi-
ronments. Thus we are operating in a decentralised setting
without global control. This is the setting we assume for
updates in our P-Grid P2P system [1, 3] described in this
paper. To meet the challenges imposed by a high replica-
tion factor, lack of global knowledge, and peers being on-
line only with a very low probability, we exploit epidemic
algorithms under the assumption that probabilistic guaran-
tees instead of strict consistency is sufficient and such an
approach can indeed be used in a decentralised and self-
organising environment.
Our proposed update algorithm is based on rumor
spreading. We modify existing message flooding algo-
rithms to achieve lower communication overheads but pro-
The work presented in this paper was supported (in part) by the Na-
tional Competence Center in Research on Mobile Information and Com-
munication Systems (NCCR-MICS), a center supported by the Swiss Na-
tional Science Foundation under grant number 5005-67322.
vide similar probabilistic guarantees and low latency. Since
we assume that peers are mostly offline, we propose a hy-
brid push/pull algorithm so that offline peers can inquire
for updates that they had missed when they become on-
line again. In the push phase the algorithm uses a new
mechanism, apart from traditional feedback and probabilis-
tic methods to propagate a rumor, to avoid many duplicate
messages by propagating a partial list of peers to which a
particular message has already been sent. It also employs
this list in conjunction with the number of duplicate mes-
sages received at a particular node as a local metric to es-
timate the extent to which a message has spread globally,
and thereby tunes the probabilistic parameters of the generic
algorithm locally which is a novel contribution. As will
be discussed later, the current taxonomy of epidemic algo-
rithms does not exploit the advantages of speculation (feed-
forward) for significant reduction of communication over-
head.
Further, the algorithm deals with logical connectivity
(knowledge), and is disentangled from the underlying net-
work/physical connectivity. Consecutively, the propagation
of messages in the physical network and thus the imple-
mentation of applications is an orthogonal issue and may
employ any point-to-point/multicast/ad-hoc communication
mechanism (for example, [4, 13]). Though this work was
initially motivated by P-Grid, the algorithm is generic and
can be applied in other systems too. Our modifications
to existing message flooding algorithms and other results
may as well be applied to other search/update algorithms or
broadcast/multicast schemes which employ flooding.
Another significant contribution of this paper is an ana-
lytical model of the gossiping algorithm unlike most of the
literature which relies on simulation results. Since our al-
gorithm is generic as argued above the analytical model is
valid for many of the other variants of flooding algorithms
and so are the results of our analysis.
2. Motivation and problem statement
Various global storage systems have been proposed, for
example, Freenet [7], OceanStore [24], Pastry [26], and Far-
site [5]. Their main goal is to provide distributed storage
that scales to very large numbers of users and data sets.
Additionally, they may exhibit certain specialisations that
stem from their intended application domains. For exam-
ple, Freenet wants to support free speech and anonymity
on the Internet, whereas OceanStore focuses on distributed
archival storage, which requires special system support.
From the viewpoint of data management these systems
should address two critical areas:
1. Efficient, scalable data access which is provided more
or less by all approaches, and
2. Updates to the data stored, especially with respect to
replication and low online probabilities.
Many of the access schemes are based on some mech-
anism that associates peers logically with a partition of the
search space by means of a distributed, scalable index struc-
ture (P-Grid, OceanStore) and use replication to improve
responsiveness and fault-tolerance. Some of the systems
support updates. For example, OceanStore, uses classical
schemes for updating replicas and assumes high availability
of servers, whereas in the systems we envision the peers are
fairly unreliable. Our assumptions are:
– Peers have low online probabilities and quorums can-
not be assumed.
– Eventual consistency is sufficient.
– Since we do not target database systems update con-
flicts are rare and their resolution is not necessary in
general.
– Probabilistic success guarantees for search are suffi-
cient.
– Consecutive updates are distributed sparsely.
– The required communication overhead is the critical
measure for the performance of the approach.
– The typical number of replicas is substantially higher
than assumed normally for distributed databases but
substantially lower than the total network size.
– The connectivity among replicas is high and the con-
nectivity graph is random.
The two last points require further discussion: We may
expect a few hundred to thousand replicas to be maintained.
An intuitive explanation for such numbers is that if we need
a 99.9% success guarantee for a search and only 10% of
the replicas are online on average, then a serial search will
need about 65 attempts (since
). This does
not account for load balancing among the available repli-
cas which may account for another factor of 5 to 10 times
of replication, particularly for “hot” items. Statistics from
some of the music file sharing systems show that on average
200 to 250 replicas of same files are available, not counting
the replicas that are not shared [30]. This requires devis-
ing a scheme which scales at least beyond the hundreds to
thousands.
We assume that the replicas within a logical partition of
the data space are connected among each other and each
replica knows a minimal fraction of the complete set of
replicas. If not enough replicas are known they can be effi-
ciently obtained by randomized search. Additionally repli-
cas get known through the update mechanism discussed in
this paper. Even when peers potentially know a large frac-
tion of the complete replica population the use of rumor
spreading bears a number of advantages as compared to im-
mediately contacting the complete neighborhoods for up-
dates: better load balancing, reduced delay due to paral-
lel propagation, improved robustness against changes in the
peer network, and only partial knowledge of the neighbor-
hood is required.
3. System model
In the analysis of the update algorithm we focus on
the amount of communication required to achieve quasi-
consistency and provide probabilistic guarantees for suc-
cessful and appropriate results for queries. As observed in
[22] we assume a very low rate of conflicts. Indeed, many
applications, for example, music file sharing or news dis-
semination, have such a profile where, if data is altered, it
may be treated as distinct and coexists as different versions.
Similarly, deletions may use conventional tombstones and
death certificates. These issues are relatively orthogonal
to the communication mechanism used to convey the up-
dates among the replicas. Further, in a decentralised sys-
tem, such as P-Grid the “data” may indeed be knowledge
regarding the system’s topology, for example the routing ta-
bles used in P-Grid [1]. Most of these systems operate with
a relatively high degree of imperfect knowledge, which is
why probabilistic guarantee of information dissemination
in such application scenarios is sufficient.
Our system assumes an infrastructure-less peer-to-peer
system, i.e., all peers are equal and no specialised infras-
tructure, e.g., hierarchy, exists. No peer has a global view
of the system but base their behaviour on local knowledge,
i.e., its routing tables, replica list, etc. The peers can go of-
fline at any time according to a random process that models
the behaviour when peers are online. Physical connectiv-
ity and topology are ignored. We can assume that if two
peers are online a communication channel may be estab-
lished between them. This is not a serious relaxation of
assumption, since if two peers may not communicate with
each other, they will simply perceive each other to be of-
fline. It is primarily the erratic behaviour of online avail-
ability and the complete lack of global knowledge, as well
as the absence of any centralisation, and thus the need of
self-organisation, which prompts us to call this environment
unreliable. The limited resources, particularly bandwidth
(and power in wireless/mobile environments), and the vary-
ing degree for tolerance of latency makes the environment
even more challenging.
Our update propagation scheme has a push phase and a
pull phase which are consecutive but may overlap in time. A
new update is pushed by the initiator to a subset of responsi-
ble peers it knows, which in turn propagate it to responsible
peers they know similar to a constrained flooding scheme.
By “responsible” we denote peers that are affected by the
update because they hold the original version of the data
item. In our analysis of the push phase in the next section
we assume a synchronous model which is a standard model
for analysing epidemic algorithms [18].
Peers that have been disconnected (offline, disruption
of communication) and get connected again, peers that do
not receive updates for a long time (locally determined), or
peers that receive a pull request, but are not sure to have the
latest update, enter the pull phase to synchronise and recon-
cile. The pull scheme is similar to anti-entropy [9], in the
sense that the pulling party tries to synchronise itself with
the pulled party. Since the pulled party itself may be out of
sync, it is preferable to contact multiple peers and choose
the most up to date peer(s) among them.
Push phase of the update algorithm: When a peer
re-
ceives a update request
!
, where
is the up-
dated data item,
its version
1
,
is a counter which counts
the number of push rounds that have already been executed
for the update, from a peer
"
, it also receivesa partial (flood-
ing) list
, to which the same update has been sent (not
necessarily received by all peers in
). Then
chooses a
random set
$#
of its replicating peers and forwards the re-
quest
%
'&
(# )*+!
with a probability
,.-/0 !
to the
set
#1
.
,.-/0 !
can be any function, and is a self tuning
parameter, determined locally by
. Another benefit of
2
is that
possibly discovers replicas unknown to her.
3
4
5
6
/* At replica ‘p’ (push phase) */
IF received(Push(U, V, R_f, t)) THEN
/* process the update if not precessed it yet */
IF ProcessedUpdate(U, V, R_f, t) == FALSE THEN
Select a random subset Rp of replicas
with |Rp| = R * f_r;
With probability PF(t):
Push(U, V, Rf union Rp, t+1) to Rp \ Rf ;
/* PF(t): deterministic or self
tuning function */
ProcessedUpdate(U, V, R_f, t) = TRUE
Since any replica pushes the update at most once, the ter-
mination decision is trivial, and the number of push rounds
gives the latency of propagating the update to all online
replicas with high probability (arbitrarily close to 1).
Pull phase of the update algorithm: When a peer gets
connected again because it was offline or suffered from
a communication disruption, received no update for some
time, or receives a pull request but is not sure whether it is
in sync, then it enters the pull phase and inquires for missed
updates.
3
4
5
6
/* At replica ‘p’ (pull phase) */
IF online_again OR no_updates_since(t) or
(received_pull and not_confident) THEN
/* not_confident is true:
no update received within time T */
Contact online replicas;
Inquire for missed updates
based on version vectors;
4. Analysis
4.1. Setup and notation for the analysis
The goal of our update algorithm is not to achieve com-
plete consistency but rather to know what is the probability
of a correct answer given certain model parameters. We
assume that every peer knows a subset of all replicas that
replicate the same data. We consider the replica network to
be a small P2P network itself but with no internal structur-
ing. It handles updates/requests for a partition of the data
space.
In the analysis we start from a completely consistent
state, analyse a single update request, and evaluate the num-
ber of messages and time (rounds of message exchange) re-
quired to reach a consistent state again. Since most of the
1
This actually is a vector of version identifiers of the form
798:;=<?>A@BDC=E+FGH8:;=<?>A@BDC=EJIJG?K?KK?GH8:;=<?>A@BDC=E=LM
. Version identifiers are
universally unique identifiers computed locally by applying a a crypto-
graphicly secure hash function to the concatenated values of the current
date and time, the current IP address and a large random number.
replicas are offline most of the time, our notion of consis-
tent state is more related to the online population
2NO P
at a
given time
Q
rather than the whole set of replicas
R
. Though
our analysis is generic, we evaluate the algorithm for real-
istic scenarios: availability of the peers to be a random pro-
cess with expected value of being online between 10% to
30%. The replication factor is assumed to be between 100
to 1000 and though scalability is not a major issue for such
small numbers larger replication factors too have been in-
vestigated. Table 1 shows the notation used in our analysis.
When an update
is initiated for a set
R
of replicas
with cardinality
, in general the online population in push
round
will be
O P
S !%TU
O P
SJVWX!
Y
)/Z [V\
OP
0]V^+!_
`
where
Y
Ta(Vcb=d
and
`
Tebf
, where
O P
g!hTi2N
O P
if the
update starts at time
Q
.
b=d
is the probability of an online peer
going offline in one push round and
b
f
is the probability of
an offline peer coming online in a push round. These values
are typically small and may vary in different push rounds.
For the sake of simplification, we will initially ignore the
effect of replicas coming online, and will further assume a
constant
Y
, hence we have
OP
0 !jTk
O P
SVlX!
$Y
. Ne-
glecting the effect of positive
b
f
is justified because peers
coming online need to execute pull any way, and thus do not
contribute to the push phase. Thus, even if some peers come
online during a push phase, and receives update through
push, it will not make any major difference to the whole
system’s behavior or the analysis of the same. The assump-
tion of a very small
bmd
is justified because a single push
round will take a very small time (network delay for a sin-
gle message), and unless there is any kind of catastrophic
failure, a very small number of peers will suddenly decide
to go offline. Further, we choose a discrete time model for
the rumor spreading algorithm, just like most other rumor
algorithms. This in itself does not mean that we need syn-
chronous rounds. It is indeed possible that because of vari-
ation in network latency, messages of different push rounds
live in the network at the same instant of time. Thus, instead
of treating
strictly as time, it needs to be interpreted as the
round number, and the replicas which get infected by that
round are effectively replicas that eventually gets updates
from this round. Thus
does not in itself necessarily define
an ordered chronology of receiving updates among all peers
in the system.
Typically the parameters, such as
"on
,
Y
,
,
O P
g!
, may
vary over time. But for the purpose of analysis we may as-
sume that they remain constant throughout a single update
push phase. In Section 6 we will give some indication of
how the parameters can adapt over time to the varying net-
work topologies.
The choice of two parameters
,'p
(probability of for-
warding an update) and
"qn
(fraction of total replicas to
which peers initially decide to forward an update) rather
than defining only one parameter which couples both of
them together is because we wanted to study the effects of
both these factors in limited flooding algorithms. For exam-
ple, a protocol like Gnutella [8] uses flooding with a fixed
fanout, but uses no notion of
,
p
. Actually its use of TTL
effectively means that
,
p
is 1 for TTL rounds, and 0 after
that. Some other systems, for example one using gossip for
Cardinality of the set of replicas
R
Number of the push round for a particular update
The update message or its size (notation depends on the context)
rts
0 !
Size of messages in round
s
0 !
Normalised size of the partial list of replicas which have the update in round
. This is equal to the
number of entries in the list divided by
.
OP
0 !
Number of replicas online in round
Y
Probability that a peer stays online in the next push round
"
n
Fraction of replicas to which peers initially decide to forward the update message
uwvXx$yov
{zS|~}=S !
Number of new replicas receiving update in round
JS !
Number of messages in round
, including messages to offline replicas
"mw
n
S !
Increment in fraction of online replicas which are aware of the update after round
"
mw
n+0 !
Total fraction of online replicas which are aware of the update at the beginning of round
.
,
p
S !
Probability that a peer pushes an update in round
if it received it in round
V
.
Size of data required to describe one replica (e.g., 10 bytes).
Table 1. Notation used in the analysis
ad-hoc routing [13], on the other hand uses probability of
forwarding rumors as a design parameter. In order for our
analysis to be general enough, such that all these variations
of limited flooding can be reduced to special cases of our
model, we included the notion of both fanout and probabil-
ity of forwarding.
4.2. Analysis of the push phase
Round 0
The replica initiating the update propagation sends
to
"on
fraction of replicas. Thus we obtain a total number of mes-
sages,
Jg!T
"on
(including messages to offline
replicas). The number of new replicas which receive the
update is
uwvXx$yov
z0|~}=S!Tl
OP
S! "
n
. The number of on-
line replicas without update is
OP
S!=?V."onJ!
. The message
length in this round is
rts
S!TU)
"on
Round 1
Assuming message flooding, where every replica which re-
ceived an update message decides with probability
,
p
?+!
to forward it to
"on
replicas, we have:
J?X!Tl
OP
S!
Y
,
p
?+!?."
f
n
(Vc"
n
!
The expression may be explained as follows.
OP
S!"on
of the online population received the update in the previous
round, a fraction
Y
of these replicas continue to stay online
in the present round, a
,
p
X!
fraction of these replicas de-
cide to forward the message. Each of the
OP
S!"on
Y
,'p$?+!
peers decide to push the update, forwarding it to
[g"onV/"
f
n
!
replicas, since it knows that the update has already been sent
to
"
f
n
of the
"
n
fraction of randomly chosen replicas. Actu-
ally, in case a replica receives update information from more
than one replica, it can use the list of ‘updated replicas’ in
each of those messages, and hence the number of messages
can be further trimmed, at an additional computational cost.
uwvXx$yov
{z0|~}=?+!T
O P
g!
Y
$V"onJ!
Z(V $Vc"qnJ!?wHD
o¡
d
H_
The expression may be explained as follows: Of the
OP
S!
Y
V¢"onJ!
uninformed online peers, a fraction
?.VU"
n
!
wg
?o¡
d
peers continue to stay uninformed
when each of the
OP
S! "on
Y
,£p$ X!
informed peers forward
(push) to
"on
fraction of random peers. The others receivethe
update after this round. For the message length we have:
rts
?X!¤T ¥)¦
g"on%)"on?(V"onJ! !
T ¥)¦
?(V¥?$V"onJ!
f
!
Round
¨§©
The results may be generalised as follows:
uwvXx$yov
z0|~}=0 !¤T
OP
0VX!= (Vc"
mw
n]SV+! !
Yª
?(V¥?$V"
n
!
wg«¬
d
w®?¯D®
~°
«¬
d
o¡
«0
!
Thus we obtain the fraction
"mw
n
0 !
and
"qmw
n
0 !
as:
"mw
n
0 !±T ?(V"qmw
n
0 !!
?(V¥?(V"
n
!
H
«¬
d
®?¯D®
~°
«¬
d
o
¡
«0
!
Then,
"
mw
n]S !±T "
mw
nX0V+!w)"
mw
n]S£VX!
T (V (Vc"
=w
nXS'VX!!
?(V"onJ!?wHD«¬
f
® ¯®
~°
«¬
f
o¡
«¬
d
Note that this is a recursive relationship and
"
=w
n
rapidly
grows to 1. The expression for
"
mw
n
may exceed the value
of 1, but that will have no physical relevance, and thus the
function needs to be determined using a ceiling function and
"mw
n
too needs to be reevaluated accordingly in the fi-
nal push round. Also note that
,£pj0 !
can be any arbitrary
function of
, which individual nodes can define in an ad-
hoc manner, and we will see that this will be a self-tuning
parameter for our push phase algorithm.
It is subtle to determine the number of messages and
length of these messages. If the partial list of replicas, to
which the update has already been transmitted along with
the update information U, is ignored, we have
=0 !T
O P
S'VX!"mw
n
0V+!
Y
,
p
0 ! 2"
n
since each of
OP
0V²X! "mw
n
SV*+!
Y
,
p
0 !
replicas
(these replicas received the update in the previous round,
and continued to stay online, and decided to forward the
same) forward the update to
."on
replicas. If the partial list
of replicas is accounted for, then the number of messages
decrease to
J0 !¤T
OP
0VX! "mw
n
S'VX!
Y
,
p
0 !
2"qnq?(V"on]!?«
and the length of each message in round t is given as
rts
S !Ti¥)
$V (Vc"
n
!
«0³
d
!
We now prove the two equations above by induction. Let
the normalised length of the partial list of replicas in a mes-
sage be denoted by
s
S !
. The normalised length of the par-
tial list is the fraction of the total replicas that the partial list
contains. Then
rts
0 !Te¥)¦
s
0 !
.
Induction hypothesis:
s
S !T´V¥?$V"
n
!
«0³
d
Now
s
0)kX!µT¶"on.)
s
S !$Ve"on
s
S !
, since the
."on
replicas chosen randomly are independent of the replicas in
the partial list. Now, if our hypothesis is true then,
s
0w)lX!¤T "qn%)U(V¥?(V"on]!?«0³
d
V"onq (V¥ (Vc"qnJ!~«0³
d
!
T (V¥?(V¦"on]!
«0³
d
)"onq $V"onJ!
«0³
d
T (V¥?(V¦"on]!~«0³
f
Thus the hypothesis is consistent. Since the hypothesis is
true for
¨Te]
, using induction, we conclude that
s
S !¨T
¨V?(Vc"onJ!
«0³
d
where
rts
S !T¥)
s
0 !
. Thus,
J0 !¤T
OP
0V+! "=w
n
0VX!
Y
,
p
S !?
"qn?(V
s
S£VX!!
T
OP
0V+! "
=w
n]0VX!
Y
,£p$S !?."onq $V"onJ!~«
As may be observed,
s
0 !
increases with round number
and a legitimate question to ask is its effect on the re-
source (Memory/CPU/Bandwidth/Power) available at each
of the replicas. A way to deal with increasing
s
S !
may be
to chose a normalised threshold length
s%·
m¸0 !
such that
s
0 !.T
|
u
sh·
=¸0 !=
s
S !¹X!
where
s
0 !¹/T
s
SV+!)
"on¨V
s
S£VX!"on
. This can be achieved by discarding either
random entries or the head or tail of the partial list. In this
case,
JSD)c+!Tl
O P
S ! "
=w
nXS !
Y
,'p$0)cX! ."on 'V
s%·
m¸S ! !
.
"mw
n
and
"qmw
n
stay unchanged, since the extra
messages generated by reducing the
s
S !
are all duplicate
messages. Thus the nodes which push the update in the
next round pay the penalty of forwarding extra messages.
Note that the case where
s%·
m¸S !
is zero for all replicas
corresponds to the case where no list is propagated, and will
enhance the number of duplicate messages, without any im-
provement in coverage of unreached replicas.
4.3. Analysis of the pull phase
If a replica
comes online at a random time (after the
push phase is over), then it will (very likely) find the update
information from any of its online replicas. The underly-
ing assumption for such an optimism is that any replica that
came online in the meantime must have pulled the update
information by the time the concerned peer
came online.
This justifies the eagerness of the Update Pull algorithm.
What is more interesting is what happens if
comes on-
line while a push of an update is underway. If
"
mw
n
frac-
tion of the replicas
OP
are already aware of the update, the
probability of a replica
getting the update in
attempts is
VUZ(V¥S
O P
"
mw
nºq.!_
which implies that a constant number of pull attempts
should give the update information with high probability.
Since updates are propagating by push as well, the above
term gives a worst case estimate. Indeed if
"mw
n
(refer
to push phase analysis) fraction of online replicas received
updates in the previous push round (
$V»
), then (if they
continue pushing) the probability of getting a push is
(V $Vc"qnq?(V
s
S ! ! !
H
«¬
d
®?¯D®
~°
«¬
d
o
¡
«0
4.4. Query (request)
Servicing requests under (possibly relatively frequent)
updates is similar to the Pull phase of updates. For simple
servicing of requests, we may indeed use the same analy-
sis as in the Pull section. Since requests are more sensitive
(updates can be lazy, and strong consistency is not our goal,
however we intend to return correct and most recent result
for any query) we may define some majority logic, or use a
version scheme for identifying latest updates, or a hybrid of
the two.
5. Analytical results
Based on the analytical model developed in the previous
section we investigated for various environmental parame-
ters the performance of the push phase of the propagation
of a single update. For the evaluation of the recursive ana-
lytical functions a C-program has been developed.
Our performance criterion for this analysis is primarily
the number of messages that are generated as part of a single
update, compared to the extent to which the update propa-
gates among the online population. As a simplifying (and
for fixed networks, realistic) assumption we ignore message
size, as single messages can accommodate the messages of
maximal size that can occur in our setting.
In the following result plots (e.g., Fig. 1) we will show
on the y-axis the number of messages generated per mem-
ber of the initial online population. As assumed in the pre-
vious section peers coming online are not participating in
