SAMOA: Framework for Synchronisation Augmented Microprotocol Approach
∗
Paweł Wojciechowski, Olivier R
¨
utti, and Andr
´
e Schiper
EPFL, School of Computer and Communication Sciences
1015 Lausanne, Switzerland
{First.Last}@epfl.ch
Abstract
We address programming abstractions for building pro-
tocols from smaller, reusable microprotocols. The existing
protocol frameworks, such as Appia and Cactus, either re-
strict the amount of concurrency between microprotocols,
or depend on the programmer, who should implement all
the necessary synchronisation using standard language fa-
cilities. We develop J-SAMOA: a framework for a Synchro-
nisation Augmented Microprotocol Approach in Java. It has
been designed to allow concurrent protocols to be expressed
without explicit low-level synchronisation, thus making pro-
gramming easier and less error-prone. In this paper, we
describe versioning concurrency control algorithms. They
are used by the runtime system of our framework to guaran-
tee that the concurrent execution of a protocol is equivalent
to a serial execution of its microprotocols. This guarantee,
called the isolation property, ensures consistency of session
or message-specific data maintained by microprotocols.
1 Introduction
Modularization is a well-known technique for simpli-
fying complex communication systems. Protocol frame-
works, such the x-kernel [12], Cactus [24], Appia [17], and
Ensemble [9] have been built. They allow complex pro-
tocols to be implemented as compositions of separate mi-
croprotocols that communicate using the framework’s in-
terface [23]. This approach helps to clarify the dependen-
cies among properties required by a given communication
system, allows for code reuse, and makes it possible to con-
struct systems that are customized to the specific needs of
the application or underlying network environment. The
protocol frameworks also support primitives that can sim-
plify the construction of network protocols, such as support
∗
Research supported by the Swiss National Science Foundation under
grant number 21-67715.02 and Hasler Stiftung under grant number DICS-
1825.
for processing messages, marshalling messages to the net-
work format, and timeouts.
The design of modular protocols is however problematic:
some microprotocols may have to synchronise their actions
in order to maintain consistency of session or message-
specific data (the problem will be illustrated on the example
of group communication protocols in Section 3). The goal
of our work was to design programming support for leaving
the exact implementation of this synchronisation to a run-
time system rather than requiring programmers to identify
what synchronisation is needed, and where to acquire and
release locks. For the class of synchronisation problems we
consider here, a sufficient correctness condition is the isola-
tion property. Execution of concurrent microprotocols satis-
fies the isolation property if the protocol’s state is equivalent
to a state that can be produced by some serial execution of
the microprotocols. Our approach has close analogies with
the concept of transactions. The isolation property is like
isolation in database transactions, however, it is not accom-
panied by additional properties defined in the transactional
model, i.e. atomicity, consistency, and durability. In this pa-
per, we f ocus on a lightweight implementation for enforcing
the isolation property in protocol frameworks.
The existing software frameworks do not support the iso-
lation property in a completely satisfactory way. For in-
stance, Appia does not allow for concurrent execution of
microprotocols: all actions which result from a single ex-
ternal event (such as message arrival) are handled by mi-
croprotocols sequentially using FIFO communication chan-
nels; this excludes a number of reasonable concurrent exe-
cutions of microprotocols, e.g. for dealing with messages
from the application and network at the same time, for
processing time consuming I/O operations in background,
or in order to support multiprocessor architectures. Cac-
tus does not restrict the amount of concurrency but, on the
other hand, it depends on the programmer, who must im-
plement the required synchronisation policy using standard
language facilities (such as locks, semaphores, and mon-
itors). The synchronisation code is however rather subtle
and error-prone, especially for highly-concurrent protocols.
0-7695-2132-0/04/$17.00 (C) 2004 IEEE
Proceedings of the 18th International Parallel and Distributed Processing Symposium (IPDPS’04)
In this paper, we try to find a better solution.
To study the problem of the isolation property in the pro-
tocol design, we develop SAMOA – a programming lan-
guage for a Synchronisation Augmented Microprotocol Ap-
proach. The main feature of the approach is separation of
the low-level synchronisation from the protocol’s logic. The
SAMOA programmer has only to declare which events are
external; the runtime system will use concurrency control
algorithms to enforce that the effects of one computation,
defined informally as the execution of all microprotocols
involved in processing of a single external event, are not
visible to other computations executing concurrently; from
the perspective of a computation, it appears that computa-
tions execute sequentially rather than in parallel. Compu-
tations correspond to (possibly multi-threaded) transactions
with no support for atomicity and durability. However, un-
like transactions whose termination is defined statically us-
ing “commit” and “abort”, computations are never aborted,
and have scope which depends on a given composition of
microprotocols.
In this paper, we describe J-SAMOA – a protocol frame-
work that ensures the isolation property. J-SAMOA is
based on the SAMOA language and implemented as a Java
[1] library; the implementation is available [6]. We have
designed several deadlock-free concurrency control algo-
rithms for our framework’s runtime system. They can be
classified into two groups: 1) versioning algorithms with
allocation of access to event handlers, and 2) timestamp-
ordering algorithms with rollback/recovery. In this paper,
we focus on the first group, and describe three algorithms
that we use in J-SAMOA: the basic version-counting al-
gorithm (VCA
basic
) and two extensions of this algorithm,
VCA
bound
and VCA
route
. The latter two algorithms can
support more parallelism, however, they achieve that by de-
manding some additional properties of the protocols to be
specified by the programmer. The VCA
bound
algorithm re-
quires the least upper bound on the number of times a given
microprotocol can be executed by a computation, VCA
route
requires the pattern of handler calls to be specified.
The advantages of our framework with respect to exist-
ing protocol frameworks are twofold: 1) it makes program-
ming of concurrent protocols easier since the programmer
does not need to implement the low-level synchronisation
(we illustrate this point in the paper), and 2) it can also
greatly simplify reasoning about protocols and doing the
correctness proofs. The latter claim follows from the fact
that the isolation property allows us to reason about concur-
rent computations as they would be executed sequentially.
The paper is organized as follows. Section 2 describes
our model. Section 3 gives a brief illustrative example of
J-SAMOA code. The code is used to show some non-trivial
programming errors in the design of concurrent protocols.
The construct of the SAMOA language that eliminates this
type of errors is informally described in Section 4. Sec-
tion 5—the heart of our paper—describes the concurrency
control algorithms that we designed for J-SAMOA. Section
6 discusses related work, and Section 7 sketches our plans
for the future development of J-SAMOA.
2 Model
Protocol frameworks, such as Cactus (which builds on
the x-kernel), Appia, and our framework, can be described
using a simple event model. In this model, protocols consist
of code blocks that communicate (synchronously or asyn-
chronously) using internal events, and may also react on ex-
ternal events and output values. Below we define the model,
and use a small example of a protocol in this model to illus-
trate the notion of the isolation property.
Protocols consist of code blocks called event handlers.
Several related handlers can be grouped into a single mi-
croprotocol and share a local state of the microprotocol’s
object. Execution of a handler can directly modify only the
local state of its own microprotocol. The protocol’s state
is the union of (disjoint) local states of all microprotocols,
and encompasses all of the in-memory and on-disk data
items that affect the protocol’s operation. For simplicity, we
confuse handlers and microprotocols (simply assuming that
each microprotocol has only one handler). In J-SAMOA
programs, a microprotocol can consist of many handlers.
Executions of handlers are triggered by events. An event
is a request (at run time) to call a handler. Each event must
specify an event type — only handlers that have been bound
to this event type will be executed as the result of the event.
Any events generated by the execution of handler(s) that
have been triggered by an event a,arecausally dependent
on a; the causality relation is reflexive and transitive. Some
events are not causally related; they are concurrent.An
event a is pending if there is at least one handler requested
by a that has not commenced yet in response to event a.
We have two kinds of events: internal and external. An
internal event generated during a handler’s execution trig-
gers the execution of another (or the same) handler. The
execution of a handler can generate zero, one or more in-
ternal events. External events of a protocol are normally:
1) requests by the network layer (or application) to inject a
message received from the network (or application) to the
protocol, and 2) a timeout action. In practice, some internal
events may be also regarded as external.
Execution of a protocol is modelled as a run, defined as
a list of pairs (a, H), where a is an event and H is a handler
requested by a that has begun execution; if H has not com-
menced yet, we write H
∂
. The list is ordered according to
the order of time when handlers commenced, or, in case of
pairs (a, H
∂
) where a is pending, when events were issued.
Aruniscomplete if it does not have pending events.
0-7695-2132-0/04/$17.00 (C) 2004 IEEE
Proceedings of the 18th International Parallel and Distributed Processing Symposium (IPDPS’04)
b0
S
R
PQ
a0
a1 b1
a2 b2
Figure 1. Events and handlers
To illustrate the notions introduced so far, let us take an
example protocol with four handlers P , Q, R, and S, as il-
lustrated in Fig. 1. It can, for example, receive UDP packets
either from ad-hoc network (to be processed by P )orfixed
network (to be processed by Q), and deliver them to han-
dler S; we abstract away from details. Handlers R and S
are assumed to be atomic (i.e. only one instance at a time).
We assume that two external events a
0
and b
0
have oc-
curred. In all possible executions of this protocol triggered
by these events, event a
0
(correspondingly b
0
) triggers the
execution of handler P (correspondingly Q); and handler
R is executed twice, once as the result of internal event a
1
(which causally depends on event a
0
), another time as the
result of internal event b
1
. The events a
0
and b
0
are concur-
rent, also events a
1
and b
1
are concurrent. Example com-
plete runs of the protocol are:
r
1
=((a
0
,P), (a
1
,R), (a
2
,S), (b
0
,Q), (b
1
,R), (b
2
,S))
r
2
=((a
0
,P), (b
0
,Q), (a
1
,R), (a
2
,S), (b
1
,R), (b
2
,S))
r
3
=((a
0
,P), (b
0
,Q), (a
1
,R), (b
1
,R), (b
2
,S), (a
2
,S))
An external event c spawns a computation, defined
as a subsequence of a run, containing c together with a
set of all events that causally depend on c but exclud-
ing any other (causally dependent) external events d, e, ...,
and any events that causally depend on d, e, ....Com-
putations spawned by such events d, e, ... are caused by
the computation spawned by c. In a protocol run, we re-
quire all computations to eventually complete. A com-
putation is complete when the execution of all handlers
triggered by (events of) the computation have completed,
and no event is pending. In the runs above, we have two
computations: k
a
=((a
0
,P), (a
1
,R), (a
2
,S)) and k
b
=
((b
0
,Q), (b
1
,R), (b
2
,S)); they are not causally related.
Consider a complete run with a finite set of external
events E = {a
0
,b
0
, ...}. The protocol execution is serial
if for each two (distinct) external events a
0
and b
0
in E,ei-
ther (each handler of) the computation spawned by a
0
com-
mences after the computation spawned by b
0
has completed,
or vice versa. In the serial run, a computation k always pre-
cedes in time any computations caused by k, i.e. they can
commence only after k has completed. In our example, run
r
1
is serial since computation k
b
begins in this run after k
a
has completed; but runs r
2
and r
3
are not serial.
Two protocol executions are equivalent if, considering
the same sequence of external events and the same initial
state of the protocol, they produce the same (or equivalent)
state. A protocol execution satisfies the isolation property
if the execution is equivalent to some serial execution.
Consider runs r
1
, r
2
, and r
3
(with the sequence of ex-
ternal events (a
0
,b
0
)). Note that in runs r
1
and r
2
,the
computation k
a
visits all the microprotocol’s objects that
are shared with computation k
b
(i.e. R and S) before k
b
visits these objects. Thus, the effects of computation k
a
in
these runs, such as any modifications of the protocol’s state,
do not affect computation k
b
that is executing concurrently.
Hence, runs r
1
and r
2
satisfy the isolation property. Note
however that run r
3
does not, since k
a
can see any modifi-
cation of object S done by event b
2
of computation k
b
, and
k
b
can see any modification of object R done by k
a
.
If we would express and execute our example protocol
in Cactus, all the example runs r
1
, r
2
, and r
3
can occur, un-
less the protocol programmer would explicitly synchronise
the execution of given handlers to forbid some runs. In J-
SAMOA, the programmer can easily declare constraints on
the protocol’s concurrent executions so that only those runs
that satisfy the isolation property are permitted (e.g. r
1
and
r
2
, but not r
3
). Appia also supports the isolation property,
however it only permits serial executions, such as r
1
; other
correct, concurrent runs, such as r
2
, cannot occur in Appia.
3 Example
Below are fragments of three microprotocols, which are
part of the group communication system that we develop
using J-SAMOA; the system’s architecture is described in
[15], see also [6]. In complex middleware protocols such
as ours, it is often necessary to execute some activities con-
currently, in order to: 1) achieve good response time (for
instance when performing slow I/O operations), 2) avoid
blocking while processing different types of messages, or
3) simply to gain benefit of the multi-processor architec-
tures. Unfortunately, introducing a lot of concurrency can
lead to programming errors that are notoriously difficult to
detect.
The goal of this section is to:
• illustrate on concrete protocols the model defined in
Section 2, and J-SAMOA’s programming constructs,
• explain the problem of errors that are due to lack of
synchronisation between concurrent events,
• describe some simple solutions to the problem that are
often used by protocol designers (in the next section,
0-7695-2132-0/04/$17.00 (C) 2004 IEEE
Proceedings of the 18th International Parallel and Distributed Processing Symposium (IPDPS’04)
we demonstrate a more straightforward and efficient
solution using J-SAMOA’s isolated construct).
The J-SAMOA framework, implemented as a Java library,
supports the event-based model of communication between
microprotocols, with basic primitives for binding event
types to event handlers and triggering the execution of
handlers. Microprotocols in J-SAMOA can be expressed
as subclasses of two (abstract) classes Protocol and
handler. Event types are defined as instantiations of a
class Event. Event types and names of handlers are first-
class programming objects. They can be passed as argu-
ments to functions and handlers, returned as results and
stored in data structures. Events can be bound to handlers
using bind; in our program this takes place upon creation
of the microprotocol objects.
Group Communication Example. The first microproto-
col, called RelCast, implements a reliable broadcast. It
builds upon a second microprotocol, called RelComm, that
implements reliable point-to-point communication. We as-
sume that sites can crash at any time. The third micropro-
tocol — Membership, maintains a view, i.e. a current set
of all sites that are considered to be nonfaulty. This set is
kept consistent across all sites. We omit a lot of uninterest-
ing details, focussing on the synchronisation problem and
the way how it can be solved in J-SAMOA. (For simplicity,
we present code in a Java-like language instead of giving a
concrete syntax in Java.)
Protocol Membership (ViewChange:Event, ...) =
{ ...
handler joinleave (op: {+,-}, site: Site)
trigger ABcast [op site];
handler deliverView (op, site: Site) {
view = view op site;
triggerAll ViewChange view; } ...
}
The Membership microprotocol transforms a view into a
new view each time a site joins or leaves the system (volun-
tarily or because it is suspected to have failed). To join a site
s,thejoinleave handler must be called. It uses trigger
to synchronously call a (single) handler bound to the event
type ABcast, passing as the handler’s argument an opera-
tion ’+’ paired with the new site to be joined. The event is
handled by the atomic broadcast microprotocol that atom-
ically broadcasts the value [+ s] to other sites using the
distributed consensus microprotocol (both microprotocols
depend on RelCast, and are omitted here). Upon deliv-
ering the value to handler deliverV iew (on each site) by
the atomic broadcast protocol, the updated view view + s
is propagated locally to all interested microprotocols using
a (synchronous) event of type V iewChange. To issue the
event, the triggerAll construct is used, which accepts
event types that can be bound to many handlers.
Protocol RelCast (SendOut : Event,
DeliverOut : Event, Bcast : Event,
FromRComm : Event, ViewChange : Event) =
{
GroupView view = top.initialView();
handler bcast (m : Message) {
for all site in view
trigger SendOut (m, site); }
handler recv (m : Message) {
if (new message m) then {
bcast m;
asyncTriggerAll DeliverOut m; }}
handler viewChange (new_view)
{ view = new_view; }
bind Bcast bcast;
bind FromRComm recv;
bind ViewChange viewChange;
}
To reliably broadcast a message m to a group of sites, the
bcast handler of RelCast can be called. The body of bcast
(see above) issues an event of type SendOut (or, we simply
say “event SendOut”), one for each site in the view;the
events carry the message m and a target site site. The event
type SendOut is bound to a handler send of the RelComm
microprotocol (explained below), which tries to send the
message to a given target site. Note that upon receipt of the
message on another site for the first time (see handler recv
above), the message is rebroadcast again so that it will be
delivered to all sites in the current view, even if the sender
would crash in the middle. The message is delivered locally
using the (asynchronous) Deliv erOut event.
Protocol RelComm (FromRComm : Event,
SendOut : Event, FromNet : Event,
ViewChange : Event) =
{
GroupView view = top.initialView();
handler send (m:Message, target:Site) {
if (target in view)
try to send m to target; }
handler recv (m:Message, sender:Site) {
if (sender in view)
asyncTriggerAll FromRComm m; }
handler viewChange (new_view)
{ view = new_view; }
bind SendOut send;
bind FromNet recv;
bind ViewChange viewChange;
}
To reliably send a network message to a target site, the han-
dler send of the RelComm microprotocol can be called; it
tries to deliver the message (with possible retransmissions
in case of failures, omitted here). The message is discarded
if the target is not known. Before delivering a message on
the target site, the microprotocol checks if the sender is in
0-7695-2132-0/04/$17.00 (C) 2004 IEEE
Proceedings of the 18th International Parallel and Distributed Processing Symposium (IPDPS’04)
the current group view (see handler recv ); if so then the
F r omRComm event is issued. All microprotocols bound
to this event (such as the recv handler of the Rel Cast pro-
tocol) will get the message.
Both RelComm and RelCast depend on a group view.
RelComm tries to deliver a message only to a site that is
in the current view (this requirement is necessary to imple-
ment finite buffers, we omit details). Therefore RelCast
can reliably broadcast only to those processes that are within
the view seen by Rel Comm. Thus, each time the view
has changed, a new consistent view must be propagated
to Rel Comm and RelCast by Membership, soon af-
ter Membership’s distributed view-change algorithm has
terminated on each site. For this, Membership notifies
each interested microprotocol about the new view by is-
suing the V iewChange event; to deliver views to all in a
sequential order a synchronous triggerAll is used (see
Membership’s code). Then, the viewChange handlers
of RelCast and RelComm, which have been bound to
V iewChange, can update their local copy of the view.
Problem. Imagine that Membership has just installed a
new view, with a new site s added, i.e. on each site it has
added s to its local view, and issued a local V iewChange
event. Suppose a message is being broadcast using this new
view. Then, if the message has been received by RelCast
on some site for the first time, it has to be rebroadcast again
(to satisfy RelCast’s algorithm). However, if RelComm
which is used by RelCast to reliably send the message has
not yet handled the local V iewChange event informing
(locally) about the new view, it will not send the message to
s, thus breaking the algorithm. The message will be silently
discarded since RelComm does now know about s.
Restricted Solutions. Simple, although restricted solu-
tions to this problem are, either to: 1) piggyback the view
with every message and require each microprotocol to refer
to it (then a local copy of the view is obsolete), 2) demand
the current view from M embership by each microprotocol
(in particular RelComm) before processing any incoming
message, or 3) provide some form of synchronisation be-
tween events so that the application/network messages can-
not be processed by any microprotocol while the micropro-
tocols are in the process of updating their local views. The
first two solutions provide a mechanism which is somehow
stronger then actually needed; they either require redundant
data in messages (in the first case), or depend on some ad-
ditional intermodule communication (in the second case).
The third solution may unnecessarily restrict the amount of
internal parallelism.
Solution by Isolation. Note that to solve our problem it is
sufficient if the (correct) computations of the protocol, i.e.:
processing each message from the network, and processing
each message from the application, would be executed con-
currently, but in such a way that the isolation property holds.
(So essentially, the outcome would be the same as when us-
ing synchronisation suggested in the paragraph above.) In
J-SAMOA, such desired behaviour can be simply declared,
by implementing any external events of a protocol using a
construct isolated. In the next section, we describe the
construct and explain how to use it in our example protocol.
4 Language Support for Isolation
Execution of isolated Mein J-SAMOA spawns a
new computation, where M is explained below, and e is
code that calls some handler (or handlers) of the new com-
putation. Each computation is executed by a separate J-
SAMOA thread (if parts of a computation need to be ex-
ecuted concurrently, new threads can be created dynami-
cally, either explicitly with a suitable construct, or implicitly
through asynchronous event triggers). The runtime system
guarantees that the concurrent execution of computations
satisfies the isolation property defined in Section 2.
The basic construct isolated Merequires to specify
a collection M of all microprotocols whose handlers may be
called by the computation. For instance, an external event
a
0
in Figure 1 can be expressed as isolated [PRS]
{ trigger a
0
m; }, where P , R and S are all micro-
protocols (handlers) that may be called, and m is a mes-
sage. An error exception is thrown in the thread that called
isolated, if the computation will attempt to call a han-
dler of a microprotocol that is not in M. There is no prob-
lem if some microprotocol declared in M is not called by
the computation. In the strongly-typed language, the proper
value of argument M could be inferred statically.
We currently do not deal with dynamic binding; all han-
dlers declared in M must be bound before isolated com-
mences and cannot be (re)bound inside any computation.
Example Revisited. In our example in Section 3, external
events are requests, which are made by Network Module to
call handler(s) of event type FromNet , and by Application
Module to call handler(s) of Bcast. Thus, a message m
received from the network must be injected to upper pro-
tocols by Network Module using isolated [relComm
relCast ...] {trigger FromNet m;}, where
relComm, relCast , ... are all microprotocol objects whose
handlers may be called by the spawned computation; simi-
larly for the other external event type.
We have also implemented a few more variants of
isolated (each one to be used exclusively) that permit
to have more parallelism by optimising unnecessary block-
ing (in the worst case, they behave like isolated). How-
ever, they demand some additional (orthogonal) properties
of protocols to be known. It may not always be possible to
0-7695-2132-0/04/$17.00 (C) 2004 IEEE
Proceedings of the 18th International Parallel and Distributed Processing Symposium (IPDPS’04)
