A Case Study on Event Dissemination in an
Active Overlay Network Environment
Sérgio Duarte, J. Legatheaux Martins, Henrique J. Domingos, Nuno Preguiça
DI/FCT/UNL
Quinta da Torre, 2829-516 Caparica - Portugal
+351-212948300
{smd, jalm, hj, nmp}@di.fct.unl.pt
ABSTRACT
In this paper, we describe a case study of the design and
development of a group-conferencing tool suite, built on top of an
overlay network based event dissemination framework, which is
extensible via quality of service template plug-ins. We explain, for
each of the tools, how the framework built-in conveniences were
explored to create simple but effective distributed solutions,
backed by the appropriate quality of service templates, whose
design we also discuss.
Keywords
Case study, event dissemination, quality of service (QoS),
multicasting, overlay networks, active networks.
1. INTRODUCTION
Distributed application design is closely tied to the problem of the
quality of service offered by the support communication channels.
In general, for a given problem, a too weak quality of service
tends to put an excessive burden on the application, which has to
overcome the communication infrastructure shortfalls on its own.
On the other hand, an excessive quality of service is wasteful
because it normally comes with a matching price tag somewhere.
Ideally, one should strive for a balanced compromise between the
two, aiming at simpler applications backed by communication
support with the “right” quality of service. This has been
recognized in many fields of distributed computing and, naturally,
also in the more specific context of messaging middleware and
event systems [1][2][6].
Our work in the context of distributed event dissemination tackles
this precise challenge of designing a flexible, generic event
dissemination framework, capable of providing the means to
easily and incrementally build communication support channels
with just the “right” quality service needed in each situation. We
have addressed this problem by creating a solution based on
pluggable QoS templates that leverages its overlay-network
oriented architecture to achieve those goals. We want to show that
this may prove to be a viable alternative to the “one size fits all”
approach.
In this paper, we intend to describe the experience gained from
the development of a group-conference tool suite built on top of a
framework that advocates principles that go deliberately against
rigid, “one size fits all” approaches in the context of distributed
event dissemination.
2. CASE STUDY APPLICATION
The case study JAVA application is a barebones group-conference
tool suite, comprising videoconference, moderator and chat tools.
It allows a user to join a named group session, monitor the status
of other users and engage in chat or videoconference activities. A
moderator tool is included to help the audio coordination of
videoconference sessions involving multiple participants.
The objective of this case study is to test the claim that an
expected positive impact on application development supported
by data dissemination with the “right” quality of service (QoS) is
achievable and viable in an event dissemination framework
extensible via specific QoS template plug-ins.
In broad terms, the application developed consists of a desktop
where the individual tools are launched and manipulated. A
sample screen capture is shown in Figure 1. The desktop provides
an updated view of the status of the users enlisted in the current
session. Videoconference activities, within a session, are achieved
using complementary sender and a receiver tools and involve
encoding, multicasting and presenting RTP [3] A/V streams. An
optional moderator tool allows informal dialog coordination, by
enabling and muting the appropriate audio streams, according to
the evolving state of a global queue of enrolled participants. A
chat tool makes up the last of the desktop components.
Figure 1 - Sample screen capture of an ongoing session.
To test the aforementioned claim, the entire communication
requirements posed by this tool suite have been strictly fulfilled
by the amenities of the event dissemination framework, by
developing framework plug-ins with the appropriate QoS classes,
as required by each application component. Therefore, we must
Permission to make digital or hard copies of part or all of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full cit ation on the first page. To copy otherwise, to
republish, to post on servers, or to redistribute to lists, requires prior
specific permission and/or a fee.
Copyright 2003 ACM - 1-58113-843-1…$500.
1
highlight that this case study focuses on the problem of flexible
event transportation and sidelines other key aspects of event
dissemination such as filtering. In doing so, we intentionally
stressed the event transport facet of the framework by evaluating
its feasibility in dealing with a scenario with communication
needs closer to the data multicasting problem.
3. DEVELOPMENT FRAMEWORK
The tool suite is built on top of a JAVA-based event
dissemination platform named DEEDS. DEEDS has been
designed to be as flexible and adaptable as possible and aimed at a
broad range of applications and execution scenarios. The guiding
goals of the framework are the extensibility and configurability of
existing features, as a way of satisfying the requirements of large-
scale, heterogeneity and mobility in specific contexts.
DEEDS advocates a general-purpose solution in the sense that it
can be easily adapted to particular problems, or greatly eases the
creation of custom solutions using existing features as guiding
blueprints. A small set of simple and intuitive concepts have been
deliberately used to foster an incremental approach towards
problem solving that capitalizes on existing experience.
3.1 Event Dissemination Model
DEEDS implements the well-known publish/subscribe paradigm,
enhanced with a feedback operation allowing event consumers
and event sources to engage in one-to-one event exchange
dialogs. These operations are supported over active event
channels that designate named instances of particular QoS
templates. These QoS templates correspond to system-level plug-
ins that execute in the nodes of the event dissemination overlay
network and provide the routing logic needed to direct the event
stream produced by the publish and feedback operations.
The event dissemination model offered is also protocol
transparent, meaning that there are no references to specific
communication protocols at either the application level or within
the QoS templates themselves. Actual protocol bindings are
relegated to the deployment phase and subjected to the
administrative policies of each particular site.
Figure 2 – DEEDS’ Overlay network architecture, showing
the tree node types linked by various kinds of transports.
3.2 Platform Architecture
The event dissemination model summarized above is matched by
a distributed architecture designed with large-scale and
heterogeneity support in mind. A three-tier overlay network of
nodes makes up the core of the event dissemination infrastructure,
as shown in Figure 2. The first tier of this logical network is
known as the backbone and its server nodes typically handle the
more demanding routing operations. The second level is made of
a mix of secondary server nodes and client nodes (applications),
while solely client nodes compose the third tier. In every node,
regardless of type but, with different contexts, instances of the
QoS template plug-ins are executed to handle event forwarding.
Routing of events and exchange of control messages between
nodes is forwarded over transports, which are wrappers that
abstract the actual communication links connecting the involved
processes. Use of a heterogeneous mix of transports to form the
overlay network is allowed, thus it is possible to use TCP, UDP,
IP Multicast, HTTP or other protocol based transport at the same
time to accommodate different administrative policies.
The dissemination architecture also comprises a data repository,
known as the system registry, where static-persistent configuration
and dynamically collected volatile data is kept in the form of
attribute-value pairs. Portions of the registry have a local scope
and deal, essentially, with information about each node particular
configuration and runtime status. The remaining of the registry is
globally reachable (on demand) and is replicated (lazily) among
all server nodes. This is the place where DEEDS stores persistent
information that is relevant to every node, such as the event
channel directory that lists the names of known channels and their
bindings to the QoS templates.
3.2.1 Node architecture
The primary job of a DEEDS node is to provide the execution
environment for the event channel QoS template instances. Event
routing within a QoS plug-in typically involves accepting
incoming events and control messages, updating the state of the
node, and sending event and control messages to other nodes over
the appropriate transports. A node, depending on its type, also
runs a number of background services. These services exchange
information with their counterparts on other nodes to perform
housekeeping functions and provide a monitored view of the
status of the dissemination network. One of these services, for
instance, is responsible for maintaining the node’s system registry
replica. An explanation of the most relevant services comes next.
3.2.1.1 Backbone Monitoring Services
These services are two intertwined, complementary processes that
only run on the first tier, backbone nodes. Their purpose is to
monitor the overlay network and assemble a structured view of
the overlay network backbone.
One of the two is the Hello service, which continuously probes
the list of currently known (backbone) nodes, one by one, to
determine which are active and to obtain an estimate of their
distance. A scheduler within the service assigns higher priorities
to nearby or “critical” nodes, so that the allotted bandwidth is not
wasted on probing irrelevant nodes that are too distant in terms of
latency or spanning tree hops. The Linkstate service completes the
pair; its task is to efficiently deliver the data gathered by the Hello
service to the other backbone nodes and collect theirs, so that a
global perception of state of the backbone is achieved. To attain
this, each node periodically publishes its “hello data” in a
dedicated special broadcast event channel. The data is encoded in
such a way that, with a modest increase in size, also carries the
node’s current assessment of the “best” backbone spanning tree.
Embedding a spanning tree in each of these messages allows the
broadcasting to be achieved by source routing the message to next
nodes in the tree path. This scheme is advantageous because no
special coordination among the nodes is required to avoid cycles
or to detect duplicates; it permits the Linkstate service to rely on
1
st
tier (backbone)
2
nd
tier
primary server nodes
secondary server nodes
c
li
e
nt application nodes
unicast transports
m
ulticast
transports
3
rd
tier
2
itself to improve recursively its own routing information. As a
result, the global view of the backbone these services provide
makes it possible to obtain good spanning trees directly with
graph theory algorithms. The minimum spanning tree (MST)
algorithm is one of them but, although simple and lightweight, it
tends to produce deep, meandering trees, which is not desirable.
Instead, we prefer to use a spanning tree derived from a spanner
graph algorithm, which adds shortcuts to the MST so that the
distance between any two nodes in the spanner does not exceed by
a given factor their direct distance. The depth of the resulting
spanning trees can be finely controlled using the spanner factor,
while keeping the tree cost effective.
The information received through this service is also used to
gather knowledge about fresh backbone nodes. Finally, the
spanning tree advertised by the node with the lowest identifier is
taken as the official one and used to produce multicast and unicast
routing tables that, in turn, can be employed to drive the event
routing in other QoS templates plug-ins, such as the one used by
the system registry management service summarized next.
3.2.1.2 Registry Management Service
This node service manages the global, replicated portion of the
system registry. The service runs on every node but, since client
nodes only keep a volatile cache of the system registry, the
operation of the service in these nodes is somewhat restricted.
The service updates the registry in two different ways. There is a
low bandwidth proactive replication process that periodically
multicasts registry items in a dedicated event channel. But, more
often, updates to the registry are the result of lookups that cannot
be resolved locally and are sent to other nodes in the form of
queries. Both processes rely on a tailored event channel QoS
template to send and receive information. This event channel can
both multicast registry items and queries away from a source or
unicast replies towards a destination, one single hop at a time in
both cases, querying and feeding system registries along the way.
3.3 Programming Model
DEEDS programming model is expressed in the JAVA
programming language and assumes execution in a standard
JAVA environment. The programming library consists of a set of
user-level programming interfaces intended for the development
of applications. And, a set of system-level classes for system
enhancement, which allow the creation of additional node support
services, novel QoS template plug-ins and transport classes.
A flexible concept of event is used, representing a reasonably
small, self-contained notification, composed by a pair of items: a
main payload, in the form of an arbitrary “serializable” JAVA
object; and an envelope object, whose particular class may be
specific to each event channel type (represented by its supporting
QoS template). Both event components are optional, which means
that empty events are allowed. Data overlap between the two is
not restricted in any way but is wasteful and should be avoided.
The role of envelope objects can be seen as a way of passing
arbitrary control information to the event dissemination
infrastructure to avoid the need to scrutinize the main event
payload for that same purpose at a greater cost. For instance, the
envelope can be a rough description of the main event payload, to
assist QoS templates in optimizing event dissemination based on
aggressive event filtering practices. Or, more simply, an envelope
can be an expiration deadline to allow the QoS template of the
event channel to automatically discard late events before reaching
some of its subscribers and, thus, free network resources earlier.
The counterpart of the envelope is the criteria object used in
subscription operations. These are generic event filters operating
over envelope types that are used to check the envelopes of
incoming events to select those to be delivered to the application.
Together, envelopes and criteria form the basis of the event
filtering capabilities of the framework.
The event model also includes the notion of receipt objects,
whose purpose is to aggregate and return system-generated
information associated with an event, such as event-source
identifiers, sequence numbers and subscription “handbacks”.
These receipts cannot be fabricated and are important for the
feedback operation because they identify the event source targeted
by the operation.
3.3.1 Application Programming Interfaces
The basis of the programming interfaces is the EventChannel
class, which provides the access points to the event dissemination
operations according to the publish/subscribe/feedback model.
References to these objects are obtained by performing a lookup
operation on a global event channel directory. The only parameter
required is the string name of the desired event channel. Creation
of a new event channel is accomplished with the clone operation,
which takes the intended name for the new channel and the name
of the QoS template plug-in, in which the new channel will be
based upon. The use of “clone” for the operation name is meant as
way of emphasizing the idea that the new event channel will be a
copy or clone of a prototype channel already present and accepted
into the system.
Having obtained a reference to an EventChannel object, the
application can follow the expected programming pattern of the
publish/subscribe paradigm. The specifics being that the publish
operation requires an envelope and an object (the main payload)
and returns a receipt. To be notified an application performs
subscribe operations, specifying criteria objects to filter out
undesired events based on their envelopes. The feedback
operation fits in the model to allow a notified application to
engage into a one-to-one dialog with a specific event source; it
differs from the publish operation by requiring a receipt of a
previously received event as an extra argument.
The following code excerpt exemplifies the use of these main
programming interfaces in two basic publisher and subscriber
applications. For clarity and brevity, only partial argument lists
are shown.
import deeds.api.*;
public class Publisher implements EventFeedbackSubscriber {
EventChannel c ;
public Publisher() {
Deeds.Directory().clone( “QoStemplate”, “channel_name”);
c = Deeds.Directory().lookup(“channel_name”);
c.subscribeFeedback( criteria, …, this);
while(…) c.publish( envelope, payload );
c.unsubscribe(…);
}
void nofifyFeedback( Receipt r, Envelope e, MarshalledEvent m ) {
Object payload = m.getEvent();
…
c.feedback( r, envelope, payload2) ;
}
}
3
import deeds.api.*;
public class Subscriber implementsEventSubscriber,EventFeedbackSubscriber{
EventChannel c ;
public Subscriber() {
c = Deeds.Directory().lookup(“channel_name”);
c.subscribe( criteria, …, this) ;
c.subscribeFeedback( criteria2, …, this);
}
void nofify( Receipt r, Envelope e, MarshalledEvent m ) {
Object payload = m.getEvent() ;
…
c.feedback( r, envelope, payload2) ;
}
void nofifyFeedback( Receipt r, Envelope e, MarshalledEvent m ) {
Object payload = m.getEvent();
…
c.feedback( r, envelope, payload2) ;
}
}
3.3.2 QoS Template Development
Extending the framework capabilities is in great part tied to the
development of new QoS template plug-ins. In their essence,
event channel templates implement a particular routing protocol
across the overlay network to deliver events to interested parties.
A QoS template must deal with two separate streams of events,
the multi-point stream that is produced by publish-operations, and
the (optional) unicast stream consisting of feedback events. To
achieve this purpose, the plug-in can also format any appropriate
control messages it needs and exchange them with other nodes.
Unless the desired QoS is very basic, design of a new plug-in can
be a complex task. To make their development easier it is possible
to capitalize on useful information already available in the node.
This information is made accessible through the system registry
and is presented in the form of dynamic objects that other
processes keep updated and store in named containers. Containers
keep track of changes in the information they store and notify
interested parties. This scheme allows QoS plug-ins to
synchronize their state (a privately computed routing table, for
example) in reaction to changes in the containers they monitor.
The framework already provides a number of these containers
such as, a list of known backbone nodes and the transports
available to reach them, a list of local subscribers for each event
channel, a current view of the overlay network links, a low-cost
spanning tree covering the backbone nodes and the associated
broadcast and unicast routing tables. These resources are a great
help in the programming of new plug-ins, as will be shown in the
following sections, where we describe the ones that were
developed for the purpose of the group conference tool suite. A
source example is also provided in the appendix at the end of this
paper.
4. CASE STUDY DEVELOPMENT
In this section we describe the most relevant aspects that guided
the development of each of the applications that make up this case
study. We recall that the challenge we have undertaken has been
to show that a combination of the right quality of service in
communication can lead to simple (peer-oriented) applications
that address elaborate problems. Furthermore, we want to verify
that, with an acceptable effort, the desired QoS is feasible within
plug-in model of the DEEDS framework. In any way, we want to
advocate here that this is the best way to solve these problems but
that it is a good, promising way; a viable alternative to more
popular approaches such as the centralized client-server model.
4.1.1 Video-Conference Tools
The videoconference tools are more precisely described as being
two separate programs, the transmitter that captures, encodes and
transmits the a/v streams and the receiver that decodes and
presents them. For obvious reasons, we used the Java Media
Framework [4] to create these programs. It allows a JAVA
application to easily capture, encode or transcode audio and video
streams in a number of standard formats. One feature of JMF that
is particularly pertinent for this case study is its ability to deal
with RTP encoded media streams. RTP [3] (and its companion
protocol RTCP) is a IETF sponsored transport protocol,
specifically designed for transmitting real-time data, such as
audio, video over multicast or unicast network services. What
makes RTP so attractive is that it has been made independent of
the underlying transport and network layers, which enables us to
encode RTP streams and multicast them over our event
dissemination overlay network.
The core effort in delivering RTP streams over DEEDS rested in
the creation of the appropriate RTPConnectors adaptors according
to the JMF specifications, which are the actual objects used
internally to have a media source send out the RTP and RTCP
packets and gather reception statistics reports (RTCP packets)
from its listeners. Implementing these connectors in DEEDS was
no trouble at all, and merely consisted in having the connector
publish the RTP and RTCP packet stream in a given event
channel and use the feedback operation to report back the RTCP
packets to the source.
The greater undertaking in the development of these tools was the
selection of the best event channel type for the task and
implementing the corresponding QoS template plug-in. Given the
nature of the problem, the desired event channel type had to offer
a light-weight multicast service with as low as possible latency
and jitter. In this particular case, reliability is not an issue and
dropping a few packets is tolerable. Moreover, a simpler single-
source multicast routing protocol solution can be adopted
provided each sender uses its own channel, which is actually
desirable in this case. With these characteristics in mind, we
implemented a SingleSourceUnreliableMulticast plug-in.
The plug-in implements its multicast routing protocol capitalizing
heavily on network state data already provided by the normal
operation of the framework. It essentially creates a tree of
backbone nodes, see picture above, rooted at the node where the
4
event source is connected and spanning the nodes with registered
subscribers. A special rendezvous node selected independently for
each channel, by mapping the channel id to a node id and finding
the best match in the list of backbone nodes, acts as a temporary
root. A node joins the multicast tree, in response to changes in its
registrations container, by sending a JoinRequest control packet
towards the root of the tree. These requests travel towards the root
one hop at time (except the first time when they have to reach the
root via the rendezvous node). Each node merges all the requests
it receives from lower level nodes into a larger compound request.
As a result, the root is not flooded with many single requests but
receives just a few larger ones. When the root detects a new node
after merging together all the requests (or when it is time to
refresh the tree) the channel’s multicast tree is updated. The new
tree is obtained by finding the minimum spanning tree covering
the root and the subscribed nodes, according to the current state of
the backbone. It is then propagated down to all nodes, by having
each node send it to its children and so on, according to the
topology conveyed in the updated tree. A node knows that it has
joined the multicast group when it receives a tree update that
includes it; to leave the multicast tree it sends LeaveRequest
packets directly to the root packet until it gets a confirmation; the
root in turn updates the tree in response.
4.1.2 The Desktop
The desktop is the main application that glues everything
together. Its purpose is more than just to be a background where
the tools are launched and manipulated. It has the important role
of managing the group session by monitoring the status of its
participants and providing the necessary binding information that
turns the isolated tool instances into a closely coupled group.
The desktop relies on a dedicated event channel for its operation.
The name of this event channel identifies the session that the user
is joining. The remaining tools rely also on this name to complete
binding information by appending appropriate suffixes to derive
their own event channels’ names.
During the course of its operation, the desktop uses its event
channel to publish a periodic heartbeat that informs other
desktops in the same session about the presence of this
participant. The desktop collects these heartbeats (including its
own) to keep a list of the session’s participants. This list is
presented graphically on the left side of the desktop, showing both
online participants and offline ones. A participant is considered
offline if the last time its heartbeat has been heard exceeds a
preset amount of time.
The type of event channel required for the correct operation of the
desktop in the terms described differs from the one used in the
video conferencing tool in the fact that it has a clear a multi-
source requirement. An unreliable type can be used and has been
developed but we later decided to replace it with a reliable
version. The difference being that a reliable event channel allows
for a tighter tolerance in heart beat timings because with a reliable
event channel one only has to consider delayed heart beats,
whereas with an unreliable one, lost heart beats must take into
consideration and, therefore, one can only reasonably conclude
that a participant is offline if a certain number of consecutive
heart beats failed to arrive.
The two QoS template types were developed anyway, basically
because it makes sense to produce the reliable version after the
unreliable one. Moreover, the UnreliableMulticast QoS template
is essentially an extension of the single-source version developed
earlier. The changes made consisted in also having the nodes with
sources join the multicast tree, in addition to the nodes with
subscribers, and always choosing the rendezvous node as the root
of the multicast tree. The JoinRequest handling and related
multicast tree updating was kept the same. The only additional
modification required was about the routing of the actual events.
They no longer travel down the tree, as before, but at each node
are sent away from their point of origin along the branches of the
multicast tree (now interpreted as a graph).
This multicast routing algorithm will perform poorly if the
number of nodes that are exclusively a source of events is much
larger than the receiver nodes. However, this does apply in the
case of the desktop application because every node is always both
a source and a subscriber.
The ReliableMulticast template that was eventually used in the
desktop application solves the problem of lost packets with a
small fixed-sized packet queue, at each node of the multicast tree,
one for each source. Holes in queue are filled by sending a
negative acknowledgement packet, listing a certain number of
missing packets, one hop towards the source. Every so often, a
node is also required to send a packet, one hop towards the
source, acknowledging the last event in sequence it received. At
each level of the tree (in respect to the source in question) these
ACK packets are aggregated into larger compound ones to avoid
the problem known as ACK implosion. The source advances the
queue in step with the lowest sequence numbered ACK received
and drops any node that fails to advance its sequence number for
too long.
4.1.3 Moderator Tool
The purpose of this tool is to help coordinate an ongoing
videoconference session by muting the audio streams of selected
participants, while keeping the video going. This tool is rather
simple in its approach; it manages a queue of enrolled
participants, monitoring changes to the queue and only allowing
the participant at the head of the queue to talk, keeping the others
silent. The actual tool consists of a simple graphics interface that
shows the state of the queue, with its enrolled participants, and
allows a participant to enter or leave the queue. No fault-
tolerance features have been implemented but, given its overall
informal nature, this problem would addressed by allowing
anyone to remove a silent participant from the queue.
To keep it simple and peer-oriented, all instances of the tool
behave in the same way, none having a special role. Changes to
the queue are made by publishing enter or leave events to an
event channel that every moderator tool (in the same session)
subscribes, with the sanity of this whole process resting in the
event channel’s ability to keep all the queues consistent. The
actual muting and enabling of the audio streams is done indirectly
by publishing appropriate events to another event channel shared
with the all the tools running on the same desktop, video-
conference ones included. This is an event channel that only spans
one particular desktop and is a clone of the built-in LocalLoop
QoS template.
This simple approach to the moderator tool was thought viable on
the assumption that a suitable QoS template could be developed
easily enough to not completely offset what would be gained in
the first place. Specifically, the moderator tool required a multi-
source reliable multicast event channel, with the additional need
5