A feature-oriented alternative to implementing reliability connector wrappers

TL;DR: In this paper, the authors investigate the relationship between specification features and implementation-level features and suggest techniques for designing middleware frameworks and composition tools that more explicitly reify and expose the features specified by connectors and connector wrappers.

Abstract: Connectors and connector wrappers explicitly specify the protocol of interaction among components and afford the reusable application of extra-functional behaviors, such as reliability policies. Ideally, these specifications can be used for more than just modeling and analysis. We are investigating how to use them in the design and implementation of the middleware substrate of a distributed system. This paper reports our experience elaborating connectors and connector wrappers as instantiations of a feature-oriented middleware framework called Theseus, which supports the design of asynchronous distributed applications. The results of this case study indicate that the relationship between specification features and implementation-level features is not one-to-one and that some specification features have complex, often subtle, manifestations in Theseus’ design. This work reports the lessons learned designing these strategies and suggests techniques for designing middleware frameworks and composition tools that more explicitly reify and expose the features specified by connectors and connector wrappers.

Summary

1 Introduction

• Increasingly, distributed computing systems are deployed in volatile environments in which network connectivity is sporadic and unreliable.
• Automatic retry detects when a service request fails and automatically resends that request rather than propagating the exception to the client program.
• These connector wrappers impose policies 2 on an existing connector specification by extending and/or restricting its observable behaviors.
• Using refinements, both duplication of functionality and masking of behaviors can be avoided.
• The authors believe that implementing reliability strategies as refinements will enable system architects to easily construct efficient middleware solutions subject to a variety of reliability policies.

2 Background

• By way of background, the authors first introduce the use of wrappers to enhance reliability in distributed systems and the connector-wrapper formalism, which models the behavior of these wrappers.
• Wrappers and connector wrappers exhibit a useful structural correspondence between implementation and specification; however wrappers also incur redundancies and inefficiencies that are unacceptable to resource-constrained devices.
• The authors contend that reliability enhancements can be implemented so as to exhibit the same correspondence with connector wrappers but without incurring the redundancies and inefficiencies of wrapperbased implementations.
• The key is to implement the enhancements as reusable refinements under the AHEAD model of composition, which supports the definition of modules whose implementation abstractions are left open for further refinement by other modules under composition.
• This section concludes with a brief introduction to the AHEAD model.

2.1 Reliability-enhancing Wrappers

• Software architects often wish to augment communication among the components in a distributed system to incorporate extra functionality, such as logging, encryption, and even strategies for enhancing reliability in the face of network failures.
• Interception has proved useful for adding reliability to a variety of system calls (C.f., [10, 11]).
• To preserve the original interface, these wrappers are implemented based on the proxy pattern [15]2.
• Such orphans, even though no longer actively contributing to the behavior of the system, remain in place and continue to consume resources, both computational and spatial.
• More recent work has investigated the development of software modules that, when composed with a base system, are able to reconfigure that system.

2.2 Connector Wrappers

• The term connector refers to a mechanism for composing architectural components [20].
• Connectors abstract communication mechanisms and protocols, such 5 as procedure calls, remote procedure calls, pipes in a pipe-and-filter system, and many of the communication services that are generally referred to as “middleware”.
• Because connector wrappers faithfully model the structure and behavior of wrappers, an architect may specify a new connector as the composition of one or more connector wrappers with some connector and then semi-automatically generate a conforming implementation.
• Spitznagel’s system provides generation tools that implement this capability.
• The next section describes a model of composition that is algebraic in nature and that supports fine-grained recomposition of the base system, allowing for both reuse of abstractions and the avoidance of orphaned components.

2.3 AHEAD

• AHEAD is an algebraic model of software composition in which complex, featurerich programs are synthesized from base programs by applying reusable refinements [21].
• The composition of AHEAD refinements and simple programs is depicted visually using diagrams such as Figure 2(a).
• Here, the inner-most boxes are classes or class fragments.
• This composition is specified textually via the type equation comp1 = f1〈const〉.
• As the authors saw earlier, f1 and f2 refine the layers below them; here, this is formalized by the presence of a realm parameter that conveys that these layers augment layers of type X. Based on this small set of layers, many different compositions may be instantiated, e.g., f1〈const〉, f2〈f1〈const〉〉, and l1〈f2〈const〉〉.

3 Theseus

• One of the goals of their approach is to augment middleware services with reliability by refining the abstractions used in the implementation of these services rather than by treating the services as a black box and wrapping them with extra functionality.
• To accomplish this goal requires a middleware framework whose design exposes these major abstractions and makes them available for further 8 refinement.
• The Theseus model comprises two realms: MSGSVC, whose layers implement a variety of different message services, and ACTOBJ, whose layers implement variants of the distributed active object pattern.
• Most of the layers in these realms implement different reliability strategies (or low-level services that support different reliability strategies).
• Using Theseus, an architect can easily customize middleware services to support specific reliability policies without the duplication and efficiency burdens inherent in the use of wrappers.

3.1 Message Service

• In Theseus, the message service provides queue-like communication abstractions that implement a simple, reliable3 message-oriented middleware.
• A client of the message service sends data by enqueuing a message in a peer’s inbox (which typically resides in a separate address space on a remote machine) and receives data by retrieving messages from its inbox.
• Figure 4 depicts the MSGSVC realm, whose layers define and refine these abstractions.
• One such refinement is bndRetry (bounded retry), which augments an existing PeerMessenger5 to, in the event of a communication failure, suppress the communication exception(s) and retry some number of times (maxRetries > 0) before giving up and throwing the exception.
• In Figure 5, bndRetry refines classes of the rmi layer, namely, PeerMessenger.

3.2 Active Objects

• Theseus’ second realm is called ACTOBJ because its layers define classes and class refinements that implement different variations of the distributed active object pattern [6].
• The execution phase is initiated by the scheduler, which is a loop (running in the execution thread) that dequeues requests from the activation list to be executed.
• Once dequeued, requests are passed to the dispatcher to be invoked on the servant, which is an object that actually implements the behavior modeled by the active object [6].
• The distributed active object pattern follows the same basic architecture, except that the operations invoked by a client are executed in an object that lives in a separate address space.
• Among others, core contains two classes, StaticDispatcher and FIFOScheduler, which implement the DispatcherIface and SchedulerIface interfaces respectively, and which are designed to use subordinate services that are defined in the MSGSVC realm type.

3.3 Synthesizing Middleware Services

• To create a set of middleware services, the authors instantiate objects from the classes defined in the assembly core〈rmi〉, which is their most basic middleware assembly and the one that is refined to create all of the other variants.
• Notice also that the rmi classes are still visible in the assembly and are thus available for further refinement, as are the classes in core.
• The first two requirements are met by the bounded retry augmentation of the message service.
• This class implements the InvocationHandler interface; thus its instances can be passed to the static factory method that generates the dynamic proxy.
• To accommodate this possibility, the authors refine the TheseusInvocationHandler to transform these exceptions into the exceptions that the active object’s interface declares in its throws clause.

3.4 Efficiency Improvements in Bounded Retry

• To see how their design forestalls the duplication that arises when implementing reliability strategies via wrappers, consider a wrapper-based implementation of the bounded retry policy.
• The wrapper would have to be applied to the server stub, i.e., the object returned by RMI’s Naming.
• In Theseus, by contrast, the class of the object that is used to send the marshaled invocation, i.e., the class that implements the PeerMessengerIface interface, is available for further refinement.
• The composite (bold) layer in Figure 8 depicts the synthetic collaboration whose participants include the eeh-augmented invocation handler and the bndRetry-augmented peer messenger.
• Instances of these refined classes collaborate to implement their bounded retry strategy.

4 An AHEAD Model of Reliable Middleware

• In the previous section the authors implemented a strategy for bounded retry in two phases.
• The first phase refined the message service to suppress exceptions and retry some bounded number of times before giving up.
• All of them can be implemented using collectives that comprise multiple layers.
• As the authors will see, a base middleware, such as core〈rmi〉, corresponds to a middleware connector specification and collectives that implement a reliability strategy correspond to reliability connector wrappers.
• As the authors describe their model, they will make these correlations explicit.

4.1 A Reliable Middleware Model

• The authors base middleware (BM) is {coreao ◦ rmims}, which is equivalent to {coreao, rmims} 8 where the subscripts ao and ms indicate layers in the active object and message service realms, respectively.
• Second, the order of refinement is preserved.
• The collectives RS0 and RS1 correspond to reliability connector wrappers that augment BM.

4.2 Reliable Middleware Examples

• The authors now illustrate three applications of synthesis using the THESEUS model.
• As noted earlier, their bounded retry strategy is implemented by the bounded retry refinement of the message-service realm and the exposed exception handler refinement of active-object realm.
• Figures 8 and 9 visually depict the tighter binding of AHEAD refinements that is shown equationally in Equations 12-14.
• Upon failure of the primary, a failover refinement will suppress 16 the exception and silently switch over to the backup.
• Under these refinements, 1. A communication exception thrown by rmims will be suppressed by bndRetryms, which will attempt to reconnect and resend the marshaled request some bounded number of times.

5 Contrasting Implementation Strategies

• The authors believe that their AHEAD-style implementation of reliability strategies offers a useful alternative to wrappers without sacrificing the flexibility of composition and reasoning provided by the connector-wrapper formalism.
• The authors now describe the implementation of a more complex reliability policy, warm failover, and identify where their approach eliminates both redundancy and the need for additional logic for suppressing behavior.

5.1 Warm Failover

• Warm failover is a reliability policy that uses a backup server providing reliability via redundancy in a client-server architecture.
• The backup is “warm” in the sense that it is kept in sync with the primary via some strategy dependent mechanism.
• The client then promotes the backup to the role of primary, which means the client sends requests to and expects responses from the backup and the backup, correspondingly, accepts requests from and sends responses to the client.
• This policy assumes a “perfect” backup that will not fail, and, as such, does not account for the failure of the backup.
• To maintain the cache as such, the client is obligated to send acknowledgements (that comprise a response id) to the backup when it receives a response from the primary, indicating that response may be removed from the cache.

5.2 Theseus Refinements

• To implement silent backup, the authors must augment the client and create a backup that fulfills the responsibilities outlined above.
• To implement the client’s responsibilities, the authors send each request to both the primary and the backup, activate the backup in case the primary fails, and send acknowledgements to the backup as the primary’s responses arrive such that the backup may purge these from the response cache.
• The authors refine the invocation handler that participates in marshaling responses to store these in the cache rather than send them to the client.
• Further, the refined invocation handler implements ControlMessageListenerIfaceand is registered with the control message router 20 to listen for both acknowledgement and activate messages.

5.3 Implementing Silent Failover with Middleware Wrappers

• The authors now contrast their implementation of silent failover with a wrapper implementation constructed using Spitznagel’s wrapper transformations.
• Because their approach refines the peer messenger to send the marshaled request to both the primary and the backup in the message service, the authors avoid the processing redundancy inherent in marshaling the same invocation twice.
• While these wrappers work, the introduction of unique identifiers is redundant with the corresponding middleware identifiers used to coordinate requests and responses (such as CORBA’s object id[25] and RMI’s UID[26]).
• The more difficult part of managing the response cache is sending the expedited control messages needed to acknowledge responses and activate the backup server.
• If the backup is already sending responses, the client’s promotion of the backup to the primary should cause the client to begin accepting responses from the client rather than discarding them as before.

5.4 Discussion

• Both wrappers and AHEAD collectives an be used to augment middleware with reliability.
• Wrappers are more reusable in that they do not require the underlying middleware to have been designed according to an AHEAD model.
• With this reusability comes the potential for redundancy and inefficiency.
• Further, their type expressions make the structural composition of these systems clear.
• In such cases, computational and storage resources are at a premium.

Figures

Fig. 1. Adding functionality using wrappers.

Fig. 2. Layered refinement in AHEAD.

Fig. 7. Layers of a simple middleware.

Fig. 9. Grouping bounded-retry layers into a collective.

Fig. 3. Interfaces in the message service.

Fig. 10. Silent backup client configuration.

Fig. 8. Layered implementation of the bounded retry strategy.

Fig. 4. Message service realm layers.

Fig. 5. Visual stratification of bndRetry〈rmi〉.

Fig. 11. Backup server configuration.

Full text

A Feature-Oriented Alternative to Implementing
Reliability Connector Wrappers
J.H. Sowell and R.E.K. Stirewalt
Michigan State University, East Lansing, Michigan 48824 USA,
{sowellje,stire}cse.msu.edu
Abstract. Connectors and connector wrappers explicitly specify the
protocol of interaction among components and afford the reusable appli-
cation of extra-functional behaviors, such as reliability policies. Ideally,
these specifications can be used for more than just modeling and analysis.
We are investigating how to use them in the design and implementation
of the middleware substrate of a distributed system. This paper reports
our experience elaborating connectors and connector wrappers as in-
stantiations of a feature-oriented middleware framework called Theseus,
which supports the design of asynchronous distributed applications. The
results of this case study indicate that the relationship between specifi-
cation features and implementation-level features is not one-to-one and
that some specification features have complex, often subtle, manifesta-
tions in Theseus’ design. This work reports the lessons learned designing
these strategies and suggests techniques for designing middleware frame-
works and composition tools that more explicitly reify and expose the
features specified by connectors and connector wrappers.
1 Introduction
Increasingly, distributed computing systems are deployed in volatile environ-
ments in which network connectivity is sporadic and unreliable. In response, a
variety of reliability policies have been devised to shield users from the effects
of this volatility. For example, automatic retry detects when a service request
fails and automatically resends that request rather than propagating the excep-
tion to the client program. More sophisticated policies, such as failover, exploit
redundant servers: When a request to one server fails that request is automati-
cally forwarded to another rather than propagating the exception to the client
program. In each case, an unreliable service is promoted to one that is reliable
without altering the implementation of the unreliable service. Consequently, such
policies tend to be incorporated into the middleware layer by wrapping an un-
reliable middleware implementation with code that intercepts service requests
and performs the extra functionality.
To understand how to use wrappers to apply and compose reliability policies,
Spitznagel and Garlan [1] developed a technique based on a formal behavioral
model of architectural connection [2]. These connector wrappers impose policies

2
on an existing connector specification by extending and/or restricting its observ-
able behaviors. Moreover, each connector wrapper has an implementation coun-
terpart that can be applied to incorporate the policy into an existing middleware
implementation. These implementation wrappers compose with the flexibility of
their specification counterparts by treating the underlying middleware as a black
box. Unfortunately, the resulting implementations may incur redundancies and
inefficiencies that are unacceptable on the resource-constrained devices that are
most often exposed to volatile environments.
This paper describes an alternative implementation of connector wrappers
as reusable refinements rather than black-box wrappers. Reusable refinements
appeal to an algebraic model of software composition called AHEAD [3] and are
similar to ML functors [4] and mixin layers in C++ [5]. Refinements compose
functionally, just like wrappers; however refinement composition is more fine
grain and thus affords a tighter integration that enables the reuse and extension
of existing abstractions. Designers may thus customize a base middleware with
reliability strategies by applying refinements in a manner analogous to the ap-
plication of connector wrappers, and the resulting configurations will not exhibit
the redundancy and inefficiency introduced by implementation wrappers.
Our results exploit the fact that reliability strategies often employ the same
design abstractions that are used to implement basic middleware services. For
instance, many middleware systems use the asynchronous completion token pat-
tern [6] to demultiplex asynchronous operation requests and responses. This pat-
tern is also used to implement a strategy for warm failover [7] whereby clients
copy outgoing requests to a redundant backup server, which silently serves each
request in parallel with the primary. Were such a strategy implemented using
a wrapper, the wrapper would require logic for demultiplexing requests and re-
sponses with the backup server even though such logic exists in the underlying
middleware. Further, some reliability strategies may need to suppress behav-
iors, such as suppressing the responses sent by a server that is intended to play
the role of a silent backup. Wrappers suppress behavior by masking the observ-
able effects rather than forestalling the behavior itself. Using refinements, both
duplication of functionality and masking of behaviors can be avoided.
We believe that implementing reliability strategies as refinements will en-
able system architects to easily construct efficient middleware solutions subject
to a variety of reliability policies. To validate this claim, we implemented and
applied a set of reliability refinements to Theseus, which is an asynchronous
middleware implementation that we designed using the AHEAD model. In prior
work, we described three reliability refinements and showed that these exhibit
the compositional properties as their specification counterparts [8]. We have since
implemented other refinements, including one for warm failover that is useful for
comparing our approach with black-box wrappers. In the sequel, we describe the
design of Theseus and how refinements compose to produce new middleware that
incorporates various reliability policies. We then compare our refinement-based
implementation of warm failover to the wrapper-based design of [9].

3
2 Background
By way of background, we first introduce the use of wrappers to enhance relia-
bility in distributed systems and the connector-wrapper formalism, which mod-
els the behavior of these wrappers. Wrappers and connector wrappers exhibit
a useful structural correspondence between implementation and specification;
however wrappers also incur redundancies and inefficiencies that are unaccept-
able to resource-constrained devices. We contend that reliability enhancements
can be implemented so as to exhibit the same correspondence with connector
wrappers but without incurring the redundancies and inefficiencies of wrapper-
based implementations. The key is to implement the enhancements as reusable
refinements under the AHEAD model of composition, which supports the def-
inition of modules whose implementation abstractions are left open for further
refinement by other modules under composition. This section concludes with a
brief introduction to the AHEAD model.
2.1 Reliability-enhancing Wrappers
Software architects often wish to augment communication among the compo-
nents in a distributed system to incorporate extra functionality, such as logging,
encryption, and even strategies for enhancing reliability in the face of network
failures. Augmentation is accomplished by intercepting messages that cross the
boundary between client components and the communication or middleware
layer. Once intercepted, these messages are then either dropped, transformed
and then forwarded to their original destination, or routed to another destina-
tion. Interception has proved useful for adding reliability to a variety of system
calls (C.f., [10, 11]). The technique is now supported directly in modern middle-
ware systems (C.f., CORBA’s portable interceptor interface [12,13]).
1
This paper is concerned with reliability enhancements that are implemented
using wrappers, which serve to both mediate client access to a service as well as
augment that service with extra-functionality, as with interception. To preserve
the original interface, these wrappers are implemented based on the proxy pat-
tern [15]
2
. As an example, consider the addition of logging and data encryption
to messages sent by client components to a remote server component. Suppose
the class MiddlewareStub represents the type of the client-side stub object, such
as might be generated from an IDL specification of the server component. The
additional functionality is implemented by a hierarchy of wrapper objects that
conform to the class model in Figure 1. Commensurate with the wrapper pat-
tern, each class implements a common interface, which in this example is called
1
A powerful model of interception is captured by the composition-filters object model,
which allows designers to deploy filters that intercept and drop, modify, or reroute
messages among arbitrary objects in a system [14].
2
Readers familiar with GoF design patterns will notice wrapper is a synonym for the
adapter and not the proxy pattern; we use the term wrapper for consistency with
Spitznagel’s wrappers, which also implement the proxy pattern[9].

4
MiddlewareStubIface, and the two wrappers implement their methods by del-
egation. Software architects typically use wrappers to augment middleware with
LoggingTransformation EncryptionTransformation
<<wrapper>> <<wrapper>>
MiddlewareStub
MiddlewareStubIface
<<interface>>
Fig. 1. Adding functionality using wrappers.
new functionality, to suppress existing behaviors, or to mask faults.
While easy to implement, wrappers (and interception techniques generally)
suffer from two problems. First, components of the original system (that being
wrapped) may be “orphaned” when their behaviors are suppressed in favor of
behaviors introduced via interception and wrapping. Such orphans, even though
no longer actively contributing to the behavior of the system, remain in place
and continue to consume resources, both computational and spatial. Second, the
extra functionality may incur redundancies and inefficiencies that are unaccept-
able on the resource-constrained devices that are most often exposed to volatile
environments. In the case of wrappers, these redundancies owe to the treatment
of the service being wrapped as a black box whose internal resources (i.e., those
that might be reasonably reused by the extra functionality) are not accessible
to the wrapper. Interception-based techniques suffer a similar problem.
To overcome this obstacle requires the ability to reconfigure a base system
when augmenting it with extra functionality. The earliest work in this regard
uses static recomposition based on object-oriented frameworks, the most notable
of these being Schmidt’s ACE framework [16]. More recent work has investigated
the development of software modules that, when composed with a base system,
are able to reconfigure that system. Such compositions must be able to refine
existing components, including those hidden behind an opaque API, to support
extra-functional behaviors or replace them with components that do support
the desired behaviors. More recent work (C.f., [17, 18]) applies dynamic recom-
position to introduce reliability enhancements. In these cases, reflection and/or
meta-object protocols are used to reconfigure the original application to use the
components that best fit the environment at hand. Further still, such techniques
have also been combined with class loading technologies, in particular the JBoss
extensible middleware platform, to completely add and/or remove components
from a running system [19].
2.2 Connector Wrappers
The term connector refers to a mechanism for composing architectural compo-
nents [20]. Connectors abstract communication mechanisms and protocols, such

5
as procedure calls, remote procedure calls, pipes in a pipe-and-filter system, and
many of the communication services that are generally referred to as “middle-
ware”. More abstractly, a connector represents a pattern of interaction among
a set of components, called the roles of the connector. Allen and Garlan de-
veloped a formal model of connectors as stylized CSP specifications, thereby
enabling an architect to rigorously specify an architecture as the composition of
components and connectors and use formal analysis tools to reason about the
resulting behavior [2].
To facilitate reasoning in the presence of wrappers, Spitznagel and Garlan
developed a theory of connector wrappers, which are stylized CSP specifications
designed to compose with a connector specification to yield a new connector,
whose behaviors are an extension or restriction of the original [1]. Connector
wrappers are useful for specifying different strategies for implementing reliabil-
ity policies, such as retry and failover, in isolation without regard to the details of
a particular connector. The basic idea is to introduce (via parallel composition)
additional processes that synchronize with messages at the component–connector
interface to precisely model the behavior of the aforementioned wrapper mod-
ules. Because connector wrappers faithfully model the structure and behavior of
wrappers, an architect may specify a new connector as the composition of one
or more connector wrappers with some connector and then semi-automatically
generate a conforming implementation. Spitznagel’s system provides generation
tools that implement this capability.
Unfortunately, connector wrappers cannot yet utilize reconfiguration-based
mechanisms, which generally do not adhere to an algebraic model of composition.
The next section describes a model of composition that is algebraic in nature
and that supports fine-grained recomposition of the base system, allowing for
both reuse of abstractions and the avoidance of orphaned components.
2.3 AHEAD
AHEAD is an algebraic model of software composition in which complex, feature-
rich programs are synthesized from base programs by applying reusable refine-
ments [21]. Here, a base program is a collection of classes, and a refinement is a
collection of classes and/or class fragments, which can be applied to extend an
existing program with new functionality by using and/or refining the classes de-
fined by that program. AHEAD treats base programs and refinements as layers.
A base program is a stand-alone layer or constant (i.e., the layer contains no
class fragments) and a refinement is a parameterized layer (i.e., layer that must
“plug in” to another, subordinate layer).
The composition of AHEAD refinements and simple programs is depicted
visually using diagrams such as Figure 2(a). This figure illustrates the refine-
ment of a base program (const). Here, the inner-most boxes are classes or class
fragments. The solid rectangles demarcate refinements. The dotted lines from
one class to another indicate the refinement of a class with the code and data
in a class fragment; for example, b
const
is refined by b
f1
. Because const is the
bottom-most layer, it contains only classes and not class fragments.