Timed Extended Invariants for the Passive Testing of Web Services
∗
Gerardo Morales, Stephane Maag, Ana Cavalli
Télécom Sud-Paris, CNRS/IMR5157
9, rue Charles Fourier 91000, Evry, France
{gerardo.morales, stephane.maag, ana.cavalli}@it-sudparis.eu
Wissam Mallouli, Edgardo Montes de Oca, Bachar Wehbi
Montimage EURL
39, rue Bobillot 75013, Paris, France
{wissam.mallouli, edgardo.montesdeoca,bachar.wehbi}@montimage.com
Abstract
The service-oriented approach is becoming more and
more popular to integrate highly heterogeneous systems.
Web services are the natural evolution of conventional mid-
dleware technologies to support Web-based and enterprise-
level integration. Formal testing of such Web-based tech-
nology is a key point to guarantee its reliability. In this
paper, we choose a non-intrusive approach based on mon-
itoring to propose a conformance passive testing method-
ology to check that a composed Web service respects its
functional requirements. This methodology is based on a
set of formal invariants representing properties to be tested
including data and time constraints. Passive testing of an
industrial system (that uses a composition of Web services)
is briefly presented to demonstrate the effectiveness of the
proposed approach.
1 Introduction
A Web Service is defined by the World Wide Web Con-
sortium W3C as “a software system designed to support in-
teroperable machine-to-machineinteraction overa network.
It has an interface described in a machine-processable for-
mat (specifically WSDL [8]). Other systems interact with
the Web service in a manner prescribed by its descrip-
tion using SOAP-messages, typically conveyedusing HTTP
with an XML serialization in conjunction with other Web-
related standards."
∗
The research leading to these results has received funding in part from
the French ANR Webmov Project (http://webmov.lri.fr/) and the European
Community’s Seventh Framework Program (FP 7/2007-203) under grant
agreement n
0
215995 (http://www.shields-project.eu/)
The composition of Web services consists of combining
existing Web services to define higher-level functionalities.
The Business Process Execution Language (BPEL) [4] is
emerging as the standard composition language for speci-
fying business process behavior based on Web services. A
BPEL process implements one Web service by specifying
its interactions with other Web services (called partners).
As the Web service implementations grow in size and
complexity, the necessity for testing their reliability and
level of security is becoming more and more crucial. The
Web Services technology, which is based on the SOA (Ser-
vice Oriented Architecture), is hard to test because it relies
on distributed systems that complicate the runtime behavior.
In this distributed context, formal active testing [9] can
be rather difficult to achieve since it requires to control re-
mote deployed Web systems. Indeed, active testing usually
relies on the comparison between the behavior of an imple-
mentation and its formal specification by checking whether
they are equivalent. Test sequences are commonly automat-
ically or semi-automatically generated from formal models
that represent test criteria, hypothesis and test objectives.
These sequences (with an executable format) are performed
by designing a testing architecture and establishing Points
of Control and Observation (PCO, execution interfaces) that
may disturb the natural behavior of already deployed Web
services.
To avoid this problem, there is an interest in applying
the passive testing approach for the conformance checking
of Web services. Passive testing consists in observing the
exchange of messages (inputs and outputs) of a Web service
implementation under test (IUT). The term passive means
that the tests do not disturb the natural operation of the IUT.
Instead, the exchanged messages will be recorded in a trace
that will be inspected later on against the properties derived
from the Web service by system experts.
In the work we present here, we extend the concept of
invariants already defined in [1] to take data and time con-
straints into consideration. Then we define a methodology
with a new algorithm for formal passive testing based on
these extended invariants. The main contributions of this
paper are the following:
• The definition of timed extended invariants that allow
to formally specify functional properties that the Web
service has to respect. These properties are related to
the exchanged packets between Web services and may
deal with the data portion of these packets and/or with
time constraints.
• The proposition of conformance testing approach for
Web services using passive testing approach. This ap-
proach is particularly well adapted when it is difficult
to control remote partners and permits to avoid dis-
turbing the natural running of already deployed Web
services.
• Evaluation of the proposedmethodologyand tool TIPS
(Testing Invariants for Protocols and Services) on the
Travel Reservation System composed Web service.
This tool performs a dynamic packet inspection to ana-
lyze the execution traces to check the formal extended
invariants.
The rest of the paper is organized as follows. Section
3 presents the formal concept of timed extended invariants.
In section 4, we present our passive testing methodology as
well as its related tool and algorithm. Section 5 presents our
experiment on a Travel Reservation System that uses a Web
service composition orchestrated by a BPEL engine. We
apply on it our passive testing tool and analyze the obtained
results. Section 6 concludes this paper and presents some
future perspectives.
2 Related work
Many recent research work are devoted to model and
test machine-to-machine communications through Web ser-
vices. Besides, big efforts are currently dedicated to pro-
vide standards for Web services interoperability testing [7],
while very few research tackle their passive conformance
testing applying formal methods. We mention some of them
in the following.
[2] proposes to extend the architecture used to deploy
Web services (SOA) with an "observer element". The au-
thors use the model FSM+ (FSM that can take into account
some timed constraints) that are applied as the reference be-
havior to set verdicts over the tests. While timing aspects
are studied in this work, data portions and SOA architecture
used to deploy the Web services are not processed.
[5] presents the TGSE tool, based on the methodology
presented in [12]. TGSE allows the runtime coverage of
transitions based on guard conditions (by examining vari-
able values). In this approach, the authors model an execu-
tion trace as a test purpose (i.e. also referred as a TEFSM)
by giving real values to each input message. After model-
ing the composed Web service specification and the trace
as TEFSMs and declaring their synchronization rules, they
use TGSE to verify this system. If the TGSE output is a se-
quence, it means that this trace is a valid trace of the interac-
tions among the systems. However, this method requires a
formal specification of the Web service composition which
is often unavailable (or hard to provide) in such Web sys-
tems.
The methodologies and tools that are presented in our
work to perform passive testing are based on a formal in-
variant approach [1] applied to the SOA architecture of Web
service deployment.
3 Timed extended invariants
3.1 Preliminaries
Invariants are properties that the implementation under
test is expected to satisfy. They are used to express con-
straints over exchanged messages between the system enti-
ties (in this case, distributed Web services). Basically, they
allow expressing that the occurrence of an event must be
necessarily preceded or followed by a sequence of events.
In [1], a first introduction to the invariants was provided.
These invariants are described according to a finite state ma-
chine (FSM) based specification of the system under test.
This model is not well adapted to express complex func-
tional behavior of Web services that may involve data ex-
change with time constraints. To solve this problem, we
propose to extend the formalization of invariants by adding
these two aspects (data and time), and rely thus on a specifi-
cation of the IUT described using a Timed Extended Finite
State Machine (TEFSM) model ( [3]).
Notice that only the system inputs and outputs can be ob-
servable in the collected system traces. These actions rep-
resent the events that we take into account in the timed ex-
tended invariants. Based on the definition of a TEFSM, we
enhance the formalization of the messages collected within
a trace and the events described within the invariants.
In the following definitions a captured packet is a net-
work packet exchangedbetween the BPEL engine (the com-
posed Web service orchestrator) and its Web partners. This
packet is captured by a sniffer that adds some extra data (for
example the timestamp of when it was captured), i.e. cap-
tured packet = network packet + extra data.
Definition 1 (Collected trace) A collected trace is a set of
ordered captured packets.
2
• A trace T =
S
n
i=1
p
i
where n is the number of captured
packets, p
1
is the first captured packet in the trace and
p
n
the last one.
• Each packet p
i
has a rank r
i
that corresponds to its
position in the trace T .
• ∀p
i
∈ T , p
i
=
S
m
i
j=1
f
i,j
where f
i,j
is a field of the
packet p
i
and m
i
is the number of fields of the packet
p
i
. Each field f
i,j
of the packet p
i
has a value v
i,j
.
• ∀p
i
∈ T , ∃f
i,j
∈ p
i
/ f
i,j
= t
i
where t
i
is the timestamp
when p
i
was captured.
• ∀r
i
, r
j
where r
i
is rank of p
i
and r
j
is rank of p
j
, if r
i
> r
j
then t
i
> t
j
Definition 2 (Value function φ) Let T be a collected trace
of n packets, F the set of fields of all the packets p
i
of the
trace T , V the domain of values and P the set of packets.
V = R ∪ S ∪ NULL where S is a finite set of strings.
We define the function: φ: P × F → V as the function
allowing to provide the value of a field in a specific packet
of the trace T :
• φ(p
i
, f
m,n
) = v
i,n
if f
m,n
∈ p
i
and
• φ(p
i
, f
m,n
) = NULL if f
m,n
/∈ p
i
3.2 Definition o f timed extended invari-
ants
An invariant is an If-Then property. It allows expressing
the desired behavior of a communicating system and de-
scribes the correct order of messages collected in the trace
with potential time constraints. If specific conditions on the
exchanged messages hold, then the occurrence of a set of
events must happen. An event is a set of conditions on some
field values of exchanged packets.
Definition 3 (Conditions) Conditions are predicates on
packets’ fields values. Let p
i
and p
i
′ be two captured pack-
ets, V be the domain of values, f
i,j
be a field of the packet
p
i
, f
i′,j′
be a field of p
i
′ and x ∈ V . Two types of condi-
tions can be defined: c
s
(simple condition) and c
c
(complex
condition)
c
s
::= φ(p
i
, f
i,j
) op x
We say that the packet p
i
satisfies c
s
iff v
i,j
op x is true.
c
c
::= φ(p
i
, f
i,j
) op φ(p
i′
, f
i′,j′
)
We say that packet p
i
satisfies c
c
iff
v
i,j
op v
i′,j′
is true.
op is an element of O
T
= O
R
∪O
S
where O
R
={≤ , ≥ , =,
6=, ∈} and O
S
={contain, contain not}. The operator must
respect these rules:
• If v
i,j
∈ R then op ∈ O
R
.
• If v
i,j
∈ S then op ∈ O
T
and:
– the operators that are in O
S
are used when a
field’s value has a string type.
Notice that v
i,j
, v
i′,j′
and x must be part of the same do-
main.
Definition 4 (Basic event) A timed extended event e
j
is a
set of conditions on relevant fields of captured packets. e
j
=
S
m
j
k=1
c
j,k
, m
j
being the number of conditions.
Definition 5 (Event satisfaction) Let p
i
be a packet and e
j
an event with m
j
conditions and c
j,k
the k
th
condition of e
j
A packet p
i
satisfies an event e
j
iff
∀ k ∈ [1, m
j
], c
j,k
is true
Definition 6 (Abstention of having an event)
If e is an event, then ¬e is also an event. ¬e is satisfied if
no packet that satisfies the event e occurs in the collected
trace.
Definition 7 (Successive Events) Let n ∈ N ∪ {−1}, t ∈
R
+∗
∪ {−1} and e
1
and e
2
be two basic events. (e
1
; e
2
)
n,t
is a complex event denoted also by Follows((e
1
, e
2
),n,t). It
is composed of two basic events.
[p
1
, p
2
] satisfies (e
1
; e
2
)
n,t
⇔
• p
1
satisfies e
1
and
• p
2
satisfies e
2
and
• time(p
1
) < time(p
2
) < time(p
1
)+t if (t 6= −1) and
• rank(p
1
) < rank(p
2
) < rank(p
1
)+n if (n 6= −1).
In other words, [p
1
, p
2
] satisfies (e
1
; e
2
)
n,t
iff p
2
follows
p
1
and they are separated by at most n packets and t units
of time.
Definition 8 (complex events: AND) Let n ∈ N ∪ {−1},
t ∈ R
+∗
∪ {−1} and e
1
and e
2
two basic events. (e
1
∧
e
2
)
n,t
is a complex event denoted also AND((e
1
,e
2
),n,t). It
is composed of two basic events.
[p
1
, p
2
] satisfies (e
1
∧ e
2
)
n,t
⇔
[p
1
, p
2
] satisfies (e
1
; e
2
)
n,t
or (p
1
, p
2
) satisfies (e
2
; e
1
)
n,t
Intuitively, p
1
and p
2
satisfy (e
1
∧e
2
)
n,t
iff p
2
and p
1
are
separated by at most n packets and t units of time.
Notice that in this definition the value of n or t can be
-1, this is to “disable” the constraint on this variable (n or
t). This is highlighted later in the definition 10
3
Definition 9 (complex events: OR) Let n ∈ N ∪ {−1}, t ∈
R
+∗
∪{−1} and e
1
and e
2
two basic events. (e
1
∨e
2
)
n,t
is a
complex event denoted also OR((e
1
,e
2
),n,t), It is composed
of two basic events.
p
1
satisfies (e
1
∨ e
2
)
n,t
⇔ p
1
satisfies e
1
or p
1
satisfies e
2
Definition 10 (Timed Extended Invariant) Let
′′
W hen
′′
∈
{BEF ORE, AF T ER}, n ∈ N ∪ { −1}, t ∈ R
+∗
∪ {−1}
and e
1
and e
2
two events (basic or not). A timed extended
invariant is an IF-THEN expression that allows expressing
a property regarding the messages exchanged in a captured
trace P = {p
1
, ..., p
m′
}. It has the following syntax:
e
1
W hen,n,t
−−−−−−→ e
2
This property expresses that if the event e
1
is satisfied (by
one or several packets p
i
, i ∈ {1, ..., m}), then event
e
2
must be satisfied (by another set of packets p
j
, j ∈
{1, ..., m}) before or after (depending on the When value)
at most n packets (if n 6= −1) and t units of time (if t 6= −1).
It is important to highlight that in the invariant it is pos-
sible to disable n and/or t by setting their values to -1.
If (W hen = AF T ER), we say that the invariant is a
simple invariant. If (When = BEF ORE), we have an
obligation invariant.
Example:
According to this last definition, invariants can express
complex properties including data and time constraints. For
instance in a HttpPost restful web service, we can express
that if an OK message (the code 200) is received, then a
POST request must have been sent before. This is described
by the following obligation invariant:
e
1
= {φ(p
1
, htt p.resp onse.code) = 200 }
BEF ORE,−1,6
−−−−−−−−−−→
e
2
= {φ(p
2
), http.request.method) = P O ST,
φ(p
2
, tcp.srcport) = φ(p
1
, tcp.dtsport),
φ(p
2
, tcp.dtsport) = φ(p
1
, srcport)}
Notice that in this invariant, we do not take into account
the number of packets that are between the OK and the
POST messages, on the other hand, we specify that the de-
lay of time between the messages must be at most 6 seconds
which is the timeout value for this service.
4 Conformance passive testing approach for
Web services
4.1 Methodolog y overview
Conformance testing in the passive testing approach
aims to test the correctness of a Web service implementation
through a set of properties (invariants) and monitored traces
(extracted from the running Web service through probes
called also Points of Observation (PO)). The conformance
passive testing procedure follows these four steps:
• Step 1: Properties formulation. The relevant Web ser-
vice properties to be tested are provided by the stan-
dards or by the Web service experts. These properties
express the main requirements related to the communi-
cation between the Web partners in the context of Web
services.
• Step 2: Properties as invariants. Properties have to be
formulated by means of invariants that express local
properties; i.e. related to a local entity. Moreover,
the properties could be formally verified on the for-
mal system specification (as a timed automata) if it is
available, ensuring that they are correct vis-à-vis the
requirements.
• Step 3: Extraction of execution traces. In order to ob-
tain such traces, different PO are to be set up by means
of a network sniffer. The captured traces can be stored
in an XML format.
• Step 4: Test of the invariants on the traces. The
traces are processed in order to obtain information con-
cerning particular events as well as relevant data (e.g.
source and destination address, origin of data to ini-
tialize a variable, etc.). During this process, the test of
the expected properties is performed and a verdict is
emitted (Pass, Fail or Inconclusive). An inconclusive
verdict may be obtained if the trace does not contain
enough information to allow a Pass or Fail verdict.
4.2 Passive testing tool
Based on the methodology of our work, we developed
the TIPS tool [6] that performs automated analysis of the
captured traces to determine if the given timed extended in-
variants are satisfied or not. The TIPS tool aims to passively
test a deployed communicating system under test to verify if
it respects a set of properties. In the case of TIPS, invariants
describe the correct order of exchanged messages among
system entities with conditions on communicated data and
time. Passive testing consists of observing input and output
events of the system implementation in run-time and detect-
ing potential misbehavior or errors.
Figure 1 illustrates the modules of the TIPS tool. It has
three different inputs:
1) SUT Information on the Web service under test that
is being observed. This information represents data of in-
terest (for example protocol packets field names) that are
relevant to the automated analysis of the captured traces. 2)
The invariants defined in XML format. The non-respect of
an invariant may imply an error in the Web services.3) The
4
Figure 1. TIPS architecture for Web service
checking
collected traces represented in XML format (captured using
Wireshark
1
for instance).
In order to use the TIPS tool, the first step consists in
defining the invariants and the data of interest. This can be
done by an expert of the Web service under test. The invari-
ants are then verified with respect to their XML schema.
The next step consists in capturing the communication
traces using a sniffer (we used Wireshark) and analyzing
them using the TIPS tool.
In the case of Web services, what we analyze are the
SOAP messages. A SOAP request is an XML-based Re-
mote Procedure Call (RPC) sent using the HTTP transport
protocol. The payload of the SOAP message is an XML
document that specifies the call being made and the param-
eters being passed.
Within the body of a SOAP message, XML namespaces
are used to qualify element and attribute names within the
parts of the document. Element names can be global (ref-
erenced throughout the SOAP message) or local. The local
element names are provided by namespaces and are used in
the particular part of the message where they are located.
Thus, SOAP messages use namespaces to qualify element
names in the separate parts of a message. Namespaces also
identify the SOAP envelope version and encoding style.
4.3 Invariants storage
Within an invariant, the <if> tag identifies the triggering
events. We call this part of the invariant the Trigger Context.
The events that need to be verified on the trace are found in
the <then> tag. We call this part of the invariant the Verdict
Clause.
1
http://www.wireshark.org/
The invariants are written in XML format and are repre-
sented in an If-Then structure. Basically, if in the trace we
find the events of the Trigger Context, then we need to ver-
ify that the events of the Verdict Clause are part of the trace
as well.
The invariant in Figure 2 expresses that if a system re-
ceives a reply message, then this means that it wa sent a re-
quest message 10 seconds before. In this invariant we have
the Trigger Context from lines 5 to 15, this means that if in
the collected trace, we find a message respecting the event
described in the <if> context (lines 6 to 14), the process of
verifying the invariant over the trace is triggered. The Ver-
dict Clause (lines 18 to 33) contains the events that must be
satisfied in the tracee once the verification process is trig-
gered.
1. ...
2.
3. <invariant name="INVARIANT 1">
4. <!- Trigger Context start here ->
5. <if>
6. <event reference= “true” id= “001”>
7. <condition>
8. <variable>Type</variable>
9. <operation>=</operation>
10. <value>REPLY</value>
11. </condition>
12.
13. ...
14. </event>
15. </if>
16. <!- Trigger Context end here ->
17. <!- Verdict Clause start here ->
18. <then type="BEFORE" max_skip="-1"
max_time="10">
19. <event id= “032”>
20. <condition>
21. <variable>Type</variable>
22. <operation>=</operation>
23. <variable>REQ</value>
24. </condition>
25. <condition>
26. <variable>Source</variable>
27. <operation>=</operation>
28. <variable type=="e001">Destination</value>
29. </condition>
30.
31. ...
32. </event>
33. </then>
34. <!- Verdict Clause end here ->
35. </invariant>
36.
37. ...
Figure 2. XML format for defining an invariant
4.4 Passive testing algorithm
This section presents the algorithm that allows the de-
duction of a verdict by analyzing the system trace with re-
spect to a set of predefined obligation invariants. The ob-
tained verdict for an invariant can be either: Pass, Fail or
Inconclusive meaning respectively that all events were sat-
isfied, that at least one event was not satisfied or that it is
5
