Assessing Business Processing Modeling Languages
Using a Generic Quality Framework
Anna Gunhild Nysetvold and John Krogstie
Norwegian University of Science and Technology,
Institute of Computer and Information Sciences
and SINTEF Telecom and Informatics, Norway.
krogstie@idi.ntnu.no
Abstract. We describe in this paper an insurance company that has recently
wanted to standardize on business process modeling language. To perform the
evaluation, a generic framework for assessing the quality of models and
modeling languages was specialized to the needs of the company. Three
different modeling languages were evaluated according to the specialized
criteria.
The work illustrates the practical utility of the overall framework, where
language quality features are looked upon as means to enable the creation of
models of high quality. It also illustrates the need for specializing this kind of
general framework based on the requirements of the specific organization.
1 Introduction
There exists a large number of business process modeling languages. Deciding which
modeling language to use for a specific task is often done in an ad-hoc fashion by
different organizations. In this paper we present the work within an insurance
company, which have a perceived need for using process modeling to support the
integration of their business systems across different functions of the organization.
We have earlier developed a general framework for assessment of quality of
models, where one type of means to support different quality goals, is criteria for the
language to be used for modeling, also termed language quality (Krogstie, 2001). This
paper presents an example of using and specializing the quality framework for the
evaluation and selection of a modeling language for enterprise process modeling for
the insurance company. The need for such specialization is grounded on work on
task-technology fit (Goodhue et al, 1995). A similar use of the framework for
comparing process modeling languages in an oil company has been reported in
(Krogstie and Arnesen, 2004). Although similar, we will see that due do different
overall goals of process modeling, the criteria derived from the quality framework
ended up different in the work reported in this paper. This is also due to that we in this
investigation have added aspects of organizational appropriateness of the approach.
1.1 The structure of this paper
The next section describes the quality framework, with a focus on language quality.
Section three describes the case in more detail, and is followed by the results of the
evaluation. The conclusion highlights some of our experiences from using and
specializing the quality framework for evaluating modeling languages for enterprise
modeling.
2 FRAMEWORK FOR QUALITY OF MODELS
The model quality framework (Krogstie, Lindland and Sindre 1995; Krogstie and
Sølvberg 2003; Krogstie 2001) is used as a starting point for the discussion on
language quality. We have taken a set-theoretic approach to the discussion of model
quality at different semiotic levels, which has been defined as the correspondence
between statements belonging to the following sets:
• G, the (normally organizationally defined) goals of the modeling task.
• L, the language extension, i.e., the set of all statements that are possible to make
according to the graphemes, vocabulary, and syntax of the modeling languages
used.
• D, the domain, i.e., the set of all statements which can be stated about the situation
at hand.
• M, the externalized model.
• K
s
, the relevant explicit knowledge of the set of stakeholders being involved in
modeling. A subset of the audience is those actively involved in developing
models, and their knowledge is indicated by K
M
.
• I, the social actor interpretation, i.e., the set of all statements which the audience at
a given time thinks that an externalized model consists of.
• T, the technical actor interpretation, i.e., the statements in the model as 'interpreted'
by different modeling tools.
The model quality types are defined as relations between these sets.
• Physical quality: The basic quality goals on the physical level are externalization,
that the knowledge K of the domain D of some social actor has been externalized
by the use of a modeling language, and internalizeability, that the externalized
model M is persistent and available to the audience.
• Empirical quality deals with HCI-ergonomics for documentation and modeling-
tools.
• Syntactic quality is the correspondence between the model M and the language
extension L of the language in which the model is written.
• Semantic quality is the correspondence between the model M and the domain D.
The framework contains two semantic goals; Validity which means that all
statements made in the model are correct relative to the domain and completeness
which means that the model contains all the statements which is found in the
domain.
• Perceived semantic quality is the correspondence between the audience
interpretation I of a model M and his or hers current knowledge K of D.
• Pragmatic quality is the correspondence between the model M and the audience's
interpretation of it (I).
• The goal defined for social quality is agreement among audience members’
interpretations I.
• The organizational quality of the model relates to that all statements in the model
directly or indirectly contributes to fulfilling the goals of modeling (organizational
goal validity), and that all the goals of modeling are being addressed through the
model (organizational goal completeness).
Language quality relates the modeling languages used to the other sets. It is
distinguished between two types of criteria:
Fig. 1. Language quality related to the quality framework
As illustrated in Figure 1, six quality areas for language quality are identified:
Domain appropriateness
Ideally, the language must be powerful enough to express anything in the domain, i.e.
not having construct deficit (Wand & Weber, 1993). On the other hand, you should
not be able to express things that are not in the domain; i.e. what is termed construct
excess (Wand & Weber, 1993). The only requirement to the notation is that it is able
to represent all concepts in the language. Domain appropriateness is primarily a mean
to achieve physical quality, and through this, to be able to achieve semantic quality.
Knowledge externalizability
appropriateness
Technical actor
interpretation
appropriateness
Comprehensibility
appropriateness
Model
externalization
M
Social
actor
interpretation
I
Technical
actor
interpretation
T
Modeling
domain
D
Language
extension
L
Modeller
explicit
knowledge
Km
Domain appropriateness
Modelling Goal
G
Social actor
explicit
knowledge
Ks
Participant language
knowledge appropriateness
Organisational
appropriateness
Participant language knowledge appropriateness
This area relates the knowledge of the stakeholder to the language. The conceptual
basis should correspond as much as possible to the way individuals perceive reality.
This will differ from person to person according to their previous experience, and thus
will initially be directly dependent on the stakeholder or modeler. On the other hand
the knowledge of the stakeholder is not static, i.e. it is possible to educate persons in
the use of a specific language. In that case, one should base the language on
experiences with languages for the relevant types of modeling, and languages that
have been used successfully earlier in similar tasks. Participant language knowledge
appropriateness is primarily a mean to help achieve physical and pragmatic quality.
Knowledge externalizability appropriateness
This area relates the language to the knowledge of the modeler. The goal is that there
are no statements in the explicit knowledge of the modeler that cannot be expressed in
the language. Knowledge externalizability appropriateness is primarily a mean to
achieve physical quality.
Comprehensibility appropriateness
This area relates the language to the social actor interpretation. For the concepts of the
language we have:
• The concepts of the language should be easily distinguishable from each other.
(Vs. construct redundancy (Wand & Weber, 1993)).
• The number of concepts should be reasonable. If the number has to be large, the
concepts should be organized hierarchically and/or in sub-languages of reasonable
size linked to specific modeling tasks. or viewpoints.
• The use of concepts should be uniform throughout the whole set of statements that
can be expressed within the language.
• The language must be flexible in the level of detail.
As for the notation, the following aspects are important:
• Symbol discrimination should be easy.
• It should be easy to distinguish which of the symbols in a model any graphical
mark in the model is part of (What Goodman (1976) terms syntactic disjointness).
• The use of symbols should be uniform i.e. a symbol should not represent one
phenomenon in one context and another one in a different context. Neither should
different symbols be used for the same phenomenon in different contexts.
• One should strive for symbolic simplicity.
• One should use a uniform writing system: All symbols (at least within each sub-
language) should be within the same writing system (e.g. non-phonological such as
pictographic, ideographic, logographic, or phonological such as alphabetic).
• The use of emphasis in the notation should be in accordance with the relative
importance of the statements in the given model
Comprehensibility appropriateness is primarily a mean to achieve empirical and
through that, pragmatic quality.
Technical actor interpretation appropriateness
This area relates the language to the technical actor interpretation. For the technical
actors, it is especially important that the language lend itself to automatic reasoning.
This requires formality (i.e. both formal syntax and semantics. The formal semantics
can be operational, logical, or both), but formality is not sufficient, since the
reasoning must also be efficient to be of practical use. This is covered by what we
term analyzability (to exploit the mathematical semantics) and executability (to
exploit the operational semantics). Different aspects of technical actor interpretation
appropriateness are a mean to achieve syntactic, semantic and pragmatic quality
(through formal syntax, mathematical semantics, and operational semantics
respectively).
Organizational appropriateness
relates the language to standards and other organizational needs within the
organizational context of modeling. These are a mean to support organizational
quality.
A number of sub-areas are identified for each of the six areas of language quality, and
in (Østbø, 2000) approximately 70 possible criteria were identified. We will return to
how this extensive list has been narrowed down and specialized for the task at hand.
3 Description of the case and the evaluation approach
The insurance company in our case has a large number of life insurance and pension
insurance customers. The insurances are managed by a large number of systems of
different age and using different technology. The business processes of the company
go across systems, products and business areas, and the work pattern is dependant on
the system being used. The company has modernized its IT-architecture. The IT-
architecture is service-oriented, based on a common communication bus and an EAI-
system to integrate the different system. To be able to support complete business
processes in this architecture, there is a need for tools for development, evolution and
enactment of business processes.
Goals for business process modeling in the insurance company
Before discussing the needs of the case organization specifically, we outline the main
uses of enterprise process modeling. Five main categories for enterprise modeling
can be distinguished:
1. Human-sense making and communication: To make sense of aspects of an
enterprise and communicate this with other people.
2. Computer-assisted analysis: To gain knowledge about the enterprise through
simulation or deduction.
3. Business process management e.g. in connection to following up ISO-certification.
