Requirements engineering: In search of the dependent variables
Tony Gorschek
a,
*
, Alan M. Davis
b
a
Department of Systems and Software Engineering, School of Engineering, Blekinge Institute of Technology, P.O. Box 520, SE-37225 Ronneby, Sweden
b
College of Business, The University of Colorado at Colorado Springs, 1420 Austin Bluffs Parkway, P.O. Box 7150, Colorado Springs, CO 80933-7150, USA
Available online 13 October 2007
Abstract
When software development teams modify their requirements engineering process as an independent variable, they often examine the
implications of these process changes by assessing the quality of the products of the requirements engineering process, e.g., a software
requirements specification (SRS). Using the quality of the SRS as the dependent variable is flawed. As an alternative, this paper presents
a framework of dependent variables that serves as a full range for requirements engineering quality assessment. In this framework, the
quality of the SRS itself is just the first level. Other higher, and more significant levels, include whether the project was successful and
whether the resulting product was successful. And still higher levels include whether or not the company was successful and whether there
was a positive or negative impact on society as a whole.
2007 Elsevier B.V. All rights reserved.
Keywords: Requirements engineering; Project focus; Product focus; Dependent variables; Requirements process; Product management; Project manag-
ement; Process improvement; Organizational perspective
1. Introduction
Organizations throughout the world are attempting to
improve their ability to perform requir ements engineering.
All of them need to determine if they have been successful
in their efforts. The basis of this determination is the sub-
ject of this paper.
Assessment in most companies targets one of two bases:
• The requirements process itself, e.g., some measurement
of the time and/or resources consumed during the perfor-
mance of requirements engineering, and/or benchmark-
ing the process itself against a set of ‘‘best practices’’.
• The primary product of the requirements process, e.g.,
some measurement of the quality of the software require-
ments and/or the entire requirements specification.
However, as pointed out by Davis and Zowghi [1], per-
forming well on either or both of these ‘‘tests’’, does not
necessarily translat e into true success. Most requirements
engineering process improvement efforts are performed
with an aim of somehow improving the projects with little
or no concern for the resulting products. Thi s is supported
by the following:
• General software process improvement (SPI) frame-
works address the area of requirements engineering
in the context of development projects. Initial process
assessments are performed mostly on projects, and
improvements are assessed on projects. This is true
for model-based prescriptive SPI frameworks such
as CMMI [2–4] and SPICE [5,6], as well as for
inductive bottom-up frameworks like QIP [7]. Thus,
improvement is determined to be successful if the
project ends up costing less or completing earlier,
not if the resulting product is more successful in
the marketplace.
• Process improvement experiences reported from indus-
try (for example, [8–11]) have largely focused on assess-
ment and improvement efforts performed from project
perspectives. This should not be a surprise of course,
considering the previous bullet.
0950-5849/$ - see front matter 2007 Elsevier B.V. All rights reserved.
doi:10.1016/j.infsof.2007.10.003
*
Corresponding author.
E-mail addresses: tony.gorschek@bth.se (T. Gorschek), adavis@
uccs.edu (A.M. Davis).
www.elsevier.com/locate/infsof
Available online at www.sciencedirect.com
Information and Software Technology 50 (2008) 67–75
• Requirements engineering process models and best-
practice guides (for exampl e, [12–17]) center on activities
performed on development projects.
Our argument is that the effect of requirements engineer-
ing transcends pro ject instances. The success of activities
performed as part of the requirements engineering process
cannot be determined simply by exami ning a development
instance, but are instead a pa rt of something larger involv-
ing product management activities [18,19]. A requirements
engineering process change that, say, ha lves the effort to
construct a system, but lowers the resulting product’s
acceptance by its intended customers is a failure, not a suc-
cess. Although this applies equally to every software
process improvement effort, it is most crucial to apply this
to requirements engineering because requirements engi-
neering’s primary purpose is to increase the likelihood that
the as-built prod uct meets the needs of the intended cus-
tomers. Requirements engineer ing improvement efforts as
well as process assessment measurements have to take this
into account.
Researchers and engineers alike attempt to make
changes to their processes with the hope that some posi-
tive outcome will result. The processes that they change
are typically called independent variables, and the out-
comes that are to be observed are called dependent vari-
ables. As requirements engineers and as requirements
engineering researchers, we attempt to change the way
requirements engineering is conducted and expect that
positive outcomes will result. The question, however, is
what are these dependent variables? If the dependent
variables are selected deviously, then we can likely prove
that almost any change to the requirements engineering
process was positive. Dependent variables for require-
ments engineering can vary tremendously from the short-
est term (e.g., minimizing the time we spend doing
requirements engineering) to the longest term (e.g., hav-
ing the best possible result for the health of the planet
Earth). Deciding what is the right dependent variable
for any specific case is nontrivial, yet of major conse-
quence. This paper presents a framework of dependent
variables, each of which can serve as a requirements
engineering quality assessment basis.
This paper is structured as follows. In Section 2,we
present the framework for dependent variables suitable
for requirements process quality assessment. In Section
3, measures are proposed for each of the dependent vari-
ables. Suggestions for future research are proposed in
Section 4, and finally our conclusions are summarized
in Secti on 5.
2. Levels of quality
As requirements processes are changed, we may assess
their impact on many depende nt variables. These depen-
dent variables reside within at least five distinct levels, as
shown in Fig. 1. Starting at the center and working out:
• Requirements phase. By far the most commonly used
measure among companies trying to improve their
requirements process, this level includes dependent vari-
ables that relate to [20–22]:
Requirements cost and time. Such as total cost of the
requirements effort, average cost per requirement,
percentage of total development duration used for
requirements.
Requirements quality. That is, the quality of the
requirements specification itself, and so on.
• Project. This level is classically equated with ‘‘project
success’’ [23,24], i.e., whether the project was completed
on time, within budget, and did it meet the require-
ments. It includes dependent variables that relate to:
Project cost and time. Such as total cost of the project
and project duration.
Project estimates. Such as the degree of meeting the
budget, degree of meeting the schedule, degree to
which the as-built product meets the as-specified
requirements.
Degree of requirements change.
Notice that a requirements process change that is
assessed as successful using dependent variables
from only the requirements level cou ld still fail mis-
erably at the project level. For example, a project
could reduce the cost of the requirements phase
by skipping it altogether, and the result could be
so much project entropy that the project ends up
costing tw ice as much as planned.
• Product. Dependent variables within this level determine
the degree of product success, e.g., did the product suc-
ceed in fulfilling the needs of its intended customers/
users. It includes measures relating to:
Requirements selection. That is, the degree to which
the requirements selected for the product reduces
errors, increases efficiency, and so on, for its users
and or customers.
Fig. 1. Levels of requirements process change dependent variables.
68 T. Gorschek, A.M. Davis / Information and Software Technology 50 (2008) 67–75
Degree of Impact. That is, the degree to which the prod-
uct increases revenues, decreases costs, and so on, for
the company developing the product. If the product
is to be sold to a market, then it includes measures of
the units sold, revenues produced, margins attained,
average cost of sales, average order size, and so on.
Notice that a requirements process change that is
assessed as successful using dependent variables from
only project level could still fail miserably at the prod-
uct level. For example, a project could come in on bud-
get, on schedule, and meet its written requirements, but
because the docu mented requirements did not reflect
the true need s of the users (or because the user needs
changed during development without the project team
noticing), the resulting product could fail in the market-
place. Another example could be that unnecessary
requirements wer e included in the project effectively
increasing development time and cost without adding
product value.
• Company. Suppose a project is delivered late and/or
over budget, but the resulting product succeeds tremen-
dously in the marketplace. Using dependent variables
from the previous level, we could declare the product
successful. However, what happens if the lateness and/
or excessive cost is the cause of another project within
the same development company failing. Furthermore,
what if the adversely affected project was more critical
to the company’s success than the original project? In
such a case, we cannot declare the project to be success-
ful. The company level includes such measures as
Portfolio management. The degree to which products
compliment each other, and the degree to which they
each demonstrate optimal use of resources.
Strategic alignment. The degree to which the product
aligns with corporate strategies.
Degree of impact. Total corporate revenues, revenue
growth, meeting of revenue targets, profits, profit
growth, meeting of profit targets, market share,
return on investment, cash situation, and so on.
• Society. Although a company may provide great finan-
cial returns to its shareholders, a corporation also has
a greater responsibility to society at large. Thus, a pro-
ject that contrib utes to a company’s bottom line but pol-
lutes the environment or kills people must be considered
a failure. Products always give rise to
positive and negative externalities [25] that have to be
taken into consideration.
3. Rel ated research
Measures for all five levels have been proposed by oth-
ers. For example:
• Requirements phase. Impact on the requ irements phase
may be measured in terms of its outputs and in terms
of its activity. As the easiest of the five levels to measure,
no shortage exists of research aimed at determining
dependent variables related to the quality of software
requirements specifications. Boehm [26] was the first to
define measures of quality of an SRS. This was followed
by Alford and Burns [27], Zave and Yeh [28], the IEEE
[29], and many others, all of which were summarized by
Davis et al. [21]. More focused work has been reported
by Hunter and Nuseibeh [30] for consistency of require-
ments, Yue [22] for completeness of requirements, Zow-
ghi and Gervasi [31] for consistency, completeness, and
correctness, Agusa [20] for verifiability, and Kovitz [32]
for ambiguity. Meanwhile, in terms of the activity,
COCOMO [33] reports that on average the requirements
phase consumes 8% of the total project resources during
a period of time averaging 20% of the total project dura-
tion. In a study of NASA projects, Forsberg [23] pre-
sents a graph (see Fig. 2) that shows optimal project
success occurs when 7% to 15% of total project
resources are used during the requirement s phase.
• Project. Project performance is often measured in terms
of cost, schedule and how many of the documented
requirements were actually implemented. At least four
distinct requirements-related causes exist for a project
succeeding or failing at the project level:
(1) Poor estimation techniques. Because requirements
define what is to be done in a project, they serve
as a natural basis for estimation of pro ject costs
and schedules. The primary attempts to estimate
costs and schedules using requirements employ func-
tion points or feature points [34,35]. Obviously, if a
project completes its mission and significantly
exceeds the resource expectations, one cause could
be that the requirements were stated in a manner
that rendered them ineffective as an estimation basis.
(2) Mismatch between requirements specified and
requirements satisfied. The level of fulfillment of
the specified requirements are usually measured
through verification and validation (V&V) activities,
of which system testing is the most common [36]
.
Failure of V&V can be an indication of a poor
requirements engineering process, e.g., the require-
ments could be too poor to serve as a basis for cre-
Fig. 2. Forsberg study of NASA projects.
T. Gorschek, A.M. Davis / Information and Software Technology 50 (2008) 67–75 69
ating test cases. Inspections of requirements can be
good way to ascertain and in fact measure the inher-
ent testability of requirements [37–40].
(3) Evolving requirements. Some see changing require-
ments during a project as a problem, and use nega-
tive expressions such as requirements creep,asif
requirements change is somehow evil. The reality is
that if the real-world requirements are actually
changing (or even if our perceptions of them are
actually changing), we have just two choices: ignore
the changes and build the wrong product, or change
the requirements being addressed by the project (see
Table 1). In any case, the degree of requirements
change will significantly affect the success of the pro -
ject. The gross number of requirement changes made
to the project’s requirements and severity of these
changes are examples of dependent variables used
at the project level [41–44].
(4) Mismanagement. This cause is unrelated to the sub-
ject of this paper.
• Product. The success of an individual product (or a
product line/family) is directly linked to what require-
ments were selected for realization, i.e., the degree of
alignment between what the product does and what
the needs really are. This is not the same as assuring that
the requirements wer e realized in the right way, which is
measured by project success, as described in the previous
section. Two issues are directly responsible for the suc-
cess or failure of requirements engineering at the prod-
uct level :
(1) Requirements selection. Requirements in a market
driven environment come from several sources both
internal (e.g., developers, marketing, sales, support
personnel, bug reports, etc.) and external (e.g., users,
customers and competitors, often gathered via sur-
veys, interviews, focus groups, competitor analysis,
etc.) [45–47]. The volume of requirements that need
to be handled makes initial screening important in
order to decrease the risk of overloading in the eval-
uation and realization process [48]. Regnell et al.
present an analytical model based on product strat-
egies and business strategies aimed at enabling initial
handling of large volumes of requirements [49]. The
use of requirements abstraction and indirectly prod-
uct strategies has also been tried with promising
results through the use of a Requirements Abstrac-
tion Model [77].
After initial screening, requirements have to be prior-
itized, interdependencies need to be defined, and cost
estimations have to be established in order for the
selection process to take place. Several methods for
attaining requirement priority exist, including AHP
[50], the 100-point method [51], triage [52,53], attain-
ment [54], negotiation [55], and the planning-game
[56]. Prioritization can be performed by customers
and/or customer representatives, as well as by internal
experts from marketing and product management.
Prioritization of requ irements can be used as input
to the requirements selection before the final choices
for realization are made. Interdependencies between
requirements are also a determining factor in selec-
tion; studies show that only about 20% of require-
ments are relatively independent [57]. This may
influence the selection radically as high priority
requirements can be dependent on low priority
requirements, and vice versa. In addition to priority
and interdependencies, cost estimations have to be
made before final selection can be made, determining
what (customer) needs are to be satisfied by the prod-
uct/release, and which requirements are to be post-
poned or dismissed.
Being able to measure the success of the requirement
selection process is crucial. Regnell et al. present a
model that allows for principal reasoning about both
the quality and the current capacity of the requ ire-
ment selection process [49]. Karlsson et al. presents
the PARSEQ method which focuses on post-release
analysis of requirements selection quality, examining
the requirements actually selected for realization [58] .
In addition, the perceived customer value of a product
can be measured through the use of GAP analysis [59–
61]. GAP analysis measures positive and negative
‘‘gaps’’ between what the product offers and what
the customer perceives. Features and characteristics
of the product are identified and their fulfillment of
customer needs is mapped. A positive gap represents
when a product delivers more than is expected, a neg-
ative gap the opposite. One of the earliest descriptions
of the need to measure this ‘‘gap’’ was described in
[62]. Customer Value Analysis (CVA) is similar to
Table 1
The impact of requirements change
Real-world (e.g., market) requirements changing
Yes No
Project’s requirements changing
Yes Good for the Product, but Project may complete late or over
budget
This is the situation that should be called requirements
creep. it is bad for the Product and bad for the Project.
No Guaranteed that the Product Will Fail, but Project may be
completed on-time and on budget
Good for Product and Project
70 T. Gorschek, A.M. Davis / Information and Software Technology 50 (2008) 67–75
GAP analysis but also includes the perspective of
using competitor products in the analysis and the
evaluated product is graded with regards to the value
of a need in comparison to alternatives [63]. Both
GAP and CVA can be used to measure selection qual-
ity post-release, but the results can also be used
actively as input to the next round of requirements
selection for the product in question. The relationship
between requirements change, developm ent time, and
customer satisfaction was explored by [64].
(2) Product strategies. The development of product
strategies is a way of planning for a product, and
documenting in what way the product will support
business strategies [65]. Product strategies can be
seen as roadmaps for the long-term development
of a product and should reflect not only current
market knowledge and customer priorities but also
the long-term goals set for a certain product. The
requirements selection described above should be
done within the boundaries set by product strategies.
Ignoring product strategies (and only looking at cur-
rent priorities) may mean that a product is successful
short-term, but at the expense of the long-term goals
[40,47]. For example, security requirements may be
deemed less important at present but the long-term
plans for the product is to eventually break into
new market segments where security is deemed cru-
cial. One of the fundamental aims of a product strat-
egy is to explicitly plot the goals and the limits of a
product – focusing all efforts and aligning them in
one deliberate direction [47,66]. On the other hand,
sometimes constraining yourself too much to a busi-
ness strategy could cause a company to miss new,
out-of-the-box business opportunities that can lever-
age business technology strengths.
Requirements selection within the boundaries of
product strategy does not guarantee success, as the
strategies followed can be flawed. However having
up-to-date product strategies, which reflect all vital
knowledge – from both technology and marketing
perspectives – will most certainly increase the chance
of developing a successful product [40,47,67] .
Internal Value Analysis (IVA) is a technique to mea-
sure whether or not a prod uct is in line with the
product strategies (and the company strategies), tak-
ing limited resou rces and other products into
account [40,47]. Using IVA, factors like resources
available (time, money, risk, and knowledge) can
be taken into consideration, complementing data
from both GAP/CVA analysis, priori tizations of
requirements, dependency mapping, and cost esti-
mates. Combined, this can form decision supp ort
material for requirements selection taking product
strategies into consider ation.
(3) Did the product make money (revenue and/or
profit)? Selecting the ‘‘right’’ requir ements from a
customer perspective and thus satisfying customer
expectations should in theory have a strong correla-
tion with market success. However, many factors,
including the presence of unexpressed, tacit knowl-
edge, can adversely effect the correlation (see [68]
for discussions of elicitation techniques suitable for
discovering tacit requirements). Whether or not the
product is really successful requires a wider perspec-
tive, assessing the internal value of realizing require-
ments, risks taken, resources used, and so on, into
consideration, as well as working within the long-
term perspective of product strategies. On the prod-
uct level, this may suffice, especially since many of
the larger issues are filtered down through product
strategies, but regarding taking the ‘‘best alternative
investment’’ into consideration the company scope
has to be taken into consideration.
• Company. The company level of dependent variables in
requirements engineering is closely related to the product
scope described above. However, at the company level, an
overview of the entire product portfolio consisting of
products already deployed, those under development,
and those just being planned, is essential. The product
strategies mentioned in the previous section are created
and compared to the overall strategy of the company
(a.k.a. business strategies) as such [47,66].
Product line literature uses the term ‘‘Product Portfolio
Scoping’’ with regards to the activity of deciding the
overall focus of strategies on a company level as it per-
tains to product lines – helping to leverage existing
investments where it pays [69]. This activity is based
on traditional portfolio management [46], but has been
adapted to the software product line domain where
reuse is paramount and core assets are established
[70]. The selection of requirements for realization has
to be viewed from the company level when it regards
product line organizations, as the selection of one
requirement can potentially impact several products
that share core asset s. This is one of the reasons why
development in product line organizations often take
the company level into consideration with regards to
requirements engineering.
The main idea at the company level is to maximize the
total economic benefits of several products, as opposed
to the product perspect ive of maximizing the benefits of
a particular product. Many of the same tools, e.g.,
GAP, CVA, and IVA, can be used. The use of bubble
diagrams like the Boston Consulting Group Matrix is
also fairly common [71]. They are used to visualize the
placement of individual products in relation to market
maturity, as well as in relation to each other. The idea
is to get a distribution of products over time relative
to market maturity, risk, and reward in an attempt to
minimize risk and maximize reward in the long run
[72]. These tools help the company choose between
products (and thus indirectly requirements) in a manner
that maintains a balance between high risk – high
reward and low risk – low reward development.
T. Gorschek, A.M. Davis / Information and Software Technology 50 (2008) 67–75 71
