scispace - formally typeset
Search or ask a question
Proceedings ArticleDOI

Actionable Software Metrics: An Industrial Perspective

TL;DR: An exploration of industry's views onactionable metrics help characterize actionable metrics in practical terms and facilitate their definition and development right from the start of a software metrics program.
Abstract: Background: Practitioners would like to take action based on software metrics, as long as they find them reliable. Existing literature explores how metrics can be made reliable, but remains unclear if there are other conditions necessary for a metric to be actionable. Context & Method: In the context of a European H2020 Project, we conducted a multiple case study to study metrics' use in four companies, and identified instances where these metrics influenced actions. We used an online questionnaire to enquire about the project participants' views on actionable metrics. Next, we invited one participant from each company to elaborate on the identified metrics' use for taking actions and the questionnaire responses (N=17). Result: We learned that a metric that is practical, contextual, and exhibits high data quality characteristics is actionable. Even a non-actionable metric can be useful, but an actionable metric mostly requires interpretation. However, the more these metrics are simple and reflect the software development context accurately, the less interpretation required to infer actionable information from the metric. Company size and project characteristics can also influence the type of metric that can be actionable. Conclusion: This exploration of industry's views on actionable metrics help characterize actionable metrics in practical terms. This awareness of what characteristics constitute an actionable metric can facilitate their definition and development right from the start of a software metrics program.

Summary (3 min read)

1 Introduction

  • One of the benchmarks of a software metrics program’s effectiveness is how actionable it is [22], since software practitioners are not interested only in mining data for insights but also in using those insights to guide their actions [20, 24].
  • Even a non-actionable metric can be useful, but an actionable metric mostly requires interpretation.
  • This exploration of industry’s views on actionable metrics help characterize actionable metrics in practical terms.
  • The goal of the EU H2020 Project, Q-Rapids, is to develop an agile-based, data-driven, and quality-aware rapid software development process [3].

3.1 Research context

  • The following table characterizes the context of the four companies, to which the participants of this study belong: CC1 used the Solution in the context of a modeling tool for model-driven development, which is part of a mature product line, 2 Little’s Law: Avg. Cycle Time = Avg. Work in Progress / Avg. Throughput.
  • With multiple releases already in the market.
  • A comparable Solution use at CC3 was hindered due to several unforeseeable technical and managerial circumstances.
  • Following a positive and a formative experience from their pilot use case, CC4 continued to use the Solution in the next UC, albeit with customizations that are exclusive to this case company only.
  • The experience accumulated from these two UCs inform CC4’s views on actionable metrics.

3.2 Data collection

  • The case companies shared metrics data with the Project researchers on a monthly basis, along with a short report documenting their use of the Solution.
  • These 15 statements and their sources of reference are shown in the table below: With Q1-Q5 statements, the authors enquire about the general characteristics of an actionable metric.
  • Both these sections were followed by an option for additional comments.
  • The five data quality characteristics used in the questionnaire refer to the inherent data quality, as prescribed in ISO/IEC 25012:2008 [6].
  • The aim was to elicit responses from the users that were involved in the metrics-driven improvement actions at their companies.

3.3 Data analysis

  • For analyzing the questionnaire, the authors used the data analysis approach adopted in [5].
  • Similar to their research, the questionnaire in [5] (N=15) is also characterized by small sample size.
  • The authors calculated the following indicators to interpret the responses: 1. % Agree: Percentage of participants responding with either ‘Agree’ or ‘Strongly agree’ 2. Top-Box: Percentage of participants responding with ‘Strongly agree’ 3. Net-Top-2-Box - Percentage of participants that chose bottom two responses (Strongly disagree and Disagree) subtracted from the participants that chose top two responses (Strongly agree and Agree).
  • Coefficient of Variance (CV): Standard deviation divided by the mean.
  • The first three indicators are a measure of central tendency, which indicates a single value that helps to identify the central value in a set of data.

4 Results

  • Results from the questionnaire helps answer RQ2, which are also complemented by inputs from the invited participants.
  • Using the ‘well-defined issues jira’ metric, the UC Champion learned that the developers were not always following the practice of maintaining the ‘Definition of Done’ (DoD)4 field in Jira5.
  • Due to a relatively shorter development period and smaller project size, CC4’s UC team have been able to contextualize their metrics, without putting a strain on the available resources.
  • In contrast, CC2 are neutral (Md=3), and the comment that an actionable metric should be, “easy to understand what it means, quickly”, provides a rationale for the said perspective.
  • CC1 exercises caution in labeling a metric actionable, as most metrics can only be one of the several contributors and not the primary driver of action, an argument justified by their experience of metrics’ use in the Project so far.

5 Discussion

  • First, the authors discuss the results of RQ1, as presented above, informed by the views and rationale provided by the invited participants.
  • Choice of metrics and their utility can be dictated by company size and project characteristics [8].
  • Just like CC3, company size and project characteristics significantly influence a metric’s role in CC4.
  • There is some support for the perspectives that every metric has to be actionable, and conversely, only certain metrics need to be actionable.
  • The ‘complete’ data quality requirement can be classified as a contextual quality [10], because the data quality requirement’s relevance and importance is based on the context of the task at hand.

6 Threats to validity

  • The authors address threats to their study’s validity based on the guidelines recommended by Runeson and Höst [18].
  • This shortcoming can still pose a threat to their study’s construct validity.
  • Combining the questionnaire responses with the empirical accounts of actionable metric use, and further validation by a practitioner from each case company help us mitigate this threat.
  • The authors study involves only four companies, and the participants for the questionnaire are based on convenience sampling.
  • This affects the external validity of their study.

7 Conclusion

  • Ideally, a metrics program should facilitate data-driven decisionmaking.
  • In the context of the EU H2020 Project, Q-Rapids, the authors attempted to address this research gap by collaborating with the four industrial partners.
  • Building upon the empirical accounts of metrics’ use for decision-making at the case companies, the authors administered an online questionnaire to document the involved practitioners’ views on actionable metrics.
  • There is some support for the perspectives that every metric has to be actionable and only certain metrics can be actionable.
  • The evidence suggests that both can be valid.

Did you find this useful? Give us your feedback

Content maybe subject to copyright    Report

Actionable Software Metrics: An Industrial Perspective
Prabhat Ram
M3S, Faculty of ITEE
University of Oulu
Oulu, Finland
prabhat.ram@oulu.fi
Pilar Rodríguez
M3S, Faculty of ITEE
University of Oulu
Oulu, Finland
pilar.rodriguez@oulu.fi
Markku Oivo
M3S, Faculty of ITEE
University of Oulu
Oulu, Finland
markku.oivo@oulu.fi
Silverio Martínez-Fernández
Dept. ESSI, GESSI Research Group
UPC-BarcelonaTech
Barcelona, Spain
smartinez@essi.upc.edu
Alessandra Bagnato
Softeam
Paris, France
alessandra.bagnato@softeam.fr
Michał Choraś
ITTI Sp. z o.o, Poznań and UTP
Bydgoszcz, Poland
chorasm@utp.edu.pl
RafKozik
ITTI Sp. z o.o, Poznań and UTP
Bydgoszcz, Poland
rafal.kozik@itti.com.pl
Sanja Aaramaa
Nokia
Oulu, Finland
sanja.aaramaa@nokia.com
Milla Ahola
Bittium
Oulu, Finland
milla.ahola@bittium.com
actionable metric can facilitate their definition and development
right from the start of a software metrics program.
CCS CONCEPTS
General and reference Empirical studies; Measurement;
Metrics; Software and its engineering Empirical software
validation.
KEYWORDS
actionable metrics, data quality, context, metrics program
ACM Reference format:
Prabhat Ram, Pilar Rodríguez, Markku Oivo, Silverio Martínez-Fernández,
Alessandra Bagnato, Mich Choraś, Rafał Kozik, Sanja Aaramaa, Milla
Ahola. 2020. Actionable Software Metrics: An Industrial Perspective. In
Evaluation and Assessment in Software Engineering (EASE 20), April 15-
17, 2020, Trondheim, Norway. ACM, New York, NY, USA, 10 pages.
https://doi.org/10.1145/3383219.3383244
1 Introduction
One of the benchmarks of a software metrics program’s
effectiveness is how actionable it is [22], since software
practitioners are not interested only in mining data for insights but
also in using those insights to guide their actions [20, 24]. Software
metrics can be used to empower software practitioners to gain and
share insights from their data for better decision-making [13, 25].
In addition to being trustworthy [17] or reliable [20], such metrics
need to manifest practitioners practical requirements in order to
contribute any actionable information [25]. Essentially, a metric
that can influence action [2] to facilitate process improvement or
remedy a situation [23] is an actionable metric. Existing literature
posits data quality dimensions like accessibility, completeness,
timeliness, relevancy, etc. to evaluate a metric’s reliability, and
enhance its suitability as a contributor to decision-making [20].
ABSTRACT
Background: Practitioners would like to take action based on
software metrics, as long as they find them reliable. Existing
literature explores how metrics can be made reliable, but remains
unclear if there are other conditions necessary for a metric to be
actionable. Context & Method: In the context of a European H2020
Project, we conducted a multiple case study to study metricsuse
in four companies, and identified instances where these metrics
influenced actions. We used an online questionnaire to enquire
about the project participants views on actionable metrics. Next,
we invited one participant from each company to elaborate on the
identified metricsuse for taking actions and the questionnaire
responses (N=17). Result: We learned that a metric that is practical,
contextual, and exhibits high data quality characteristics is
actionable. Even a non-actionable metric can be useful, but an
actionable metric mostly requires interpretation. However, the
more these metrics are simple and reflect the software development
context accurately, the less interpretation required to infer
actionable information from the metric. Company size and project
characteristics can also influence the type of metric that can be
actionable. Conclusion: This exploration of industry’s views on
actionable metrics help characterize actionable metrics in practical
terms. This awareness of what characteristics constitute
an
https://doi.org/10.1145/3383219.3383244
© ACM. PREPRINT. This is the author's version of the work. It is posted here by permission of ACM for your
personal use. Not for redistribution. The definitive version will be published in Proceedings of EASE 2020:
24th International Conference on Evaluation and Assessment in Software Engineering.
The final publication can be accessed under the DOI: https://doi.org/10.1145/3383219.3383244

Actionable Software Metrics: An Industrial Perspective
P. Ram et al.
However, it is unclear if a metric’s reliability is both a sufficient
and necessary condition for it to be actionable, or if there are
additional conditions that dictate a metric’s actionability.
A common misconception in software development is that a
metric is always actionable, regardless of whether that metric is
merely an indicator of a condition, or if it can inform a practitioner
on what to do next [12]. However, literature suggests that a metric
is actionable only in the latter case [2]. There are studies that
recommend success factors for a metrics program [17, 20, 21], and
suggest measures to increase responsiveness to customers’ needs
[15], while broaching the topic of using metrics for taking actions.
However, these studies do not discuss what makes a metrics truly
actionable, beyond their requirement to be reliable [20]. The case
study by Vacanti and Vallet [23] at Siemens Health Services (SHS)
attempt to characterize actionable metrics using a real-world
implementation as an example. The authors adopt flow
1
metrics,
which they claim to be actionable, to improve overall predictability
and process performance at the company [23]. The authors propose
that an actionable metric should be able to suggest specific
interventions if and when needed [23]. Beyond this all-
encompassing property of an actionable metric, the authors do not
discuss any other specific characteristics of an actionable metric.
Based on the state of the art, we gather that an actionable metric
should be contextual, reliable, and provide an impetus to act. In this
view, we aim to investigate further to obtain an industrial
perspective on actionable metrics, which can validate the above
findings and even supplement them. We argue that this knowledge
will help practitioners focus their efforts on defining and
developing such metrics right from the start of a metrics program.
The goal of the EU H2020 Project, Q-Rapids, is to develop an
agile-based, data-driven, and quality-aware rapid software
development process [3]. Four software-intensive companies are
involved as industry partners in the Project. These companies have
been using the Project solution (Solution), including its metrics, in
their daily work. They have also initiated improvements by acting
on some of the metrics. These developments and the research aim
to characterize actionable metrics from an industrial perspective
motivated the following research questions (RQs):
1. RQ1: What role did the metrics play in the improvement
actions taken by the participants of the study?
2. RQ2: What is an actionable metric according to the
participants of the study?
a. RQ2.1: What are the characteristics of an actionable
metric?
b. RQ2.2: What is the utilitarian perspective of an
actionable metric?
c. RQ2.3: What are the high-quality data requirements
for an actionable metric?
The participants from the Project are in an ideal position to
provide rationale for their decisions, especially to highlight
1
Workflow metrics such as Work in Progress, Cycle Time, and Throughput are flow
metrics
metrics’ role in the improvement actions. These rationales inform
our following main contributions:
1. Providing empirical accounts of software metrics that are
actionable, contributing to practitioners’ decision-making
2. Analyzing practitioners’ perspective on what an actionable
metric is, and the characteristics that make a metric actionable
in the context of software development.
In the remainder of the paper, we synthesize related work in
Section 2, describe the research method in Section 3, followed by
the results in Section 4. Discussion of the results is in Section 5,
with Section 6 discussing limitations and threats to our research’s
validity, and conclusion and future research directions in Section 7.
2 Related Work
Research on metrics program (MP) has a long history [9], but what
makes a metric truly actionable remains partially explored. We first
look at the literature proposing characteristics of a successful
metrics program, followed by the literature that explicitly address
actionable metrics in some capacity. The objective is to aggregate
characteristics of an actionable metrics reported in the state of the
art, if any, and position our study accordingly.
Studies by Mendoa and Basili [11], Hall and Fenton [4], and
Staron and Meding [21] propose success factors for an MP, but do
not explore the characteristics of metrics that can make them
actionable and contribute to an MP’s success. The case study in
[20] highlight the need for a metric to be reliable to make them
actionable. Similarly, the case study in [17] highlight five success
factors for effective operationalization of an MP, and the authors
emphasize on metrics trustworthiness in order for practitioners to
find them useful. Therefore, beyond a metric’s reliability, a product
of metric’s trustworthiness, the above-mentioned studies do not
tackle what makes a metric actionable. These studies address the
bigger picture of eliciting factors that can help successfully deploy
an MP and sustain it. An organization’s default position on any MP
is to have useful metrics, but even non-actionable metrics can prove
useful [12]. Our study complements above studies, and investigates
the specific characteristics of an actionable metric.
Port and Taber [16] present examples of metrics and analytics
program used to support strategic maintenance of a critical system
at NASA’s Jet Propulsion Laboratory. The authors claim that these
metrics are actionable, as they have been validated by their proven
utility in maintaining the critical systems. In a brief account of
different types of metrics, Croll and Yoskovitz [2] posit that an
actionable metric should have the potential to induce change by
suggesting a course of action. At the same time, the authors caution
that actionable metrics cannot be “magic”, as they can only provide
guidance and not precise instructions. Buse and Zimmerman [1]
surveyed 110 practitioners from Microsoft to understand their
decision-making process, and found that managers rate
data/metrics highly for taking actions. The authors posit that even

Actionable Software Metrics: An Industrial Perspective
EASE 20, April 15-17, 2020, Trondheim, Norway
common metrics can be actionable, as long as they are context-
driven. The above studies indicate the importance of a metric’s
actionability for the decision-makers, but discussion on what is
needed to make a metric actionable has received scant attention.
The case study by Vacanti and Vallet [23] at SHS is among the
recent studies that attempts to characterize actionable metrics in an
industrial environment. In order to realize the predictability and
transparency promised by the traditional agile metrics such as story
points and velocity, SHS decided to use Kanban, along with much
simpler and more actionable metrics of flow (Work in Progress,
Cycle Time, Throughput). The flow metrics are tied together by
Little’s Law
2
, and change in one metric influences the other two. If
the said change is undesired, then the inherent relationship among
these flow metrics will suggest what steps are needed to overcome
that state. Beyond the inherent actionability due to Little’s Law and
use of simpler metrics, the authors do not discuss any further the
characteristics that make the flow metrics, or any metric, actionable.
The state of the art highlights several factors that drive an MP’s
success, and a metrics usefulness is integral to that success.
However, if a real metric has to be actionable [2], then practitioners
should be aware of the steps needed to ensure that right from the
start of an MP. Our research is an attempt to delineate
characteristics that make a metric actionable, and contribute to the
abovementioned steps.
3 Research Method
Following the guidelines recommended by Runeson and Höst [18]
we conducted a multiple case study to answer the two RQs.
3.1 Research context
The following table characterizes the context of the four
companies, to which the participants of this study belong:
Table 1 Case Company Characteristics
Parameters
Case
Company 1
Case
Company 2
Case
Company 3
Case
Company 4
ID
CC1
CC2
CC3
CC4
Size
Large
Medium
Large
Small and
Medium
sized
Enterprise
Domain
Commercial
services and
solutions
Defense &
Telecom.
Telecom
Multi-
industry
Use case
Software
modeling
tool
Product
information
system
Hardware-
oriented
project
Warehouse
Mgmt.
System
Length of
Solution use
~ 2 years
8 months
deployment
only
~ 2 years
Use case team
size
9
15
~120
10
CC1 used the Solution in the context of a modeling tool for
model-driven development, which is part of a mature product line,
2
Little’s Law: Avg. Cycle Time = Avg. Work in Progress / Avg. Throughput.
with multiple releases already in the market. The company had used
Solution in the course of development of past three releases of the
above tool, with the fourth underway. Their views on actionable
metrics is informed by this cumulative experience.
In case of CC2, we focused on the metrics, and their use, from
the second use case (UC), as the first (pilot) UC was no longer in
active development. Taking together the two UCs, duration of
Solution use at CC2 is comparable to the other case companies.
A comparable Solution use at CC3 was hindered due to several
unforeseeable technical and managerial circumstances. Therefore,
in contrast to the other case companies, CC3’s views on actionable
metrics are based on the Solution deployment and preliminary use.
Following a positive and a formative experience from their pilot
use case, CC4 continued to use the Solution in the next UC, albeit
with customizations that are exclusive to this case company only.
The experience accumulated from these two UCs inform CC4’s
views on actionable metrics.
3.2 Data collection
The case companies shared metrics data with the Project
researchers on a monthly basis, along with a short report
documenting their use of the Solution. In the course of several
follow-up interactions with them, we learned that three of the case
companies (CC1, CC2, and CC4) had utilized metrics to undertake
process improvements. This knowledge triggered the need for a
questionnaire to gather the rationale that underlie the aforesaid
improvement actions.
We relied on the reviewed literature to design the questionnaire.
We formulated a total of 15 statements, measured on a 1 5 Likert
scale (strongly disagree disagree neutral agree strongly
agree don’t know). These 15 statements and their sources of
reference are shown in the table below:
Table 2 Questionnaire
Questionnaire statements
Reference
An actionable metric…
...is practical
[2, 25]
…informs decision-making
…is project/process /product specific
…can be universal
[2]
...has to have high data quality
[20]
Every metric has to be actionable
[2]
Only certain metrics need to be actionable
Without interpretation, a metric cannot be
actionable
[1]
Aggregated metric is more actionable than a
low-level metric (e.g. "resolved issues
throughput" more actionable than "number of
open issues")
[1, 19, 25]
A non-actionable metric can be useful
High-quality data are…
…accurate
[6]
…consistent
…current
…credible
…complete

Actionable Software Metrics: An Industrial Perspective
P. Ram et al.
With Q1-Q5 statements, we enquire about the general
characteristics of an actionable metric. Statements Q6-Q10 enquire
an actionable metric’s utilitarian perspective, which characterizes
the practicalities under which a metric can be actionable. Both these
sections were followed by an option for additional comments. Q11-
Q15 statements capture general characteristics of high-quality data,
which can be attributed to actionable metrics data. The five data
quality characteristics used in the questionnaire refer to the inherent
data quality, as prescribed in ISO/IEC 25012:2008 [6]. This section
was followed by the question Are there any other data quality
characteristics we missed? Please mention them below, along with
the rationale”.
The questionnaire was administered online to the users of the
Solution from the four companies. The aim was to elicit responses
from the users that were involved in the metrics-driven
improvement actions at their companies. We received responses
from 17 participants.
Following the questionnaire, we invited one participant from
each case company to co-author this study. The aim was to gather
first-hand insights about the actions influenced by the metrics, and
to provide rationale to the questionnaire responses from the
participants of their respective companies.
3.3 Data analysis
For analyzing the questionnaire, we used the data analysis approach
adopted in [5]. Similar to our research, the questionnaire in [5]
(N=15) is also characterized by small sample size. We calculated
the following indicators to interpret the responses:
1. % Agree: Percentage of participants responding with either
Agree’ or ‘Strongly agree
2. Top-Box: Percentage of participants responding with
Strongly agree
3. Net-Top-2-Box - Percentage of participants that chose bottom
two responses (Strongly disagree and Disagree) subtracted
from the participants that chose top two responses (Strongly
agree and Agree).
4. Coefficient of Variance (CV): Standard deviation divided by
the mean. A higher CV indicates higher variability.
The first three indicators are a measure of central tendency,
which indicates a single value that helps to identify the central value
in a set of data.
4 Results
Answer to RQ1 is provided by the invited participants. Results
from the questionnaire helps answer RQ2, which are also
complemented by inputs from the invited participants.
3
https://www.openproject.org/
4
https://www.atlassian.com/blog/jira-software/8-steps-to-a-definition-of-done-in-jira
5
https://www.atlassian.com/software/jira
4.1 RQ1. What role did the metrics play in the
improvement actions taken by the participants of
the study?
In this section, the participants elaborate on the instances where
they used metrics for taking improvement actions. A complete list
of metrics for each case company can be found in Appendix A
(https://doi.org/10.5281/zenodo.3580893).
4.1.1 Metricsrole in CC1. The UC development team holds a
meeting every day to discuss their next release and the next steps
in the context of the forthcoming product release. In one of these
meetings, the team used the non-blocking files’ metric to identify
problems that were blocking certain development tasks, which
were critical for developing features for their upcoming product
release. To avoid delays, the team prioritized their development
activities based on the tasks the above metric helped identify.
The Solution provides a quality-alert feature, where once a
metric crosses a user-defined threshold, indicating violation of a
quality goal defined by the UC Quality Engineers (QEs), it triggers
an alert. The alert is accompanied by a recommendation to correct
the situation, which is recorded as an abstract quality issue (quality
requirement) in CC1s project management tool OpenProject
3
. The
Project Manager (PM) decides if the quality issue should be
integrated as a concrete development task in the project or not. For
example, the PM accepted the alert generated by the metric critical
issues ratio, which tracks completed development tasks with
critical severity issues, and converted it into a development task
that called for review and update of the team’s validation process.
Owing to the above described Project feature, CC1 has formalized
their quality requirements management, and moved away from the
informal approach of mobilizing resources on an ad-hoc basis to
address a quality issue.
In case of the non-blocking file metric, the UC team was
familiar with the metric and knew it to be reliable. In case of the
critical issues ratio’ metric, again the familiarity of the metric and
PM’s close scrutiny of the generated alert played an important role.
In summation, the metrics’ role in the improvement actions is
undeniable, but they were among several other contributing factors.
4.1.2 Metricsroles in CC2. Using the well-defined issues jira
metric, the UC Champion learned that the developers were not
always following the practice of maintaining the Definition of
Done(DoD)
4
field in Jira
5
. DoD is a concise list of requirements
that a product should meet for a development team to call it
complete, and the above metric helps track the Jira issues
6
where
the field is defined. The metric enabled the UC Champion to take
the decision of reinforcing the practice of maintaining DoD across
the team. Soon thereafter, the metrics value started increasing,
reflecting improvement in the practice of maintaining the DoD
fields.
6
https://confluence.atlassian.com/adminjiracloud/issue-types-
844500742.html

Table 3 Overview of the questionnaire results
ID
Questionnaire statement
Scale
(1 5)
N
% Agree
Top-Box
Net-Top-
2- Box
Coefficient of
Variance (CV)
An actionable metric…
Q1
...is practical
16
94%
63%
94%
14%
Q2
…informs decision-making
17
88%
47%
76%
23%
Q3
…is project/process /product specific
15
47%
20%
20%
33%
Q4
…can be universal
16
56%
6%
25%
35%
Q5
...has to have high data quality
17
82%
71%
76%
21%
Q6
Every metric has to be actionable
17
47%
18%
12%
35%
Q7
Only certain metrics need to be actionable
17
35%
0%
0%
37%
Q8
Without interpretation, a metric cannot be
actionable
15
73%
27%
67%
22%
Q9
Aggregated metric is more actionable than a low-
level metric (e.g. "resolved issues throughput"
more actionable than "number of open issues")
17
18%
6%
-24%
35%
Q10
A non-actionable metric can be useful
16
75%
25%
69%
22%
High-quality data are…
Q11
…accurate
17
100%
59%
100%
11%
Q12
…consistent
17
100%
59%
100%
11%
Q13
…current
17
82%
29%
76%
20%
Q14
…credible
15
100%
80%
100%
9%
Q15
…complete
17
88%
41%
82%
20%
*The graph distribution scale from left to right: Strongly disagree, Disagree, Neutral, Agree, and Strongly Agree
General characteristics of an actionable metric
Utilitarian perspectives for an actionable metric
High-quality data for an actionable metric
Figure 1 Questionnaire responses (case company-wise)
Right after deploying the metrics, the UC team started
customizing them to have them reflect their development practices.
However, the ‘well-defined issues jira metric did not warrant any
change, as the team found it simple enough to rely on them. This
perception and the trust it instilled among team members, made it
the main driver of the undertaken improvement action.
4.1.3 Metrics’ potential role in CC3. CC3 engages in very large
and complex projects, involving globally distributed development
teams. Combining all of CC3’s business lines, terabytes of test log
data are generated on a daily basis. It is humanly impossible and
unfeasible to sort and parse all that data to infer insights, and act on
them to improve processes, if needed. In absence of specific
solutions to guide practitioners in their decision-making, they are
0
1
2
3
4
5
Q1 Q2 Q3 Q4 Q5
CC1 CC2 CC3 CC4
0
1
2
3
4
5
Q6 Q7 Q8 Q9 Q10
CC1 CC2 CC3 CC4
0
1
2
3
4
5
Q11 Q12 Q13 Q14 Q15
CC1 CC2 CC3 CC4

Citations
More filters
Journal ArticleDOI
TL;DR: Tim as discussed by the authors is a tool, implemented at a company developing embedded systems, where software development occurs in parallel branches and nightly testing is partitioned over software branches, test systems and test cases.
Abstract: Abstract Software testing is key for quality assurance of embedded systems. However, with increased development pace, the amount of test results data risks growing to a level where exploration and visualization of the results are unmanageable. This paper covers a tool, Tim, implemented at a company developing embedded systems, where software development occurs in parallel branches and nightly testing is partitioned over software branches, test systems and test cases. Tim aims to replace a previous solution with problems of scalability, requirements and technological flora. Tim was implemented with a reference group over several months. For validation, data were collected both from reference group meetings and logs from the usage of the tool. Data were analyzed quantitatively and qualitatively. The main contributions from the study include the implementation of eight views for test results exploration and visualization, the identification of four solutions patterns for these views (filtering, aggregation, previews and comparisons), as well as six challenges frequently discussed at reference group meetings (expectations, anomalies, navigation, integrations, hardware details and plots). Results are put in perspective with related work and future work is proposed, e.g., enhanced anomaly detection and integrations with more systems such as risk management, source code and requirements repositories.

6 citations

Journal ArticleDOI
TL;DR: Tim as discussed by the authors is a tool, implemented at a company developing embedded systems, where software development occurs in parallel branches and nightly testing is partitioned over software branches, test systems and test cases.
Abstract: Abstract Software testing is key for quality assurance of embedded systems. However, with increased development pace, the amount of test results data risks growing to a level where exploration and visualization of the results are unmanageable. This paper covers a tool, Tim, implemented at a company developing embedded systems, where software development occurs in parallel branches and nightly testing is partitioned over software branches, test systems and test cases. Tim aims to replace a previous solution with problems of scalability, requirements and technological flora. Tim was implemented with a reference group over several months. For validation, data were collected both from reference group meetings and logs from the usage of the tool. Data were analyzed quantitatively and qualitatively. The main contributions from the study include the implementation of eight views for test results exploration and visualization, the identification of four solutions patterns for these views (filtering, aggregation, previews and comparisons), as well as six challenges frequently discussed at reference group meetings (expectations, anomalies, navigation, integrations, hardware details and plots). Results are put in perspective with related work and future work is proposed, e.g., enhanced anomaly detection and integrations with more systems such as risk management, source code and requirements repositories.

4 citations

Proceedings ArticleDOI
09 Nov 2022
TL;DR: A metodologia empregada em revisão da literatura, considerando artigos relevantes de bases acadêmicas consistentes, como ACM, IEEE and Elsevie, além de dissertações, teses e revistas conceituadas na temática as discussed by the authors .
Abstract: A introdução das Metodologias Ágeis transformou a forma como as organizações trabalham. Menor tempo de desenvolvimento para o mercado, custos baixos e melhoria contínua, gerando autonomia para que o time encontre a solução ideal para o momento. As métricas ágeis proporcionam insights sobre o processo atual e ajustes necessários para melhorar o fluxo de trabalho. Isso ajuda a avaliar a qualidade de um produto, monitorar o desempenho do time e melhorar seu desempenho ao longo do tempo. Neste artigo, discutimos métricas ágeis que são relevantes para o sucesso das organizações. Citamos também algumas métricas ágeis essenciais para o desenvolvimento de negócios de uma organização. A metodologia empregada é baseada em revisão da literatura, considerando artigos relevantes de bases acadêmicas consistentes, como ACM, IEEE e ScienceDirect – Elsevie, além de dissertações, teses e revistas conceituadas na temática.
References
More filters
Journal ArticleDOI
TL;DR: This paper aims at providing an introduction to case study methodology and guidelines for researchers conducting case studies and readers studying reports of such studies, and presents recommended practices and evaluated checklists for researchers and readers of case study research.
Abstract: Case study is a suitable research methodology for software engineering research since it studies contemporary phenomena in its natural context. However, the understanding of what constitutes a case study varies, and hence the quality of the resulting studies. This paper aims at providing an introduction to case study methodology and guidelines for researchers conducting case studies and readers studying reports of such studies. The content is based on the authors' own experience from conducting and reading case studies. The terminology and guidelines are compiled from different methodology handbooks in other research domains, in particular social science and information systems, and adapted to the needs in software engineering. We present recommended practices for software engineering case studies as well as empirically derived and evaluated checklists for researchers and readers of case study research.

3,620 citations


"Actionable Software Metrics: An Ind..." refers methods in this paper

  • ...We address threats to our study’s validity based on the guidelines recommended by Runeson and Höst [18]....

    [...]

  • ...Following the guidelines recommended by Runeson and Höst [18] we conducted a multiple case study to answer the two RQs....

    [...]

Journal ArticleDOI
TL;DR: The methodology encompasses a model of IQ, a questionnaire to measure IQ, and analysis techniques for interpreting the IQ measures, which are applied to analyze the gap between an organization and best practices.

1,542 citations

Proceedings ArticleDOI
02 Jun 2012
TL;DR: The data and analysis needs of professional software engineers are presented, which were identified among 110 developers and managers in a survey, and several guidelines for analytics tools in software development are proposed.
Abstract: Software development is a data rich activity with many sophisticated metrics. Yet engineers often lack the tools and techniques necessary to leverage these potentially powerful information resources toward decision making. In this paper, we present the data and analysis needs of professional software engineers, which we identified among 110 developers and managers in a survey. We asked about their decision making process, their needs for artifacts and indicators, and scenarios in which they would use analytics. The survey responses lead us to propose several guidelines for analytics tools in software development including: Engineers do not necessarily have much expertise in data analysis; thus tools should be easy to use, fast, and produce concise output. Engineers have diverse analysis needs and consider most indicators to be important; thus tools should at the same time support many different types of artifacts and many indicators. In addition, engineers want to drill down into data based on time, organizational structure, and system architecture.

193 citations


"Actionable Software Metrics: An Ind..." refers background in this paper

  • ..."resolved issues throughput" more actionable than "number of open issues") [1, 19, 25]...

    [...]

  • ...Buse and Zimmerman [1] surveyed 110 practitioners from Microsoft to understand their decision-making process, and found that managers rate data/metrics highly for taking actions....

    [...]

  • ...Q8 Without interpretation, a metric cannot be actionable [1]...

    [...]

  • ...lower-level managers and developers tend to rely on detailed (lowlevel) metrics [1]....

    [...]

Journal ArticleDOI
TL;DR: It is shown that although Agile teams use many metrics suggested in the Agile literature, they also use many custom metrics, and the most influential metrics in the primary studies are Velocity and Effort estimate.
Abstract: ContextSoftware industry has widely adopted Agile software development methods. Agile literature proposes a few key metrics but little is known of the actual metrics use in Agile teams. ObjectiveThe objective of this paper is to increase knowledge of the reasons for and effects of using metrics in industrial Agile development. We focus on the metrics that Agile teams use, rather than the ones used from outside by software engineering researchers. In addition, we analyse the influence of the used metrics. MethodThis paper presents a systematic literature review (SLR) on using metrics in industrial Agile software development. We identified 774 papers, which we reduced to 30 primary studies through our paper selection process. ResultsThe results indicate that the reasons for and the effects of using metrics are focused on the following areas: sprint planning, progress tracking, software quality measurement, fixing software process problems, and motivating people. Additionally, we show that although Agile teams use many metrics suggested in the Agile literature, they also use many custom metrics. Finally, the most influential metrics in the primary studies are Velocity and Effort estimate. ConclusionThe use of metrics in Agile software development is similar to Traditional software development. Projects and sprints need to be planned and tracked. Quality needs to be measured. Problems in the process need to be identified and fixed. Future work should focus on metrics that had high importance but low prevalence in our study, as they can offer the largest impact to the software industry.

167 citations

Journal ArticleDOI
TL;DR: The authors identify consensus requirements for metric program success and examine how programs in two organisations measured up.
Abstract: Increasingly organisations are foregoing an ad hoc approach to metrics in favor of complete metrics programs. The authors identify consensus requirements for metric program success and examine how programs in two organisations measured up.

154 citations