PROCEEDINGS
SEKE 2015
The 27
th
International Conference on
Software Engineering &
Knowledge Engineering
Sponsored by
KSI Research Inc. and Knowledge Systems Institute Graduate School, USA
Technical Program
July 6 – 8, 2015
Wyndham Pittsburgh University Center, Pittsburgh, USA
Organized by
KSI Research Inc. and Knowledge Systems Institute Graduate School, USA
Copyright ף 2015 by KSI Research Inc. and Knowledge Systems Institute Graduate School
All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any
form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior written consent
of the publisher.
ISBN: 1-891706-37-3
ISSN: 2325-9000 (print)
2325-9086 (online)
Additional copies can be ordered from:
KSI Research Inc. and Knowledge Systems Institute Graduate School
3420 Main Street
Skokie, IL 60076 USA
Tel: +1-847-679-3135
Fax: +1-847-679-3166
Email: seke@ksiresearch.org
Web: http://www.ksi.edu
Proceedings preparation, editing and printing are sponsored by KSI Research Inc. and Knowledge Systems Institute
Graduate School, USA.
Printed by KSI Research Inc. and Knowledge Systems Institute Graduate School
ii
A Behavior Marker tool for measurement of the Non-
Technical Skills of Software Professionals: An
Empirical Investigation
Lisa L. Lacher
1
, Gursimran S. Walia
2
, Fabian Fagerholm
3
, Max Pagels
4
, Kendall Nygard
5
, Jürgen Münch
6
Department of Computer Science
University of Houston-Clear Lake
1
; North Dakota State University
2, 5
; University of Helsinki
3, 4, 6
Lacher@uhcl.edu
1
; {gursimran.walia
2
, Kendall.Nygard
5
}@ndsu.edu; {fabian.fagerholm
3
, max.pagels
4
, juergen.muench
6
}@cs.helsinki.fi
Abstract— Managers recognize that software development project
teams need to be developed and guided. Although technical skills
are necessary, non-technical (NT) skills are equally, if not more,
necessary for project success. Currently, there are no proven tools
to measure the NT skills of software developers or software
development teams. Behavioral markers (observable behaviors
that have positive or negative impacts on individual or team
performance) are beginning to be successfully used by airline and
medical industries to measure NT skill performance. The purpose
of this research is to develop and validate the behavior marker
system tool that can be used by different managers or coaches to
measure the NT skills of software development individuals and
teams. This paper presents an empirical study conducted at the
Software Factory where users of the behavior marker tool rated
video clips of software development teams. The initial results show
that the behavior marker tool can be reliably used with minimal
training.
Keywords-Non-technical Skills; behavior marker; performance.
I. INTRODUCTION
Most software is developed by teams and the success of a
software project depends on the effective performance of the
software project team. The PMI and the most recent PMBOK
Guide [1] acknowledges that, non-technical (NT) skills in
comparison to the technical skills are equally important for
project success and team development. Several authors agree
that the NT skills are critical to project success [2, 3]; and there
are even some that assert that NT skills can have the largest
impact on software development [4, 5].
The growing need for an agile workforce is one major factor
that is driving the demand for NT skills [6]. Agile Manifesto’s
[7] first principle - “individuals and interactions over processes
and tools” clearly points to the importance of NT skills. Agile
teams depend greatly on NT skills such as efficient
communication, taking responsibility, initiative, time
management, and leadership.
While it is obvious that NT skills are important, and that the
performance of individuals is very important to creating an
effective team, there are no established guidelines for measuring
team effectiveness. Different criteria for assessing team
effectiveness have been identified by different authors [8, 9].
Generally, these criteria include measurements of task
performance as well as the interpersonal skills of the team
members. The interpersonal skills include attitudes and
behaviors. Although there is extensive literature with respect to
different ways to measure task performance for software
development (e.g., lines of code) [10], scant research has been
performed on the measurement of NT skills, especially for
software developers. A couple of notable exceptions can be
found in the aviation and health care industries. Both industries
have already recognized the importance of NT skills to the
success of their teams, and have been using behavioral marker
(BM) systems (e.g., LOSA, ANTS) to structure individual and
team assessments of these NT skills. We believe that software
teams can also draw upon these BM’s from the aviation and
health care industries. It is often Software Development
managers and coaches that are responsible for assessing the
performance of their development teams – not HR departments,
thus a tool like a BM system needs to be available to them.
As educators and software project development managers,
we are concerned with questions such as: how can managers
objectively measure the NT skills of their employees to
determine if their NT skills need improvement or how would
feedback be provided to the team members so that they could
improve their performance? This research attempts to begin
answering these kinds of questions.
II. B
ACKGROUND –NT SKILLS,BEHAVIOR MARKERS
Non Technical (NT) Skills: NT skills are the cognitive,
personal resource, and social skills that complement a person’s
technical skills and contribute to overall task performance [11].
Some classic examples of NT skills include communication,
cooperation, decision making, leadership, stress management,
and workload management. Basically; NT skills cover the
cognitive and social sides of a person. In the most recent survey
released by the Association of American Colleges and
Universities [12], it was found that employers feel that NT skills
are more important than a particular major. Several different
surveys of U.S. employers have also identified
a lack of NT
skills as the area where young job-seekers have the largest
deficiency [13]. Even professional organizations such as
Professional Engineering Competence (UKSPEC), IEEE
Computer Society state that professionals have an obligation to
possess NT skills [14].
Behavior Markers (BM): Behavioral markers (BM) are
defined [15] as “observable, non-technical behaviors that
contribute to superior or substandard performance within a
work environment”. They are derived by analyzing data
regarding performance that contributes to successful and
unsuccessful outcomes. The overall purpose of a BM system is
to use markers as a method to assess both team and individual
behaviors. These BM systems provide an observation-based
(DOI reference number: 10.18293/SEKE2015-227)
409
method to capture and assess individual and team performance
on data rather than on gut feelings. The BM tool is designed in
the form of a structured list of behaviors. The Observers then
use this form during a selected work situation to rate
performance. This allows an individual’s or team’s skills to be
rated in their real context. BM systems can provide a common
language for giving feedback as well as discussing and teaching
NT skills.
Behavior Marker (BM) Systems: BM systems have
demonstrated value for assessing and providing feedback on
these NT skills, for improving training programs, and in the use
of building databases to identify norms and prioritize training
needs. It is important to recognize that BM systems need to be
specific to the domain and culture. A brief description of
successful BM systems (airline, medicine) follows:
The first BM system, Line Operation Safety Audit (LOSA) is
a very successful BM system that focuses on interpersonal
communication, leadership, and decision making in the cockpit.
Trained observers ride along in the cockpit and observe the
flight crews during normal flight operations. They score the
behaviors of the crew using the LOSA tool. LOSA has been
endored by the International Civil Aviation Organiztion
because it has been used so successful in measuring the
strengths and weaknesses of flight crews’ interpersonal skills
[16]. The Anesthetists’ NT Skills (ANTS) [17] used in
healthcare has proven very useful in assessing the NT skills of
anesthetists in simulation training and has provided important
performance feedback for the individuals. Another successful
healthcare BM system is the Observational Teamwork
Assessment of Surgery (OTAS). Many studies have shown that
poor communication, coordination, and other aspects of
teamwork, rather than technical failures, have been the primary
causes of adverse events in surgery. OTAS has been found to
be a valid measure of the NT performance of surgical teams
[18].
Our goal is to develop and validate a BM system that can
improve software professional team member performance by
providing feedback in the form of an objective and documented
assessment of the NT skills of the team members. We wanted
to create a tool that is very usable by practitioners: it requires
little or no training to use and does not require unreasonable
effort to use. It is a concern of the researcher that if the tool took
a lot of training or was too difficult to use, that the potential
practitioners, such as project managers and team leads for
whom the tool was meant to assist, would not find the tool
useful because of the amount of effort required.
III. B
EHAVIOR MARKER SYSTEM DEVELOPMENT
The development process for our behavioral marker system
for software developers is detailed in our previous work [19].
As a first step, we performed a systematic literature review to
develop NT skill inventory. The high-level question addressed
by the review was: “What are the NT skills required of software
professionals performing well in their field and how can we
discover what NT skills are valued by employers?”
Details on the review protocol (sources searched, search
execution, inclusion and exclusion criteria, quality assessment,
data extraction) can be referred to in a report [20]. The output
of this step was an initial list of 35 NT skills that were clustered
into four major categories: communication, interpersonal,
problem solving, and work ethic (see Fig. 1). The detailed
desription of each skill can be referred [20].
During the second step, the initial list of NT skills had their
quality assessed and were validated by focus group of experts
in industry and academia. Two surveys (and focus groups) were
conducted online (using a cross sectional design) to gather NT
skill priorities, missing NT skills, description clarifications, and
examples of examples of good and poor behaviors for the top
rated NT skills of software developers. So that we could
prioritize our efforts, focus group ranked the importance of each
NT skill to software professionals during the first survey. After
the survey analysis, we had a reduced list of 16 skills to focus
on. During the second focus group survey, we gather a total of
408 examples of observable actions that indicated good
performance and behavior of each NT skill as well as examples
of observable actions that indicate poor performance and
behavior of each NT skill. These examples were reviewed,
clarified, and redundancies were eliminated. The final set of NT
skills consisted of: teamwork, initiative/motivation to work,
listening, attitude, critical thinking, oral communication,
problem solving, attention to detail, flexibility,
integrity/honesty/ethics, time management, and questioning.
Some behavioral examples, such as “being a good team player”
and “body language and persona emitting that you do not enjoy
your work”, were too ambiguous and removed. It was also felt
that the “Leadership” skill did not have enough observable
Fig. 1: Desired NT skills of Software Professionals
Fig. 2: Example of “Listening” behaviors (good and bad examples)
410
behaviors that would be able to be clearly identified, so that NT
skill was removed. The result of the second survey was a
behavior-based software engineer NT skills taxonomy. Fig. 2
shows the resultant examples of good and poor behavior for the
“Listening” skill. The same process was used to create examples
of good and poor behavior for each NT skill.
During the third step, the behavior marker systems being
used in aviation, health care, rail transport and maritime
transport were examined. Each system’s structure was examined
to select which elements would have the most potential for use
in software development and our final tool was a composition of
several systems. The NT skills validated by the focus group
along with the good and bad behavior examples for those skills
were structured into a BM audit tool for software development.
For reference, we refer to the BM audit tool as the Non-
Technical Skill Assessment for Software Developers (NTSA).
The NTSA is designed to be used by an observer (i.e.
manager, team leader, coach) during routine team interactions or
meetings. It is intended that each time a behavior is observed, a
mark is placed in the appropriate column by placing a tick mark
in that column: observed and good, or expected but not observed.
Observations can be clarified by placing explanations in the
comments section. The observer can see skill definitions and
examples of good and poor behavior for a particular behavioral
marker by viewing the second page. A manager is allowed to
list as many or as few skills as desired in the behavioral marker
column. The observer will score the behaviors based on how
well the behavior meets the behavioral examples and its
definition.
IV. E
MPIRICAL VALIDATION OF BEHAVIOR MARKER
In order to evaluate our BM tool, an empirical study rated
video clips of student software development teams that were
working on industrial strength projects within the Software
Factory (as shown in Fig. 3 and explained).
1) Software Factory Background
The Software Factory is a software development laboratory
created by the University of Helsinki, Department of Computer
Science. All research was performed in Finland due to the
requirements of international privacy laws. The University of
Helsinki is consistently ranked in the top 100 out of world's
15,000 universities, in part because the university promotes
science and research together with European's top research-
intensive universities. The master’s degree programs are taught
in English in order to support the large number of international
students who study at the university. The Software Factory’s
primary participants are students, but the businesses provide
team members who work with the students, and university
faculties oversee the projects, although the faculty involvement
is kept to a minimum. Almost all project communication is in
English. Faculty involvement consists primarily of project
orientation and project intervention if problems cannot be
resolved by the students, coach, and customer. The coach is
generally an upper level student with Software Factory project
experience. University students take on the role of the
development team for projects provided by businesses. The
customer has company representatives that take on the role of
the product owner and represents the interests of the company.
Although these representatives are not co-located, they do come
by the Software Factory for weekly demos, sometimes for
meetings, and are generally available via telephone and email.
Researchers are able to observe what happens in the project due
to the seven cameras that provide multiple angles of view and
four microphones that record activities in the Factory room. In
Software Factory projects, the participants take on the core roles
of a typical Scrum project. Projects at the Software Factory last
for seven to eight weeks; the students work approximately 6
hours per day, 4-5 days per week.
2) Study Design
This study investigates whether the BM system can be used
with consistency by different raters to capture a measurement of
the NT skills of software developers, thus facilitating objective
feedback to software development teams and individuals. This
study used a blocked subject-project study. This type of analysis
allows the examination of several factors within the framework
of one study. Each of the non-technical skills to be studied can
be applied to a set of projects by several subjects and each
subject applies each of the non-technical skills under study.
In
this study, raters evaluated the NT skills of project teams using
the NTSA tool. The project teams worked together using state-
of-the-art tools, modern processes and best practices to
prototype and develop software for real business customers in an
environment that emulates industry. Video tapes of the projects
were evaluated to rate the student team’s NT skill performance.
The details of the study are provided as follows.
Independent and dependent variables:The experiment
manipulated the following independent variable:
a) Behavioral Marker System tool and Example
Behaviors: Each non-technical skill has its own set of good and
Fig. 3: Software Factory
411
