scispace - formally typeset

Proceedings ArticleDOI

A Behavior Marker tool for measurement of the Non-Technical Skills of Software Professionals: An Empirical Investigation

01 Jul 2015-pp 409-414

AbstractManagers recognize that software development project teams need to be developed and guided. Although technical skills are necessary, non-technical (NT) skills are equally, if not more, necessary for project success. Currently, there are no proven tools to measure the NT skills of software developers or software development teams. Behavioral markers (observable behaviors that have positive or negative impacts on individual or team performance) are beginning to be successfully used by airline and medical industries to measure NT skill performance. The purpose of this research is to develop and validate the behavior marker system tool that can be used by different managers or coaches to measure the NT skills of software development individuals and teams. This paper presents an empirical study conducted at the Software Factory where users of the behavior marker tool rated video clips of software development teams. The initial results show that the behavior marker tool can be reliably used with minimal training. Keywords-Non-technical Skills; behavior marker; performance.

Topics: Personal software process (63%), Software factory (61%), Skills management (60%), Software development (58%), Software (50%)

Summary (1 min read)

Jump to:  and [Fig.2:]

Fig.2:

  • Means that the raters do not have to have the same level of experience or backgrounds in order to use the tool and get reliable results.
  • Another potential threat is that both projects were fairly successful, and thus may not have exercised the poor behavior examples enough.
  • Such as tool provides a mechanism with which to improve a team and by extension the software they produce.
  • The NTSA provides a common language with which to understand and communicate about NT skills important to software professionals.
  • This would give educators and managers a rich set of NT skills and behaviors that could be evaluated.

Did you find this useful? Give us your feedback

...read more

Content maybe subject to copyright    Report

PROCEEDINGS
SEKE 2015
The 27
th
International Conference on
Software Engineering &
Knowledge Engineering
Sponsored by
KSI Research Inc. and Knowledge Systems Institute Graduate School, USA
Technical Program
July 6 – 8, 2015
Wyndham Pittsburgh University Center, Pittsburgh, USA
Organized by
KSI Research Inc. and Knowledge Systems Institute Graduate School, USA

Copyright ף 2015 by KSI Research Inc. and Knowledge Systems Institute Graduate School
All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any
form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior written consent
of the publisher.
ISBN: 1-891706-37-3
ISSN: 2325-9000 (print)
2325-9086 (online)
Additional copies can be ordered from:
KSI Research Inc. and Knowledge Systems Institute Graduate School
3420 Main Street
Skokie, IL 60076 USA
Tel: +1-847-679-3135
Fax: +1-847-679-3166
Email: seke@ksiresearch.org
Web: http://www.ksi.edu
Proceedings preparation, editing and printing are sponsored by KSI Research Inc. and Knowledge Systems Institute
Graduate School, USA.
Printed by KSI Research Inc. and Knowledge Systems Institute Graduate School
ii

A Behavior Marker tool for measurement of the Non-
Technical Skills of Software Professionals: An
Empirical Investigation
Lisa L. Lacher
1
, Gursimran S. Walia
2
, Fabian Fagerholm
3
, Max Pagels
4
, Kendall Nygard
5
, Jürgen Münch
6
Department of Computer Science
University of Houston-Clear Lake
1
; North Dakota State University
2, 5
; University of Helsinki
3, 4, 6
Lacher@uhcl.edu
1
; {gursimran.walia
2
, Kendall.Nygard
5
}@ndsu.edu; {fabian.fagerholm
3
, max.pagels
4
, juergen.muench
6
}@cs.helsinki.fi
Abstract Managers recognize that software development project
teams need to be developed and guided. Although technical skills
are necessary, non-technical (NT) skills are equally, if not more,
necessary for project success. Currently, there are no proven tools
to measure the NT skills of software developers or software
development teams. Behavioral markers (observable behaviors
that have positive or negative impacts on individual or team
performance) are beginning to be successfully used by airline and
medical industries to measure NT skill performance. The purpose
of this research is to develop and validate the behavior marker
system tool that can be used by different managers or coaches to
measure the NT skills of software development individuals and
teams. This paper presents an empirical study conducted at the
Software Factory where users of the behavior marker tool rated
video clips of software development teams. The initial results show
that the behavior marker tool can be reliably used with minimal
training.
Keywords-Non-technical Skills; behavior marker; performance.
I. INTRODUCTION
Most software is developed by teams and the success of a
software project depends on the effective performance of the
software project team. The PMI and the most recent PMBOK
Guide [1] acknowledges that, non-technical (NT) skills in
comparison to the technical skills are equally important for
project success and team development. Several authors agree
that the NT skills are critical to project success [2, 3]; and there
are even some that assert that NT skills can have the largest
impact on software development [4, 5].
The growing need for an agile workforce is one major factor
that is driving the demand for NT skills [6]. Agile Manifesto’s
[7] first principle - “individuals and interactions over processes
and tools clearly points to the importance of NT skills. Agile
teams depend greatly on NT skills such as efficient
communication, taking responsibility, initiative, time
management, and leadership.
While it is obvious that NT skills are important, and that the
performance of individuals is very important to creating an
effective team, there are no established guidelines for measuring
team effectiveness. Different criteria for assessing team
effectiveness have been identified by different authors [8, 9].
Generally, these criteria include measurements of task
performance as well as the interpersonal skills of the team
members. The interpersonal skills include attitudes and
behaviors. Although there is extensive literature with respect to
different ways to measure task performance for software
development (e.g., lines of code) [10], scant research has been
performed on the measurement of NT skills, especially for
software developers. A couple of notable exceptions can be
found in the aviation and health care industries. Both industries
have already recognized the importance of NT skills to the
success of their teams, and have been using behavioral marker
(BM) systems (e.g., LOSA, ANTS) to structure individual and
team assessments of these NT skills. We believe that software
teams can also draw upon these BM’s from the aviation and
health care industries. It is often Software Development
managers and coaches that are responsible for assessing the
performance of their development teams – not HR departments,
thus a tool like a BM system needs to be available to them.
As educators and software project development managers,
we are concerned with questions such as: how can managers
objectively measure the NT skills of their employees to
determine if their NT skills need improvement or how would
feedback be provided to the team members so that they could
improve their performance? This research attempts to begin
answering these kinds of questions.
II. B
ACKGROUND –NT SKILLS,BEHAVIOR MARKERS
Non Technical (NT) Skills: NT skills are the cognitive,
personal resource, and social skills that complement a person’s
technical skills and contribute to overall task performance [11].
Some classic examples of NT skills include communication,
cooperation, decision making, leadership, stress management,
and workload management. Basically; NT skills cover the
cognitive and social sides of a person. In the most recent survey
released by the Association of American Colleges and
Universities [12], it was found that employers feel that NT skills
are more important than a particular major. Several different
surveys of U.S. employers have also identified
a lack of NT
skills as the area where young job-seekers have the largest
deficiency [13]. Even professional organizations such as
Professional Engineering Competence (UKSPEC), IEEE
Computer Society state that professionals have an obligation to
possess NT skills [14].
Behavior Markers (BM): Behavioral markers (BM) are
defined [15] as “observable, non-technical behaviors that
contribute to superior or substandard performance within a
work environment. They are derived by analyzing data
regarding performance that contributes to successful and
unsuccessful outcomes. The overall purpose of a BM system is
to use markers as a method to assess both team and individual
behaviors. These BM systems provide an observation-based
(DOI reference number: 10.18293/SEKE2015-227)
409

method to capture and assess individual and team performance
on data rather than on gut feelings. The BM tool is designed in
the form of a structured list of behaviors. The Observers then
use this form during a selected work situation to rate
performance. This allows an individual’s or team’s skills to be
rated in their real context. BM systems can provide a common
language for giving feedback as well as discussing and teaching
NT skills.
Behavior Marker (BM) Systems: BM systems have
demonstrated value for assessing and providing feedback on
these NT skills, for improving training programs, and in the use
of building databases to identify norms and prioritize training
needs. It is important to recognize that BM systems need to be
specific to the domain and culture. A brief description of
successful BM systems (airline, medicine) follows:
The first BM system, Line Operation Safety Audit (LOSA) is
a very successful BM system that focuses on interpersonal
communication, leadership, and decision making in the cockpit.
Trained observers ride along in the cockpit and observe the
flight crews during normal flight operations. They score the
behaviors of the crew using the LOSA tool. LOSA has been
endored by the International Civil Aviation Organiztion
because it has been used so successful in measuring the
strengths and weaknesses of flight crews’ interpersonal skills
[16]. The Anesthetists’ NT Skills (ANTS) [17] used in
healthcare has proven very useful in assessing the NT skills of
anesthetists in simulation training and has provided important
performance feedback for the individuals. Another successful
healthcare BM system is the Observational Teamwork
Assessment of Surgery (OTAS). Many studies have shown that
poor communication, coordination, and other aspects of
teamwork, rather than technical failures, have been the primary
causes of adverse events in surgery. OTAS has been found to
be a valid measure of the NT performance of surgical teams
[18].
Our goal is to develop and validate a BM system that can
improve software professional team member performance by
providing feedback in the form of an objective and documented
assessment of the NT skills of the team members. We wanted
to create a tool that is very usable by practitioners: it requires
little or no training to use and does not require unreasonable
effort to use. It is a concern of the researcher that if the tool took
a lot of training or was too difficult to use, that the potential
practitioners, such as project managers and team leads for
whom the tool was meant to assist, would not find the tool
useful because of the amount of effort required.
III. B
EHAVIOR MARKER SYSTEM DEVELOPMENT
The development process for our behavioral marker system
for software developers is detailed in our previous work [19].
As a first step, we performed a systematic literature review to
develop NT skill inventory. The high-level question addressed
by the review was: What are the NT skills required of software
professionals performing well in their field and how can we
discover what NT skills are valued by employers?”
Details on the review protocol (sources searched, search
execution, inclusion and exclusion criteria, quality assessment,
data extraction) can be referred to in a report [20]. The output
of this step was an initial list of 35 NT skills that were clustered
into four major categories: communication, interpersonal,
problem solving, and work ethic (see Fig. 1). The detailed
desription of each skill can be referred [20].
During the second step, the initial list of NT skills had their
quality assessed and were validated by focus group of experts
in industry and academia. Two surveys (and focus groups) were
conducted online (using a cross sectional design) to gather NT
skill priorities, missing NT skills, description clarifications, and
examples of examples of good and poor behaviors for the top
rated NT skills of software developers. So that we could
prioritize our efforts, focus group ranked the importance of each
NT skill to software professionals during the first survey. After
the survey analysis, we had a reduced list of 16 skills to focus
on. During the second focus group survey, we gather a total of
408 examples of observable actions that indicated good
performance and behavior of each NT skill as well as examples
of observable actions that indicate poor performance and
behavior of each NT skill. These examples were reviewed,
clarified, and redundancies were eliminated. The final set of NT
skills consisted of: teamwork, initiative/motivation to work,
listening, attitude, critical thinking, oral communication,
problem solving, attention to detail, flexibility,
integrity/honesty/ethics, time management, and questioning.
Some behavioral examples, such asbeing a good team player
and “body language and persona emitting that you do not enjoy
your work, were too ambiguous and removed. It was also felt
that the “Leadership skill did not have enough observable
Fig. 1: Desired NT skills of Software Professionals
Fig. 2: Example ofListening” behaviors (good and bad examples)
410

behaviors that would be able to be clearly identified, so that NT
skill was removed. The result of the second survey was a
behavior-based software engineer NT skills taxonomy. Fig. 2
shows the resultant examples of good and poor behavior for the
Listening skill. The same process was used to create examples
of good and poor behavior for each NT skill.
During the third step, the behavior marker systems being
used in aviation, health care, rail transport and maritime
transport were examined. Each system’s structure was examined
to select which elements would have the most potential for use
in software development and our final tool was a composition of
several systems. The NT skills validated by the focus group
along with the good and bad behavior examples for those skills
were structured into a BM audit tool for software development.
For reference, we refer to the BM audit tool as the Non-
Technical Skill Assessment for Software Developers (NTSA).
The NTSA is designed to be used by an observer (i.e.
manager, team leader, coach) during routine team interactions or
meetings. It is intended that each time a behavior is observed, a
mark is placed in the appropriate column by placing a tick mark
in that column: observed and good, or expected but not observed.
Observations can be clarified by placing explanations in the
comments section. The observer can see skill definitions and
examples of good and poor behavior for a particular behavioral
marker by viewing the second page. A manager is allowed to
list as many or as few skills as desired in the behavioral marker
column. The observer will score the behaviors based on how
well the behavior meets the behavioral examples and its
definition.
IV. E
MPIRICAL VALIDATION OF BEHAVIOR MARKER
In order to evaluate our BM tool, an empirical study rated
video clips of student software development teams that were
working on industrial strength projects within the Software
Factory (as shown in Fig. 3 and explained).
1) Software Factory Background
The Software Factory is a software development laboratory
created by the University of Helsinki, Department of Computer
Science. All research was performed in Finland due to the
requirements of international privacy laws. The University of
Helsinki is consistently ranked in the top 100 out of world's
15,000 universities, in part because the university promotes
science and research together with European's top research-
intensive universities. The master’s degree programs are taught
in English in order to support the large number of international
students who study at the university. The Software Factory’s
primary participants are students, but the businesses provide
team members who work with the students, and university
faculties oversee the projects, although the faculty involvement
is kept to a minimum. Almost all project communication is in
English. Faculty involvement consists primarily of project
orientation and project intervention if problems cannot be
resolved by the students, coach, and customer. The coach is
generally an upper level student with Software Factory project
experience. University students take on the role of the
development team for projects provided by businesses. The
customer has company representatives that take on the role of
the product owner and represents the interests of the company.
Although these representatives are not co-located, they do come
by the Software Factory for weekly demos, sometimes for
meetings, and are generally available via telephone and email.
Researchers are able to observe what happens in the project due
to the seven cameras that provide multiple angles of view and
four microphones that record activities in the Factory room. In
Software Factory projects, the participants take on the core roles
of a typical Scrum project. Projects at the Software Factory last
for seven to eight weeks; the students work approximately 6
hours per day, 4-5 days per week.
2) Study Design
This study investigates whether the BM system can be used
with consistency by different raters to capture a measurement of
the NT skills of software developers, thus facilitating objective
feedback to software development teams and individuals. This
study used a blocked subject-project study. This type of analysis
allows the examination of several factors within the framework
of one study. Each of the non-technical skills to be studied can
be applied to a set of projects by several subjects and each
subject applies each of the non-technical skills under study.
In
this study, raters evaluated the NT skills of project teams using
the NTSA tool. The project teams worked together using state-
of-the-art tools, modern processes and best practices to
prototype and develop software for real business customers in an
environment that emulates industry. Video tapes of the projects
were evaluated to rate the student team’s NT skill performance.
The details of the study are provided as follows.
Independent and dependent variables:The experiment
manipulated the following independent variable:
a) Behavioral Marker System tool and Example
Behaviors: Each non-technical skill has its own set of good and
Fig. 3: Software Factory
411

Citations
More filters


25 Jun 2018
Abstract: Tecnologias digitais fazem parte do cotidiano de diferentes pessoas e sao utilizadas para muitas finalidades, inclusive para apoio ao ensino e a aprendizagem. O computador oferece recursos para o fazer pedagogico, como o software educacional. Seu desenvolvimento envolve trabalho em equipe e deve considerar aspectos pedagogicos e seu publico-alvo, ou seja, os estudantes. Assim como as criancas, atraves do Design Participativo, adolescentes podem contribuir, como parceiros de design, durante todo o processo de desenvolvimento de novas tecnologias. Ao se realizar uma revisao da literatura para encontrar contribuicoes para o desenvolvimento de software educacional com a participacao de adolescentes, constatou-se a importância de serem observados aspectos pedagogicos e sobre o conteudo em perspectiva, mas nao foram observadas contribuicoes para a criacao de softwares educacionais com a participacao de adolescentes. Assim sendo, este Trabalho de Conclusao de Curso integra praticas de Design Participativo a Engenharia de Requisitos no contexto educacional, com a participacao de adolescentes, para subsidiar o desenvolvimento de um software educacional para o ensino e a aprendizagem de conteudo de Matematica. Participaram estudantes de uma turma de oitavo ano e a professora de Matematica da Escola Estadual de Ensino Fundamental Dr. Arthur Hormain, localizada no Polo Rural dos Pinheiros – municipio de Alegrete/RS. Como resultado, tem-se a Especificacao de Requisitos do jogo Sorvete Turbo, que organiza seus requisitos funcionais e nao funcionais, alem dos prototipos gerados pelos adolescentes. A partir dos requisitos organizados, o jogo deve ser desenvolvido como parte das atividades do programa de extensao GEInfoEdu – Grupo de Estudos em Informatica na Educacao.

2 citations


Proceedings ArticleDOI
01 Dec 2017
TL;DR: Most required expertise and skills are identified and what type of skills should employees have according to their role in the organization are identified.
Abstract: Good skills and expertise of employees in an organization are the key factors for producing the quality products and completing projects successfully. Keeping the skills and knowledge of employees updated is one of the toughest and challenging tasks of the project management process. In developing countries like Pakistan, this task becomes more challenging due to lack of training and skill building institutes. The factors like changes in technology and increased demand for rapid development are addressed in this paper. Most required expertise and skills are identified. Our team has collected data by filling the questionnaires from IT professionals working on different projects in different organizations of the country. All expertise are divided into two main categories technical expertise or skills and non-technical / soft skills. Then characterized these skills according to the roles of employees. Finally, analysis of the whole data is performed by the statistical software like the SPSS and the Minitab, to find out which are the significant expertise and skills which really affects the performance of software organizations. We also identified what type of skills should employees have according to their role in the organization. The information developed by conducting this study is useful for many stakeholders in the local perspective, including professionals, project managers, and the Pakistan Software Export Board.

1 citations


Journal ArticleDOI
Abstract: Context: In the light of the swift and iterative nature of Agile Software Development (ASD) practices, establishing deeper insights into capability measurement within the context of team formation is crucial, as the capability of individuals and teams can affect team performance and productivity Although a former Systematic Literature Review (SLR) synthesized the state of the art in relation to capability measurement in ASD – with a focus on selecting individuals to agile teams, and capabilities related to team performance, productivity and success determining to what degree the SLR’s results apply to practice can provide progressive insights to both research and practice. Objective: Our study investigates how agile practitioners perceive the relevance of individual and team level measures for characterizing the capability of an agile team and its members. Here, the emphasis was also on selecting individuals to agile teams, and capabilities associated with effective teams in terms of their performance, productivity and success. Furthermore, to scrutinize variations in practitioners’ perceptions, our study further analyzes perceptions across stratified demographic groups. Method: We undertook a Web-based survey using a questionnaire built based on the capability measures identified from a previously conducted SLR. Results: Our survey responses (60) indicate that 127 individual and 28 team capability measures were considered as relevant by the majority of practitioners. We also identified seven individual and one team capability measure that have not been previously characterized by our SLR. The surveyed practitioners suggested that an agile team member’s responsibility and questioning skills significantly represent the member’s capability. Conclusion: Results from our survey align with our SLR’s findings. Measures associated with social aspects were observed to be dominant compared to technical and innovative aspects. Our results can support agile practitioners in their team composition decisions.

1 citations


Posted Content
TL;DR: In the light of swift, incremental and iterative nature of Agile Softwar, the role of human-aspects of software engineers needs to be considered in the context of agile development.
Abstract: Context: In the light of the swift and iterative nature of Agile Software Development (ASD) practices, establishing deeper insights into capability measurement within the context of team formation is crucial, as the capability of individuals and teams can affect team performance and productivity. Although a former Systematic Literature Review (SLR) synthesized the state of the art in relation to capability measurement in ASD with a focus on selecting individuals to agile teams, and capabilities related to team performance and success, determining to what degree the SLR's results apply to practice can provide progressive insights to both research and practice. Objective: Our study investigates how agile practitioners perceive the relevance of individual and team level measures for characterizing the capability of an agile team and its members. Furthermore, to scrutinize variations in practitioners' perceptions, our study further analyzes perceptions across stratified demographic groups. Method: We undertook a Web-based survey using a questionnaire built based on the capability measures identified from a previously conducted SLR. Results: Our survey responses (60) indicate that 127 individual and 28 team capability measures were considered as relevant by the majority of practitioners. We also identified seven individual and one team capability measure that have not been previously characterized by our SLR. The surveyed practitioners suggested that an agile team member's responsibility and questioning skills significantly represent the member's capability. Conclusion: Results from our survey align with our SLR's findings. Measures associated with social aspects were observed to be dominant compared to technical and innovative aspects. Our results can support agile practitioners in their team composition decisions.

References
More filters

Journal ArticleDOI
Abstract: In this article, we summarize and review the research on teams and groups in organization settings published from January 1990 to April 1996. The article focuses on studies in which the dependent variables are concerned with various dimensions of effectiveness. A heuristic framework illustrating recent trends in the literature depicts team effectiveness as a function of task, group, and organization design factors, environmental factors, internal processes, external processes, and group psychosocial traits. The review discusses four types of teams: work, parallel, project, and management. We review research findings for each type of team organized by the categories in our heuristic framework. The article concludes by comparing the variables studied for the different types of teams, highlighting the progress that has been made, suggesting what still needs to be done, summarizing key leamings from the last six years, and suggesting areas for further research.

3,428 citations


Journal ArticleDOI
TL;DR: The effects of working in an agile style is described and the problem it addresses and the way in which it addresses the problem are introduced.
Abstract: In a previous article (2001), we introduced agile software development through the problem it addresses and the way in which it addresses the problem. Here, we describe the effects of working in an agile style.

1,018 citations


Journal ArticleDOI
TL;DR: The findings of the evaluation indicated that the ANTS system has a satisfactory level of validity, reliability and usability in an experimental setting, provided users receive adequate training.
Abstract: Background. Non-technical skills are critical for good anaesthetic practice but are not addressed explicitly in normal training. Realization of the need to train and assess these skills is growing, but these activities must be based on properly developed skills frameworks and validated measurement tools. A prototype behavioural marker system was developed using human factors research techniques. The aim of this study was to conduct an experimental evaluation to establish its basic psychometric properties and usability. Method. The Anaesthetists’ Non-Technical Skills (ANTS) system prototype comprises four skill categories (task management, team working, situation awareness, and decision making) divided into 15 elements, each with example behaviours. To investigate its experimental validity, reliably and usability, 50 consultant anaesthetists were trained to use the ANTS system. They were asked to rate the behaviour of a target anaesthetist using the prototype system in eight videos of simulated anaesthetic scenarios. Data were collected from the ratings forms and an evaluation questionnaire. Results. The results showed that the system is complete, and that the skills are observable and can be rated with acceptable levels of agreement and accuracy. The internal consistency of the system appeared sound, and responses regarding usability were very positive. Conclusions. The findings of the evaluation indicated that the ANTS system has a satisfactory level of validity, reliability and usability in an experimental setting, provided users receive adequate training. It is now ready to be tested in real training environments, so that full guidelines can be developed for its integration into the anaesthetic curriculum.

810 citations


Book
01 Jan 2008
Abstract: Many 21st century operations are characterised by teams of workers dealing with significant risks and complex technology, in competitive, commercially-driven environments. Informed managers in such sectors have realised the necessity of understanding the human dimension to their operations if they hope to improve production and safety performance. While organisational safety culture is a key determinant of workplace safety, it is also essential to focus on the non-technical skills of the system operators based at the 'sharp end' of the organisation. These skills are the cognitive and social skills required for efficient and safe operations, often termed Crew Resource Management (CRM) skills. In industries such as civil aviation, it has long been appreciated that the majority of accidents could have been prevented if better non-technical skills had been demonstrated by personnel operating and maintaining the system. As a result, the aviation industry has pioneered the development of CRM training. Many other organisations are now introducing non-technical skills training, most notably within the healthcare sector. Safety at the Sharp End is a general guide to the theory and practice of non-technical skills for safety. It covers the identification, training and evaluation of non-technical skills and has been written for use by individuals who are studying or training these skills on CRM and other safety or human factors courses. The material is also suitable for undergraduate and post-experience students studying human factors or industrial safety programmes.

771 citations


Journal ArticleDOI
TL;DR: Personality type analysis can help take the guesswork out of putting together a high-performance software project team.
Abstract: Personality type analysis can help take the guesswork out of putting together a high-performance software project team.

256 citations


Frequently Asked Questions (1)
Q1. What are the contributions in "A behavior marker tool for measurement of the non- technical skills of software professionals: an empirical investigation" ?

The purpose of this research is to develop and validate the behavior marker system tool that can be used by different managers or coaches to measure the NT skills of software development individuals and teams. This paper presents an empirical study conducted at the Software Factory where users of the behavior marker tool rated video clips of software development teams.