Delft University of Technology
Questions for Data Scientists in Software Engineering: A Replication
Huijgens, Hennie; Rastogi, Ayushi; Mulders, Ernst; Gousios, Georgios; Deursen, Arie van
DOI
10.1145/3368089.3409717
Publication date
2020
Document Version
Accepted author manuscript
Published in
Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and
Symposium on the Foundations of Software Engineering
Citation (APA)
Huijgens, H., Rastogi, A., Mulders, E., Gousios, G., & Deursen, A. V. (2020). Questions for Data Scientists
in Software Engineering: A Replication. In P. Devanbu, M. Cohen, & T. Zimmermann (Eds.),
Proceedings of
the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the
Foundations of Software Engineering
(pp. 568–579). (ESEC/FSE 2020). Association for Computing
Machinery (ACM). https://doi.org/10.1145/3368089.3409717
Important note
To cite this publication, please use the final published version (if applicable).
Please check the document version above.
Copyright
Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent
of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.
Takedown policy
Please contact us and provide details if you believe this document breaches copyrights.
We will remove access to the work immediately and investigate your claim.
This work is downloaded from Delft University of Technology.
For technical reasons the number of authors shown on this cover page is limited to a maximum of 10.
estions for Data Scientists in Soware Engineering:
A Replication
Hennie Huijgens
Delft University of Technology
Delft, The Netherlands
h.k.m.huijgens@tudelft.nl
Ayushi Rastogi
Ernst Mulders
∗
Delft University of Technology
Delft, The Netherlands
a.rastogi@tudelft.nl
ernst@mulde.rs
Georgios Gousios
Arie van Deursen
Delft University of Technology
Delft, The Netherlands
g.gousios@tudelft.nl
arie.vandeursen@tudelft.nl
ABSTRACT
In 2014, a Microsoft study investigated the sort of questions that
data science applied to software engineering should answer. This re-
sulted in 145 questions that developers considered relevant for data
scientists to answer, thus providing a research agenda to the com-
munity. Fast forward to ve years, no further studies investigated
whether the questions from the software engineers at Microsoft
hold for other software companies, including software-intensive
companies with dierent primary focus (to which we refer as
software-dened enterprises). Furthermore, it is not evident that
the problems identied ve years ago are still applicable, given the
technological advances in software engineering.
This paper presents a study at ING, a software-dened enter-
prise in banking in which over 15,000 IT sta provides in-house
software solutions. This paper presents a comprehensive guide of
questions for data scientists selected from the previous study at
Microsoft along with our current work at ING. We replicated the
original Microsoft study at ING, looking for questions that impact
both software companies and software-dened enterprises and con-
tinue to impact software engineering. We also add new questions
that emerged from dierences in the context of the two companies
and the ve years gap in between. Our results show that software
engineering questions for data scientists in the software-dened
enterprise are largely similar to the software company, albeit with
exceptions. We hope that the software engineering research com-
munity builds on the new list of questions to create a useful body
of knowledge.
CCS CONCEPTS
• General and reference → Surveys and overviews.
KEYWORDS
Data Science, Software Engineering, Software Analytics.
∗
Work completed during an internship at ING.
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for prot or commercial advantage and that copies bear this notice and the full citation
on the rst page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specic permission and/or a
fee. Request permissions from permissions@acm.org.
ESEC/FSE ’20, November 8–13, 2020, Virtual Event, USA
© 2020 Association for Computing Machinery.
ACM ISBN 978-1-4503-7043-1/20/11.. . $15.00
https://doi.org/10.1145/3368089.3409717
ACM Reference Format:
Hennie Huijgens, Ayushi Rastogi, Ernst Mulders, Georgios Gousios, and Arie
van Deursen. 2020. Questions for Data Scientists in Software Engineering: A
Replication. In Proceedings of the 28th ACM Joint European Software Engineer-
ing Conference and Symposium on the Foundations of Software Engineering
(ESEC/FSE ’20), November 8–13, 2020, Virtual Event, USA. ACM, New York,
NY, USA, 21 pages. https://doi.org/10.1145/3368089.3409717
1 INTRODUCTION
Software engineering researchers try solving problems that are
relevant to software developers, teams, and organizations. Histori-
cally, researchers identied these problems from their experience,
connections in industry and/or prior research. In 2014, however, a
study at Microsoft [
7
] systematically analyzed software engineer-
ing questions that data scientists can answer and made it accessible
to a wider audience.
Switching context, in the past few years ING transformed it-
self from a nance-oriented company to a software-dened, data-
driven enterprise. From a software engineering perspective, this
includes the implementation of fully automated release engineer-
ing pipelines for software development activities in more than 600
teams performing 2,500+ deployments per month for 750+ appli-
cations. These activities leave a trove of data, suggesting that data
scientists using, e.g., modern machine learning techniques could
oer valuable and actionable insights to ING.
To that end, ING needs questions that are relevant for their
engineers which their data scientists can answer. As we started
looking for existing resources, we came across the 145 software
engineering questions for data scientists presented in the Microsoft
study [7]. However, before adopting the list, we wanted to know:
RQ: To what extent do software engineering questions relevant for
Microsoft apply to ING, ve years later?
Microsoft is a large software company, while ING that is a Fin-
Tech company using software to improve its banking solutions
(software-dened enterprise). Moreover, the two companies are at
dierent scale. In 2014, Microsoft had more than 30,000 engineers
while even today ING is almost half its size with approximately
15,000 IT employees (on a total of 45,000). More details on the
dierences in the context of the two companies are available in
Table 1. We try to understand whether the questions relevant for
a software company extend to a software-dened enterprise. We
compare the results of the original Microsoft study [
7
] with our
results at ING to understand the relevance of the questions beyond
Microsoft but also as a guide for other software-dened enterprises
that are undergoing their digital transformation. We further explore
ESEC/FSE ’20, November 8–13, 2020, Virtual Event, USA Hennie Huijgens, Ayushi Rastogi, Ernst Mulders, Georgios Gousios, and Arie van Deursen
whether the technological advances in the last ve years changed
the way we develop software. To answer this question, we carried
out a replication of the original Microsoft study at ING. Similar
to the original study, we conducted two surveys: one, to nd data
science problems in software engineering, and second, to rank the
questions in the order of their relevance (see Figure 1). For the rst
survey, we randomly sampled 1,002 ING engineers and received
116 responses with 336 questions. We grouped the 336 questions on
similarities resulting in 171 descriptive questions. We shared sub-
sets of these 171 descriptive questions with another random sample
of 1,296 ING engineers for ranking. In the end, we received 21,888
rankings from 128 ING engineers. These ranked 171 questions are
the questions that engineers at ING would like data scientists to
solve. Further, we compare our list of 171 questions to the original
list of 145 questions to answer our research question. Our study
shows that the core software development problems, relating to
code (e.g. understanding code, testing, and quality), developer pro-
ductivity (both individuals and team) and customer are same for the
software company and the software-dened enterprise. Nonethe-
less, subtle dierences in the type of questions point to changes in
market as well as dierences in the context of the two companies.
2 IMPACT OF THE MICROSOFT 2014 STUDY
In order to gain a good insight into the further course of the Mi-
crosoft 2014 study after it was published, including any implica-
tions for research, we conducted a citation analysis. In addition,
we looked at studies that have not quoted the Microsoft study, but
that are relevant to our study. Hence this section also serves as our
discussion of related work. We investigated the 136 studies that,
according to Google Scholar, quote the Microsoft study. First of all,
we looked at the number of times that the 136 studies themselves
were cited by other studies; we limited the further analysis to 70
studies with a citation per year greater than 1.00. We then charac-
terized studies into empirical approach, reference characterization,
SE topic, and machine learning (ML) topic (see Table 2). Note that
one paper can belong to multiple topics. We made the following
observations:
Microsoft itself is building on its study. 11% of the citations come
from Microsoft studies itself, mostly highly cited studies on SE
culture, such as [
18
,
41
,
51
]. we notice that all citing Microsoft
studies use a survey among a large number of SE practitioners
(ranging from 16 to 793 respondents with a median of 311), whereas
other studies based on a survey generally reach substantially lower
numbers of participants.
Table 1: Context of Microsoft in 2014 and ING in 2019.
Microsoft 2014 ING 2019
Branch Software Company Banking (FinTech)
Organization Size
Approx. 100,000 (in 2014),
about 30,000 engineers
45,000 employees of
which 15,000 IT
Team Structure Typically size 5 ± 2 600 teams of size 9 ± 2
Development Model Agile/Scrum (60%+) Agile (Scrum / Kanban)
Pipeline automation
Every team is dierent.
Continuous Integration
in many teams
Continuous Delivery as a
Service
Development Practice
DevOps (Biz)DevOps
Table 2: Characterizations of Citing Studies.
Empirical Approach (n = 70)
Number of studies
Percentage
Analysis of SE process data (e.g. IDE) 30 43%
Survey SE practitioners 17 24%
Interview SE practitioners 7 10%
Literature review 5 7%
Experiment, case, or eld study 5 7%
Reference characterization (n = 70)
Number of studies
Percentage
Plain reference in related work 38 54%
Reference as example for study setup 27 39%
Study partly answers MS question 9 13%
Study explicitly answers MS question 3 4%
Software Engineering Topic (n = 70)
Number of studies
Percentage
Software analytics, data science 20 29%
Testing, debugging, quality, code review 15 21%
Software engineering process 12 17%
Software engineering culture 9 13%
Mobile apps 3 4%
Machine Learning Topic (n = 24)
Number of studies
Percentage
Examples of Machine Learning applications 8 11%
Natural Language Processing 5 7%
Ensemble Algorithms 3 4%
Instance-based Algorithms 2 3%
Deep Learning Algorithms 2 3%
Other 4 5%
Half of the citing studies analyze SE process data, and 24% uses a
survey. Looking at the empirical approach (see the rst sub-table
in Table 2), indicates that 43% of the studies contain a quantitative
component, in which analysis of SE process data in particular is part
of the study. Good examples are [
9
,
28
]. Furthermore, 24% of the
citing studies uses a survey among SE practitioners, for example
[
18
,
22
,
45
,
69
,
75
]. Ten percent is based on interviews with SE
practitioners, such as [
20
,
41
,
42
,
50
]. Seven percent contains a
literature review, for example [
12
,
45
,
73
]. Another 7% conducts an
experiment [33, 62], case study [49, 59], or eld study [9, 10].
Only three out of 70 studies explicitly answer a question from the
initial Microsoft study. The second sub-table in Table 2 shows that
only 3 studies (4%) explicitly refer their research question to an
initial Microsoft one: [
16
,
28
,
33
]. Nine studies (13%) partly try to
answer a MS question: [
8
–
10
,
30
,
52
,
62
,
64
,
65
,
70
]. 29 studies (39%)
refer to the original Microsoft study because they used it as an
example for their own study [
17
,
59
], either with regard to the
study design [
20
,
22
,
29
,
37
,
46
,
48
,
67
], the rating approach (Kano)
[
51
,
61
], or the card sorting technique [
19
,
54
,
60
,
63
]. Furthermore,
a large part (38 studies, 54%) of the citing studies simply refers to
the original Microsoft study in a simple related work way.
A majority of citing studies is about Software Analytics, Testing
related studies, and SE Process. The third sub-table shows that most
cited studies are about software analytics, often combined with
a focus on the role of the software engineer and its perceptions,
e.g. [
42
,
51
]. In other cases the emphasis on software analytics is
estions for Data Scientists in Soware Engineering: A Replication ESEC/FSE ’20, November 8–13, 2020, Virtual Event, USA
combined with a more technical focus on machine learning, e.g.
[
21
,
48
]. Other studies within the topic software analytics are about
a variety of methods, tools, and techniques [
2
,
3
,
11
,
14
,
15
,
27
,
38
,
47
,
55
,
71
–
73
]. Many of the studies that cite the Microsoft study—
and which are often quoted themselves—relate to testing or test
automation. Fifteen studies (21%) are about testing [
8
–
10
,
13
,
23
,
24, 33, 45, 66], debugging [80] and code review [25, 46].
12 studies (17%) handle SE process related topics, such as produc-
tivity of software engineers [
52
], visualization [
6
,
31
], and continu-
ous delivery [
74
,
76
]. In addition, studies also relate to continuous
delivery pipelines and pipeline automation [
74
,
78
]. Another fre-
quent topic in citing studies is data and models, including aspects
of cloud development [
32
,
49
,
55
]. Driven by a tendency toward
automation of pipelines, software generates a large amount of data.
Many dierent data sources—such as version control systems, peer
code review systems, issue tracking systems, mail archives—are
available for mining purposes [29, 79].
34% of the cited studies includes some form of Machine Learning.
One third of the citing papers do include some form of machine
learning (ML), ranging from applying a ML technique for analysis
purposes to coming up with examples of the application of ML
in practice. As the fourth sub-table in Table 2 shows, 8 studies
include examples of applications of ML in practice, e.g. [
11
,
41
,
55
].
Text related techniques such as NLP occur 5 times, e.g. [
23
,
61
],
ensemble techniques 3 times [
30
,
37
,
60
], and instance-based and
deep learning both 2 times [
14
,
21
,
27
,
48
]. Four other techniques—
neural networks, clustering, decision trees, and regression—occur
one time. Perhaps this nding supports a trend that is visible in
SE research, where more and more machine learning techniques
are being used in SE analyzes and vice versa, also called AI-for-
Software-Engineering [1, 40, 53].
13% are about the cultural aspects of software engineering. Soft-
ware analytics is an area of extensive growth [
56
]. The original
Microsoft 2014 study inuenced ongoing research, looking at the
136 papers citing it gives the impression that it certainly did inspire
other researchers and practitioners in setting up studies on software
developers needs. Nine studies (13%) of the citing studies are about
cultural aspects of software engineering, such as topic selection in
experiments [
58
], characteristics of software engineers [
20
,
50
,
67
],
causes for frustration [
19
], or challenges for software engineers
[29, 63, 69].
3 STUDY DESIGN
Our study design comprises of two parts. In part one, we replicate
the original Microsoft study at ING. We follow the step-by-step
procedure prescribed in the original study, with slight modications
appropriate for our contextFigure 1 depicts the research methodol-
ogy we followed; the gure is an exact copy of the approach used in
the original Microsoft 2014 study with numbers from our study. In
the next step, we compare the questions identied in the Microsoft
study to ours for similarities and dierences including addition of
new questions and removal of previous questions to answer our
research questions.
This gure is a copy from the original Microsoft 2014 study, with numbers from our
study. The gure was re-used with permission of the Microsoft 2014 study authors.
Figure 1: Overview of the research methodology
3.1 The Initial Survey
We sent the initial survey to 1,002 ING software engineers randomly
chosen from a group of 2,342 employees working within the IT
department of ING in May 2018. Unlike the Microsoft study, we did
not oer any reward to increase the participation. This is a deviation
from the original study but aligns with the policy of ING. Out of the
1,002 engineers 387 engineers started the survey, 271 of them even
lled the demographics but stopped when asked to write questions.
In the end, we received 336 questions from 116 responses for a
response rate of 11.6%. Table 3 shows the distribution of responses
across discipline and role.
3.2 Coding and Categorization
Next we did an open card sort to group 336 questions into categories.
Our card sort was open, meaning that we coded independently
from the Microsoft study. To create independent codes, the rst
author who did a majority of the coding did not study the Microsoft
paper before or during the replication. The other authors knew the
paper from before and merely skimmed the methodology section
for replication.
We let the groups emerge and evolve during the sorting process.
This process comprised of three phases. In preparation phase, we
created a card for each question. Questions 1 to 40 were tagged by
ESEC/FSE ’20, November 8–13, 2020, Virtual Event, USA Hennie Huijgens, Ayushi Rastogi, Ernst Mulders, Georgios Gousios, and Arie van Deursen
the second author. Questions 41 to 80 were tagged by the fourth
author. Questions 81 to 90 were tagged by both the second and
the fourth author. The tags of questions 1 to 90 were discussed by
both the second and fourth author and based on their discussion
nal tags were prepared. The remaining questions 91 to 336 were
then tagged by the rst author, based on the tags from the previous
step. We discarded cards that made general comments on software
development and did not inquire any specic topic.
In the execution phase, cards were sorted into meaningful groups
and were assigned a descriptive title. Similar to the Microsoft study,
the questions were not easy to work with; many questions were
same or similar to one another, most were quite verbose while oth-
ers were overly specic. We distilled them into a set of so-called
descriptive questions that more concisely describe each category
(and sub-category). In this step, out of the 336 questions, 49 ques-
tions were discarded and the remaining 287 questions were divided
into 35 sub-categories. An example of reaching descriptive question
is presented below
1
:
‘What factors aect the composition of DevOps teams?’
from the following respondents’ questions:
7
“Would it be better to create specialized development teams instead
of DevOps teams?"
7
"What is your idea of an ideal team that should develop software?
How many and what kind of people should be part of it?"
Finally, in the analysis phase, we created abstract hierarchies
to deduce general categories and themes. In total, we created 171
descriptive questions, a full list of which is available in the appendix.
3.3 The Rating Survey
We created a second survey to rate the 171 descriptive questions.
We split the questionnaire into eight component blocks (similar
to the Microsoft study) and sent component blocks to potential
respondents. The idea behind using the split questionnaire survey
design is to avoid low response rate. Each participant received a
block of questions along with a text "In your opinion, how important
is it to have a software data analytics team answer this question?"
with possible answers as "Essential", "Worthwhile", "Unimportant",
"Unwise", and "I don’t understand" [39].
1
A closed balloon indicates a respondent question; an open balloon indicates a descrip-
tive question.
Table 3: Distribution of responses based on discipline and
role in the initial survey as well as rating survey.
Discipline Initial Survey Rating Survey
Development & Testing 62.0% 68.8%
Project Management 2.0% 3.9%
Other Engineering (e.g. architect) 28.0% 19.5%
Non-Engineering 8.0% 7.8%
Current Role Initial Survey Rating Survey
Developer 51.1% 20.0%
Lead 14.3% 18.7%
Architect 9.0% 11.8%
Manager & Executive 8.3% 20.0%
Other 17.3% 29.6%
The rating survey was sent to the remaining 1,296 software
engineers at ING. Here too, 360 engineers started the survey (28%),
but many of them did not complete it (36% drop-out rate). Finally,
we received 128 responses, for a somewhat low response rate of
10%. On an average each question received 21,888/177=123 ratings
making the resulting ranks stable. Table 3 shows the distribution
of responses for the rating survey based on discipline and current
role.
3.3.1 Top-Rated/Boom-Rated estions. Finally, to rank each
question, we dichotomized the ordinal Kano scale avoiding any
scale violations [
44
]. We computed the following percentages for
each descriptive question:
•
Percentage of ’
Essential
’ responses among all the responses:
Essential
Essential + Worthwhile + Unimportant + Unwise
•
Percentage of ’Essential’ and ’Worthwhile’ responses among
all the responses (to which we refer as Worthwhile+):
Essential + W orthwhile
Essential + W orthwhile + Unimportant + Unwise
•
Percentage of ’
Unwise
’ responses among all the responses:
U nw ise
Essential + W orthwhile + Unimportant + Unwise
We rank each question based on the above percentages, with
the top rank (#1) having the highest percentage in a dimension
(Essential, Worthwhile+, or Unwise). Table 5 and Table 6 presents
the most desired (Top 10 Essential, Top 10 Worthwhile+) and the
most undesired (Top 10 Unwise) descriptive questions. For all 171
questions and their rank, see the appendix.
3.3.2 Rating by Demographics. Unlike the Microsoft study, we did
not have employee database to rank responses based on demograph-
ics, and privacy regulations prevented us from asking people-related
aspects such as years of experience (another deviation from the
original study). Nonetheless, in both the initial and the rating sur-
vey, we asked the following professional background data from the
participants:
•
Discipline: Participants were asked to indicate their primary
working area: Development, Test, Project Management, Other
Engineer (e.g. architect, lead), or Other Non-Engineer (only
one selection was possible).
•
Current Role: Participants were asked to indicate their cur-
rent role: Individual Contributor, Lead, Architect, Manager,
Executive, or Other (more selections were possible).
To investigate the relations of descriptive questions to profes-
sional background (discipline or current role), we built stepwise
logistic regression models. We build our own models since the refer-
enced study did not share scripts to run statistical tests although we
did follow their procedure as is. Stepwise regression eliminated pro-
fessional backgrounds that did not improve the model for a given
question and a response. In addition, we removed professional back-
grounds for which the coecient in the model was not statistically
signicant at p-value < 0.01. For each of the 171 questions, we built
a model with Essential response (yes/no) as a dependent variable