Software development in startup companies: A systematic mapping study

TL;DR: The results indicate that software engineering work practices are chosen opportunistically, adapted and configured to provide value under the constrains imposed by the startup context.

Abstract: Context: Software startups are newly created companies with no operating history and fast in producing cutting-edge technologies. These companies develop software under highly uncertain conditions, tackling fast-growing markets under severe lack of resources. Therefore, software startups present a unique combination of characteristics which pose several challenges to software development activities. Objective: This study aims to structure and analyze the literature on software development in startup companies, determining thereby the potential for technology transfer and identifying software development work practices reported by practitioners and researchers. Method: We conducted a systematic mapping study, developing a classification schema, ranking the selected primary studies according their rigor and relevance, and analyzing reported software development work practices in startups. Results: A total of 43 primary studies were identified and mapped, synthesizing the available evidence on software development in startups. Only 16 studies are entirely dedicated to software development in startups, of which 10 result in a weak contribution (advice and implications (6); lesson learned (3); tool (1)). Nineteen studies focus on managerial and organizational factors. Moreover, only 9 studies exhibit high scientific rigor and relevance. From the reviewed primary studies, 213 software engineering work practices were extracted, categorized and analyzed. Conclusion: This mapping study provides the first systematic exploration of the state-of-art on software startup research. The existing body of knowledge is limited to a few high quality studies. Furthermore, the results indicate that software engineering work practices are chosen opportunistically, adapted and configured to provide value under the constrains imposed by the startup context.

Summary

1. Introduction

• A wide body of knowledge has been created in recent years through several empirical studies, investigating how companies leverage software engineering (SE) [1, 2].
• Startups typically aim to create high-tech and innovative products, and grow by aggressively expanding their business in highly scalable markets.
• From a software engineering perspective startups are unique, since they develop software in a context where processes can hardly follow a prescriptive methodology [11].
• The systematic map consists of 43 primary studies that were identified from an initial set of 1057 papers.
• The remainder of this paper is structured as follows: Section 2 describes background and related work; Section 3 illustrates the research methodology and discusses validity threats; Section 4 introduces the classification schema, developed from the gathered data; Section 5 presents the results of the mapping study.

2. Background and Related Work

• Modern entrepreneurship, born more than thirty years ago [16], has been boosted by the advent of the consumer Internet markets in the middle of the nineties and culminated with the notorious dot-com bubble burst of 2000 [17].
• Today, with the omnipresence of the Internet and mobile devices, the authors are assisting to an impressive proliferation of software ventures - metaphorically referred as the startup bubble.
• In fact, easy access to potential markets and low cost of services distribution are appealing conditions for modern entrepreneurs [18].
• Inspired by success stories, a large number of software businesses are created every day.
• The great majority of these companies fail within two years from their creation [8].

2.1. Software Startups

• An early account for the term startup in the SE literature can be found in Carmel [19] who studied in 1994 the time-tocompletion in a young package firm.
• Sutton [3] provides a characterization of software startups, defined by the challenges they are faced with: little or no operating history - startups have little accumulated experience in development processes and organization management.
• Multiple influences - pressure from investors, customers, partners and competitors impact the decision-making in a company.
• Overall they might be inconsistent.
• Dynamic technologies and markets - newness of software companies often require to develop or operate with disruptive technologies2 to enter into a high-potential target market.

2.2. Startup Lifecycle

• The lifetime of a startup company, from idea conception to the maturity level, has been identified and reported from different perspectives (e.g. market [37] and innovation [38]).
• A prominent contribution, from a SE viewpoint, is the model presented by Crowne [8], who synthesized the startup lifecycle in four stages.
• This time frame is characterized most from the need to assemble a small executive team with the necessary skills to start to build the product.
• The stabilization phase begins from the first sale, and it lasts until the product is stable enough to be commissioned to a new customer without causing any overhead on product development.
• Finally, the startup evolves to a mature organization, where the product development becomes robust and predictable with proven processes for new product inventions.

2.3. Software Development in Startups

• The implementation of methodologies to structure and control the development activities in startups is a major challenge for engineers [11].
• In general, the management of software development is achieved through the introduction of software processes, which can be defined as “the coherent set of policies, organizational structures, technologies, procedures, and artifacts that are needed to conceive, develop, deploy and maintain a software product” [21].
• Furthermore, startups have very limited resources and typically wish to use them to support product development instead of establishing processes [11, 24].
• Researchers emphasize the importance of time-tomarket as a key strategic objective [31, 32].
• Other peculiar aspects influencing software development in the market-driven context are related to requirements, which are reported to be often “invented by the software company” [33], “rarely documented” [34], and can be validated only after the product is released to market [35, 36].

2.4. Related work

• The prospects of evidence-based software engineering [39] have motivated researchers to conduct systematic literature reviews and mapping studies.
• Hence, reviews that study the literature on SMEs are relevant related work.
• This knowledge is best encoded in organized routines that allow widespread sharing within the firm.
• Rosenbusch et al. [50] studied the innovation-performance relationship in SMEs by conducting a meta-analysis of 42 empirical studies that cover 21,270 firms.
• Similarly, the authors argue that startups, differing from SMEs in terms of their operating history, outside influences and market dynamism, require software development solutions adapted to their context.

3. Research methodology

• The authors chose to perform a systematic mapping study (SMS), which is capable of dealing with wide and poorly-defined areas [15, 4].
• A systematic literature review (SLR) [4] would have been a less viable option due to the breadth of their overall research question:.
• The SMS procedure was led by the first and second authors, who performed the steps in Figure 1 in a co-located environment, i.e. working together on a single computer screen.
• Notetaking during screening of papers and keywording alleviated the resolution of conflicts among the reviewers during data extraction and rigor and relevance assessment.

3.1. Definition of Research Questions

• The research question driving this mapping study is:.
• To answer this question, the authors state the following sub-questions: RQ1.
• With RQ2 the authors intend to determine the scientific evidence of the results reported in the state-of-art, allowing researchers to identify worthwhile avenues for further work and providing practitioners a tool to navigate within the state-of-art.

3.2. Conduct Search

• The authors identified the primary studies by exercising a search string on scientific databases.
• The authors omitted however the outcome and context facet from the search string structure as their research questions do not warrant a restriction of the results to a particular outcome (e.g. effective/not effective work practices) or context (e.g. startups in a specific product domain).
• The core concepts, representing population, intervention and comparison, are derived from their research questions.
• The selected scientific databases on which the authors performed the search are shown in Table 2, along with the number of publications retrieved from each database (up to December 2013).
• Then, the authors proceeded to the customization of the search string, adapting the syntax to the particular database4.

3.3. Screening of Relevant Papers

• The criterion that guided the inclusion of a paper was that the study presented a contribution to the body of knowledge on software development in startups.
• A contribution can be in the form of an experience report, applied engineering practices, development models or lessons learned.
• For the screening of papers the authors followed the workflow in Figure 2.
• Then, the authors removed duplicated items in two steps: first they used the reference management tool to automatically detect duplicates based on meta-data (author, publication year and title).
• In a more in-depth review, the authors analyzed the abstract of each paper, determining whether it matched their inclusion criterion, resulting in 76 papers.

3.4. Keywording

• The goal of keywording is to efficiently create a classification schema, ensuring that all relevant papers are taken into account [15].
• The authors followed the process illustrated in Figure 3.
• The first step consisted in reading the abstracts of the primary studies, assigning them a set of keywords to identify the main contribution area of the paper.
• By progressively fitting the papers into categories, the schema underwent a refinement process, being continuously updated to account for new data.
• When performing data extraction and mapping (Subsection 3.5), the authors annotated the classification with evidence from the respective paper, further refining the schema and sorting.

3.5. Data Extraction and Mapping

• After the authors defined the classification schema, resulting from the keywording process, they proceeded to systematically extract data from the primary studies.
• The authors took advantage of the data extraction process to identify an additional relevant aspect which emerged while reading abstracts and the full text: the recurrent patterns of common attributes among startup companies resulted in themes that are reported in Subsection 5.2.
• Moreover, the authors screened the bibliography of each paper, identifying other possible relevant studies to their research, adopting the snowballing technique5 [59].
• 5Note that the authors didn’t identify any additional papers.
• In case more relevant papers would have been retrieved, a re-iteration of the keywording step would have been necessary (see Subsection 3.4).

3.6. Rigor and relevance assessment

• A major challenge of SE is to transfer research results and knowledge to practitioners, showing the findings’ validity and concrete advantages [51].
• The model provides a set of rubrics to measure rigor and relevance, dividing these two factors into different aspects, and quantifying the extent to which each aspect is considered in the study (see Ivarson and Gorschek [60] for an application of the model).
• The authors considered aspects relating to: Subjects - use of subjects who are representative of the intended users of the technology.
• The ranking function provides a rough estimation of the value that a paper provides to practitioners and the research community, giving a stronger weight to recent rigorous journal publications entirely devoted to the topic and presenting empirical results relevant to practitioners.
• The conversion tables used to quantify the internal score of each factor are shown in the supplementary material [55], while the limitations of this approach are discussed in Subsection 3.8.3.

3.7. Synthesis

• In the synthesis the authors identified the main concepts from each primary study, using the original author’s terms in a one line statement.
• Those main concepts were then organized in tabular form to enable comparisons across studies and translation of findings into higher-order working practices and classification categories.
• The authors used the classification categories from Section 4.
• This process is analogous to the method of constant comparison used in qualitative data analysis [52].
• In Section 7 the authors present the identified work practices, discussing their application in the startup context, their benefits and liabilities, and putting them in perspective with the results of other studies.

3.8. Threats to validity

• The authors identified potential threats to the validity of the systematic mapping and its results, together with selected mitigation strategies.
• The structure of this Section follows Unterkalmsteiner et al. [61].

3.8.2. Identification of primary studies

• The approach the authors used to construct the search string (see Subsection 3.2) aimed to be inclusive with respect to the number of retrieved papers, related to software development in startups.
• The authors were not able to retrieve 20 papers since they were neither available in online catalogs, in the three libraries they consulted, nor on request from the authors.
• This is a minor risk as the authors had access to their titles, keywords and abstracts, which gave us a good degree of confidence that they were not relevant.
• The authors noticed high inconsistency in the use of the term “startup” by different researchers, even in the same area.
• Under these conditions, the attempt to identify a body of knowledge and research scope has been highly challenging.

3.8.3. Study selection and data extraction

• Threats to study selection and data extraction [57] have been mitigated with an up-front definition of the inclusion/exclusion criteria [4].
• The selection of relevant primary studies can be further biased by personal opinions of researchers executing the process.
• To mitigate this threat, the authors defined and documented a rigid protocol for the study selection and, by conducting the selection together and dedicating a reasonable amount of time to review conflicts, mutually adjusting each others’ biases, as suggested by Kitchenham and Charters [4].
• The ranking itself gives an indication of the study quality, the individual contribution needs however to be qualified by the reader [51].
• For validating their ranking, the authors tried to modify scores/weights values several times, and they observed that the final ranking was not significantly altered by numerical adjustments, as long as they kept the ordering of concepts stable.

4. Classification schema

• In this section the authors present the classification schema that is adapted from other existing taxonomies or emerged from the keywording process.
• The research types were adapted from Wieringa [64].
• The contribution facet (Table 4b), similarly to the taxonomy used by Shaw [65], describes the kind of contribution a study provides.
• The authors separated thereby studies concerning software development practices (e.g. writing user stories [13]) from studies focused on higher-level process management (e.g. use Scrum methodology [66]).
• The pertinence facet (Table 4d) distinguishes the levels (full, partial, marginal) on which the study’s research focus is directed towards engineering activities in startups.

5. Results

• This section presents the results of the systematic mapping study.
• From an initial sample of 1057 papers, the authors identified 43 primary studies answering their research questions.

5.1. Startup research categorization

• Figure 4 shows the publication years’ frequency distribution, from 1994 to 2013.
• These formed the basis6 for the focus and pertinence facet of the classification schema (Section 4).
• Table 5 applies the classification schema on the primary studies, providing an overview of the field of startup research.
• Thus, the authors created three plots (Figures 5 - 7) to visualize all six possible facets combinations 6The raw data is provided in the supplementary material [55].
• In the same figure it is possible to observe that 8 studies with managerial and organizational focus contributed to the body of knowledge with a model.

5.2. Context characteristics of startups

• To illustrate how authors use the term “software startup”, the authors systematically extracted themes which characterize the companies in the selected primary studies.
• When discussing software startups, 18 authors reported a general lack of human, physical and economical resources (T1).
• Other studies refer to companies which are able to quickly react to changes in the market and technologies (T2), under remarkably uncertain conditions (T4).
• Furthermore, 14 authors write about startups as fast growing companies (T5) designed to rapidly scale-up.
• Finally, other studies agree on the highly risky nature of these businesses (T13), being newly created (T11) and therefore with no or little working history (T15).

5.3. Rigor and relevance

• Even though the scientific value of a study is not determined by the publication type, the peer-review process required for publishing a journal article is generally much more rigorous and formal than the procedure to get an article published in a scientific magazine or accepted to a conference [99].
• It can be interpreted as a first indicator of scientific quality.
• The authors formally assessed the quality of the primary studies with the rigor and relevance process described in Subsection 3.6, resulting in Figure 8 (the raw data for this figure is available in the supplementary material [55]).
• Twenty-one studies (49%) exhibit moderate industry relevance (relevance ≥ 2), showing however low scientific rigor (rigor ≤ 1.5).

6. Analysis of the state-of-art

• More than 65% of the 43 identified primary studies have been published in the last ten years (between 2004 and 2013, see 7The publication criteria are determined by the specific editor of the journal/magazine or the committee of a conference.
• The overall studies’ contribution types are for the greater part weak: advice and implication, lessons learned, tools and guidelines (27 studies, 63%, Figure 5).
• One can also observe in the same figure that 11 of these are related to managerial and organizational factors, and only 8 out of 21 have a full pertinence with engineering activities in software startups (Figure 7).the authors.
• To summarize the systematic map, the authors can state that Coleman and O’Connor [72, 11, 22], and Kajko-Mattson and Nikitina [13] represent the strongest contributions to the field of startup research, considering strength of contribution type, pertinence to engineering activities and strength of empirical evidence.

6.1. RQ1 - The context characterizing software development in startups

• The results (Table 6) indicate that there is no agreement on a standard definition, specifying the characteristics of a “startup”.
• Apparently, the definition is not strictly related to the size of the company.
• Uniqueness is not defined through the age of the company alone: some authors studied startups which have been operating for many years [79], while others are more strict and limit the definition to only recently founded companies [13].
• The most frequent reported themes concern the general lack of resources, high reactiveness and flexibility, innovation, uncertain conditions, time pressure and fast growth.
• Since the contextual boundaries of startups resulted to be highly blurred, it is the researchers’ responsibility who refer to “startups” to explicitly define the features of the company under study (e.g. company age, team size, product type, product development time).

6.2. RQ2 - Transferability of results to industry

• Figure 8 illustrates a major weakness of the state-of-art in startup research.
• Seven primary studies (16% of the total), received an average score for industrial relevance (2) but a low score (0) for scientific rigor.
• According to the authors of the rigor-relevance model [51], this represents a major threat to the transferability of the results to industry.
• The design of the ranking function considers their research question of identifying software engineering work practices in startups.
• The conversion tables to achieve a normalized score are available in the supplementary material [55].

7. RQ3 - Work Practices in startups

• The authors have extracted a total of 213 work practices8 from the 43 primary studies reviewed in this SMS and subsequently divided them in categories (Table 8), as explained in Subsection 3.7.
• The categorization of working practices is defined according to the focus facet of the classification schema, presented in Figure 4c.
• In the remainder of this section, the authors discuss the identified work practices, pointing out where gaps exist and further research is warranted.
• 8Note that this number does not reflect unique work practices but the total number; a detailed table of work practices is available in the supplementary material [55].

7.1. Process management practices

• Process management represents all the engineering activities used to manage product development in startups.
• Agile methodologies have been considered the most viable processes for software startups, given that Agile methodologies embrace changes rather than avoiding them, allowing development to follow the business strategy [93].
• This concurs with the practice of allocating varying effort for formalizing specifications, design, documentation and testing in tailored development methodologies [91, 82, 19], emphasizing the importance of minimal process management.
• Fast releases to build a prototype in an evolutionary fashion and quickly learn from the users’ feedback to address the uncertainty of the market.
• Therefore, any process tailored to the startup context needs at least to allow, but optimally even facilitate movements between complex and chaotic domains that are intrinsic in the innovation generation of startups.

7.2.1. Requirements Engineering practices

• Establishing an engineering process for collecting, defining and managing requirements in the startup context is challenging.
• RE practices are often reduced to some key basic activities [82, 8].
• Initially, as startups often produce software for a growing target market [29, 19], customers and final users are usually not well-known and requirements are therefore market-driven [90] rather than customer-specific.
• Moreover, in unexplored and innovative markets, the already poorly-defined requirements tend to change very rapidly.
• An example of a more strict customer-development process [105] that drives the identification of requirements can be found in an experience report by Taipale [93].

7.2.2. Design and Architecture practices

• Deias and Mugheddu [63] observed a general lack of written architecture and design specifications in startups.
• Development in startups is usually guided by simple principles, trying to avoid architectural constraints that are difficult to overcome as the product and user-base grows.
• Tinglings [80] results suggest that a not well analyzed architecture causes problems in the long run.
• A good-enough architecture analysis should at least identify the decisions that might cause problems before obtaining product revenue, and which can be fixed at a later point in time, accounting for increased resources after revenue cash starts to flow [92].
• Therefore, employing architectural practices and frameworks that enable easy extension of the design (e.g. pluggable architecture where features can be added and removed as plugins [108]) can better align the product to the uncertainty of the market needs.

7.2.3. Implementation, Maintenance and Deployment practices

• Silva and Kon [67] report on positive results from pairing up senior and junior developers together in pair-programming sessions.
• In a different study, Tingling [80] attested an initial high resistance to the introduction of coding standards and pair-programming.
• On the other hand, introducing refactoring may cause problems since developers had no or little experience [63] and they didn’t see immediate value in introducing high level abstractions [67].
• The process is often driven by lightweight ad-hoc systems: bug-tracking, simple code metrics and pair-programming sessions.
• Startups in the early stage keep the code base small and simple to develop only what is currently needed, maintaining focus on core functionalities to validate with first customers.

7.2.4. Quality Assurance practices

• Testing software is costly in terms of calendar time and often compromised in startups [19, 82].
• Quality assurance, in the broader sense, is largely absent because of the weak methodological management, discussed in Subsection 7.1.
• Mater and Subramanian [68] suggest to use a small group of early adopters or their proxies as quality assurance fit team.
• Outsourcing quality assurance to external software testing experts, handling the independent validation work if resources are not available [68], can also be an alternative.
• Even though testing software is costly, acceptance tests are the only practice to validate the effectiveness of the product in uncertain market conditions [29].

7.3. Managerial and organizational practices

• Flexibility, more than structure, plays an important role in startup companies [85, 96].
• To accommodate flexibility of managerial and organizational practices, the empowerment of team members represents the main viable strategy to enhance performance and chances of success [19, 78].
• Nevertheless, key performance indicators (e.g. customer attrition, cycle time) and continuous deployment processes can effectively assess the consumers’ demand using the least amount of resources possible [103].
• In building up a startup company, the team needs expertise to counterbalance the lack of resources [27, 22, 92].
• Moreover, Crowne suggests to plan organizational objectives in the short-medium term [8], measuring development cycle time to find areas for improvements [93, 13].

7.4. Tools and technologies

• Startups are often established to develop technologically innovative products, which in turn might require cutting-edge development tools and techniques.
• In general, startups take advantage of those technologies that can quickly change the product and its management [67, 8], avoiding conflicts with business strategic plans.
• Such technologies support the needed activities, accommodating changes when required [3].
• The use of easy-to-implement tools to work with fast-paced changing information.
• Lack of experience can also be a disadvantage which could be compensated by a light-weight process to select technologies; this selection could be guided by domain-specific or product specific requirements.

Figures

Table 6: Mapping Study - Recurrent themes

Figure 3: Workflow for classification schema creation (adapted from [15])

Table 3: Rationale for excluded papers

Figure 2: Screening of papers workflow

Figure 5: Systematic map - Focus, contribution and research type

Figure 6: Systematic map - Contribution, pertinence and research type

Figure 7: Systematic map - Pertinence, focus and research type

Table 9: Software development practices

Table 5: Systematic map overview

Table 7: Ranking of primary studies considering Pertinence, Rigor and Relevance, Publication Age and Type, Contribution type, Research type, and Focus

Table 8: Categorization of the identified work practices

Table 4: Classification schema

Figure 4: Publication distribution-year

Table 1: Population, intervention and comparison search string keywords

Figure 1: Systematic mapping process (adapted from Petersen et al. [15])

Table 2: Selected databases and retrieved papers

Full text

Electronic Research Archive of Blekinge Institute of Technology
http://www.bth.se/fou/
This is an author produced version of a journal paper. The paper has been peer-reviewed but
may not include the final publisher proof-corrections or journal pagination.
Citation for the published Journal paper:
Title:
Author:
Journal:
Year:
Vol.
Issue:
Pagination:
URL/DOI to the paper:
Access to the published version may require subscription.
Published with permission from:
In press: Software Development in Startup Companies: A Systematic Mapping Study
Nicolò Paternoster, Carmine Giardino, Michael Unterkalmsteiner, Tony Gorschek, Pekka
Abrahamsson
Information and Software Technology
2014
10.1016/j.infsof.2014.04.014
Elsevier

Software Development in Startup Companies: A Systematic Mapping Study
Nicol
`
o Paternoster
a
, Carmine Giardino
a
, Michael Unterkalmsteiner
a
, Tony Gorschek
a
, Pekka Abrahamsson
b
a
Blekinge Institute of Technology, SE-371 79 Karlskrona, Sweden
b
Free University of Bolzano-Bozen, I-39100 Bolzano-Bozen, Italy
Abstract
Context: Software startups are newly created companies with no operating history and fast in producing cutting-edge technologies. These
companies develop software under highly uncertain conditions, tackling fast-growing markets under severe lack of resources. Therefore, software
startups present an unique combination of characteristics which pose several challenges to software development activities. Objective: This
study aims to structure and analyze the literature on software development in startup companies, determining thereby the potential for technology
transfer and identifying software development work practices reported by practitioners and researchers. Method: We conducted a systematic
mapping study, developing a classification schema, ranking the selected primary studies according their rigor and relevance, and analyzing
reported software development work practices in startups. Results: A total of 43 primary studies were identified and mapped, synthesizing the
available evidence on software development in startups. Only 16 studies are entirely dedicated to software development in startups, of which 10
result in a weak contribution (advice and implications (6); lesson learned (3); tool (1)). Nineteen studies focus on managerial and organizational
factors. Moreover, only 9 studies exhibit high scientific rigor and relevance. From the reviewed primary studies, 213 software engineering work
practices were extracted, categorized and analyzed. Conclusion: This mapping study provides the first systematic exploration of the state-of-art
on software startup research. The existing body of knowledge is limited to a few high quality studies. Furthermore, the results indicate that
software engineering work practices are chosen opportunistically, adapted and configured to provide value under the constrains imposed by the
startup context.
Keywords: Software Development, Startups, Systematic Mapping Study
1. Introduction
A wide body of knowledge has been created in recent years
through several empirical studies, investigating how companies
leverage software engineering (SE) [1, 2]. However, research
on software development activities in newly created compa-
nies is scarce. In the past, very few publications have identi-
fied, characterized and mapped work practices in software star-
tups [3] and no structured investigation of the area has been per-
formed. Indeed, none of the systematic literature reviews [4] or
mapping studies [5] in software engineering (see the tertiary re-
view by Zhang and Babar [6]) address the startup phenomenon.
Understanding how startups take advantage from work prac-
tices is essential to support the number of new businesses
launched everyday
1
. New software ventures such as Facebook,
Linkedin, Spotify, Pinterest, Instagram, and Dropbox, to name a
few, are examples of startups that evolved into successful busi-
nesses. Startups typically aim to create high-tech and innova-
tive products, and grow by aggressively expanding their busi-
ness in highly scalable markets.
Despite many successful stories, self-destruction rather than
competition drives the majority of startups into failure within
two years from their creation [8]. Software startups face intense
time-pressure from the market and are exposed to tough com-
petition, operating in a chaotic, rapidly evolving and uncertain
1
According to a recent study, solely in the US “startups create an average
of 3 million new jobs annually” [7].
context [9, 10]. Choosing the right features to build and adapt-
ing quickly to new requests, while being constrained by limited
resources, is crucial to the success in this environment [3].
From a software engineering perspective startups are unique,
since they develop software in a context where processes can
hardly follow a prescriptive methodology [11]. Startups share
some characteristics with other contexts such as small compa-
nies and web engineering, and present a combination of dif-
ferent factors that make the development environment different
from established companies [12]. Therefore, research is needed
to support startups’ engineering activities, guiding practitioners
in taking decisions and avoiding choices that could easily lead
business failure [13, 14].
The goal of this paper is to identify and understand the main
contributions of the state-of-art towards software engineering in
startups. To this end, we perform a systematic mapping study
(SMS) [5, 15] aimed at:
• characterizing the state-of-art research on startups
• understanding the context that characterizes startups
• determining the potential for technology transfer of the state-
of-art research on startups
• extracting and analyzing software development work prac-
tices used in startups
The systematic map consists of 43 primary studies that were
identified from an initial set of 1057 papers. Practitioners may
take advantage of the 213 identified software engineering work
practices, while considering however the studies’ respective
Preprint submitted to Information and Software Technology May 5, 2014

rigor and relevance assessments. Furthermore, this first system-
atic exploration on software startups provides researchers with
directions for future work.
The remainder of this paper is structured as follows: Sec-
tion 2 describes background and related work; Section 3 illus-
trates the research methodology and discusses validity threats;
Section 4 introduces the classification schema, developed from
the gathered data; Section 5 presents the results of the mapping
study. The state-of-art of software development in startups is
discussed in Section 6, whereas in Section 7 the reported soft-
ware development work practices are analyzed. Section 8 con-
cludes the paper, answering the posed research questions and
providing an outlook for future work.
2. Background and Related Work
Modern entrepreneurship, born more than thirty years
ago [16], has been boosted by the advent of the consumer Inter-
net markets in the middle of the nineties and culminated with
the notorious dot-com bubble burst of 2000 [17]. Today, with
the omnipresence of the Internet and mobile devices, we are
assisting to an impressive proliferation of software ventures -
metaphorically referred as the startup bubble. In fact, easy ac-
cess to potential markets and low cost of services distribution
are appealing conditions for modern entrepreneurs [18]. In-
spired by success stories, a large number of software businesses
are created every day. However, the great majority of these
companies fail within two years from their creation [8].
2.1. Software Startups
An early account for the term startup in the SE literature
can be found in Carmel [19] who studied in 1994 the time-to-
completion in a young package firm. He noticed how these
companies were particularly innovative and successful, advo-
cating the need for more research on their software develop-
ment practices so as to replicate success and try to transfer it to
other technology sectors.
Sutton [3] provides a characterization of software startups,
defined by the challenges they are faced with:
• little or no operating history - startups have little accumu-
lated experience in development processes and organization
management.
• limited resources - startups typically focus on getting the
product out, promoting the product and building up strategic
alliances.
• multiple influences - pressure from investors, customers,
partners and competitors impact the decision-making in a
company. Although individually important, overall they
might be inconsistent.
• dynamic technologies and markets - newness of software
companies often require to develop or operate with disruptive
technologies
2
to enter into a high-potential target market.
2
A new technology that unexpectedly displaces an established technology.
It does not rely on incremental improvements to an already established technol-
ogy, but rather tackles radical technical change and innovation [20].
One of the objectives of this SMS is to understand the context
that characterizes startups and to what extent Sutton’s definition
from 2000 has been adopted or broadened.
2.2. Startup Lifecycle
The lifetime of a startup company, from idea conception to
the maturity level, has been identified and reported from dif-
ferent perspectives (e.g. market [37] and innovation [38]). A
prominent contribution, from a SE viewpoint, is the model pre-
sented by Crowne [8], who synthesized the startup lifecycle in
four stages. The startup stage is the time when startups create
and refine the idea conception, up to the first sale. This time
frame is characterized most from the need to assemble a small
executive team with the necessary skills to start to build the
product. The stabilization phase begins from the first sale, and
it lasts until the product is stable enough to be commissioned to
a new customer without causing any overhead on product de-
velopment. The growth phase begins with a stable product de-
velopment process and lasts until market size, share and growth
rate have been established. Finally, the startup evolves to a ma-
ture organization, where the product development becomes ro-
bust and predictable with proven processes for new product in-
ventions.
2.3. Software Development in Startups
The implementation of methodologies to structure and con-
trol the development activities in startups is a major challenge
for engineers [11]. In general, the management of software de-
velopment is achieved through the introduction of software pro-
cesses, which can be defined as “the coherent set of policies, or-
ganizational structures, technologies, procedures, and artifacts
that are needed to conceive, develop, deploy and maintain a
software product” [21]. Several models have been introduced
to drive software development activities in startups, however
without achieving significant benefits [22, 11, 3].
In the startup context, software engineering (SE) faces com-
plex and multifaceted obstacles in understanding how to man-
age development processes. Startups are creative and flexible in
nature and reluctant to introduce process or bureaucratic mea-
sures which may hinder their natural attributes [3, 23]. Further-
more, startups have very limited resources and typically wish to
use them to support product development instead of establish-
ing processes [11, 24]. Some attempts to tailor lightweight pro-
cesses to startups reported basic failure of their application [25].
Rejecting the notion of repeatable and controlled processes,
startups prominently take advantage of unpredictable, reactive
and low-precision engineering practices [3, 26, 27, 28].
Product-oriented practices help startups in having a flexible
team, with workflows that leave them the ability to quickly
change the direction according to the targeted market [24, 3].
Therefore, many startups focus on team productivity, asserting
more control to the employees instead of providing them rigid
guidelines [26, 27, 28].
Startups often develop applications to tackle a high-potential
target market rather than developing software for a specific
client [18, 29]. Issues related to this market type are addressed
2

in literature by market-driven software development [30]. In
this regard, researchers emphasize the importance of time-to-
market as a key strategic objective [31, 32]. In fact, star-
tups usually operate in fast-moving and uncertain markets and
need to cope with shortage of resources. Other peculiar as-
pects influencing software development in the market-driven
context are related to requirements, which are reported to be
often “invented by the software company” [33], “rarely doc-
umented” [34], and can be validated only after the product is
released to market [35, 36]. Under these circumstances, failure
of product launches are largely due to “products not meeting
customer needs” [30].
2.4. Related work
The prospects of evidence-based software engineering [39]
have motivated researchers to conduct systematic literature re-
views and mapping studies. Zhang and Babar [6] report on 148
SLR’s and SMS’s published between 2004 and 2010. How-
ever, none of these reviews nor the reviews conducted up to
February 2014
3
, investigated software engineering in the con-
text of startups. Nevertheless, there exist reviews that studied
software engineering topics pertinent to specific contexts or en-
vironments (as opposed to reviews that investigated individual
software engineering technologies, e.g. feature location [40] or
search-based software testing [41]) that we consider as related
work. Small and medium-sized enterprises (SMEs) and startups
possibly share some characteristics, such as the low number of
employees (fewer than 250 [42]) and limited resources [43, 44].
Hence, reviews that study the literature on SMEs are relevant
related work.
Pino et al. [45] studied the adoption of software process
improvement approaches in SMEs [46]. They point out that
very few of the SMEs that were part of the reviewed studies
did achieve one of the pursued certifications, concluding that
standard-driven, not tailored improvement initiatives are not
suitable for small companies, confirming also Staples’ et al.
findings [47]. Taticchi et al. [48] observe a similar situation
in the area of business performance measures and management
(PMM). Their review identifies a lack of PMM models specif-
ically adapted to SMEs, speculating that non-adoption stems
from fear of costs and benefits incomprehension.
Thorpe et al. [49] reviewed the literature on using knowl-
edge within SMEs. Managers/entrepreneurs are an important
organizational resource in SMEs as they are drivers for creat-
ing knowledge. This knowledge is best encoded in organized
routines that allow widespread sharing within the firm. The
challenge is to provide enough structure, allowing knowledge
creation and sharing to scale, without limiting creativity and
learning.
Rosenbusch et al. [50] studied the innovation-performance
relationship in SMEs by conducting a meta-analysis of 42 em-
pirical studies that cover 21,270 firms. Interesting to our stud-
ied context is their finding that innovation has a stronger impact
3
We performed an automatic search with the search string published by
Zhang and Babar [6].
on younger firms than on more established SMEs. Furthermore,
evidence suggests that for small and young firms it is more ben-
eficial to conduct internal innovation projects than seeking in-
novation by collaborating with external partners.
Common to these reviews, looking at different aspects of
SMEs, is the recognition that properties of small firms require
solutions and technologies that are adapted to that specific con-
text. Similarly, we argue that startups, differing from SMEs
in terms of their operating history, outside influences and mar-
ket dynamism, require software development solutions adapted
to their context. This SMS seeks to evaluate, synthesize and
present the empirical findings on software development in star-
tups to date and provide an overview of researched topics, find-
ings, strength of evidence, and implications for research and
practice.
3. Research methodology
We chose to perform a systematic mapping study (SMS),
which is capable of dealing with wide and poorly-defined ar-
eas [15, 4]. A systematic literature review (SLR) [4] would
have been a less viable option due to the breadth of our overall
research question: What is the state-of-art in literature pertain-
ing to software engineering in startups?
The review in this paper follows the guidelines developed
by Kitchenham and Charters [4] and implements the system-
atic mapping process proposed by Petersen et al. [15]. Fig-
ure 1 illustrates the adopted process, whereas the individual
steps are explained in Subsections 3.1- 3.7. Note that rigor and
relevance assessment is an extension attributed to Ivarsson and
Gorschek [51] and synthesis is based on the constant compari-
son method proposed by Strauss and Corbin [52].
The SMS procedure was led by the first and second authors,
who performed the steps in Figure 1 in a co-located environ-
ment, i.e. working together on a single computer screen. Note-
taking during screening of papers and keywording alleviated
the resolution of conflicts among the reviewers during data ex-
traction and rigor and relevance assessment. If disagreement
persisted, an in-depth review of the paper was performed and,
if necessary, the third and fourth authors were consulted to take
a final decision.
3.1. Definition of Research Questions
The research question driving this mapping study is: What is
the state-of-art in literature pertaining to software engineering
in startups? To answer this question, we state the following
sub-questions:
• RQ1 What is the context that characterizes software develop-
ment in startups?
• RQ2 To what extent does the research on startups provide
reliable and transferable results to industry?
• RQ3 What are the reported work practices in association with
software engineering in startups?
3

Figure 1: Systematic mapping process (adapted from Petersen et al. [15])
With RQ1 we intend to understand the properties that char-
acterize the nature of software development in startups. Such
a characterization illustrates the dimensions by which startups
are defined in the state-of-art. With RQ2 we intend to determine
the scientific evidence of the results reported in the state-of-art,
allowing researchers to identify worthwhile avenues for further
work and providing practitioners a tool to navigate within the
state-of-art. With RQ3 we intend to identify the software engi-
neering practices applied in startups, providing a basis for de-
termining necessary further research.
3.2. Conduct Search
We identified the primary studies by exercising a search
string on scientific databases. The search string is structured
according to population, intervention and comparison, as pro-
posed by Kitchenham and Charters [4]. We omitted however
the outcome and context facet from the search string structure
as our research questions do not warrant a restriction of the re-
sults to a particular outcome (e.g. effective/not effective work
practices) or context (e.g. startups in a specific product do-
main).
Table 1 lists the final used keywords. The core concepts,
representing population, intervention and comparison, are de-
rived from our research questions. Following Rumsey’s guide-
lines [53], we identified synonymous, related/broader/wider
concepts, alternative spelling and part of speech for each core
concept. Note that we did not include specific keywords from
existing startup definitions (e.g. Sutton [3], discussed in Sec-
tion 2.1) to the population set of terms as this could have biased
the search.
The selected scientific databases on which we performed the
search are shown in Table 2, along with the number of publi-
cations retrieved from each database (up to December 2013).
We selected the databases considering their coverage and use in
the domain of software engineering, and their ability to handle
advanced queries, following the example of Barney et al. [54].
To increase publication coverage we also used Google
Scholar, which indexes a large set of data, both peer and non-
peer reviewed. Then, we proceeded to the customization of the
search string, adapting the syntax to the particular database
4
.
4
The individual search strings are available in the supplementary mate-
rial [55].
Table 1: Population, intervention and comparison search string keywords
Core concepts Terms
Software Startups software startup*; software start-up*; early-stage
firm*; early-stage compan*; high-tech venture*;
high-tech start-up*; start-up compan*; startup com-
pan*; lean startup*; lean start-up*; software pack-
age start-up*; software package startup*; IT start-
up*; IT startup*; software product startup*; soft-
ware start up*; internet start-up*; internet startup*;
web startup*; web start-up*; mobile startup*; mo-
bile start-up*;
Development develop*; engineer*; model*; construct*; imple-
ment*; cod*; creat*; build*;
Strategy product*; service*; process*; methodolog*; tool*;
method*; practice*; artifact*; artefact*; qualit*;
ilit*; strateg*; software;
Table 2: Selected databases and retrieved papers
ID Database Papers
A Inspec/Compendex (www.engineeringvillage2.org) 640
B IEEE Xplore (ieeexplore.ieee.org) 132
C Scopus (www.scopus.com) 468
D ISI Web of Science (wokinfo.com) 293
E ACM Digital Library (dl.acm.org/advsearch.cfm) 78
F Google Scholar (scholar.google.com) 158
Total 1769
3.3. Screening of Relevant Papers
The criterion that guided the inclusion of a paper was that
the study presented a contribution to the body of knowledge
on software development in startups. A contribution can be in
the form of an experience report, applied engineering practices,
development models or lessons learned. We excluded search
results that were:
• not peer-reviewed (grey literature, books, presentations, blog
posts, etc.)
• not written in English
• clearly obsolete (more than 20 years old)
• related to non-software companies (biotech, manufacturing,
electronics, etc.)
• related to established companies (VSE, SME, research spin-
offs)
4

Frequently asked questions

Q1. What are the contributions in "Software development in startup companies: a systematic mapping study" ?

This study aims to structure and analyze the literature on software development in startup companies, determining thereby the potential for technology transfer and identifying software development work practices reported by practitioners and researchers. The authors conducted a systematic mapping study, developing a classification schema, ranking the selected primary studies according their rigor and relevance, and analyzing reported software development work practices in startups. This mapping study provides the first systematic exploration of the state-of-art on software startup research. Furthermore, the results indicate that software engineering work practices are chosen opportunistically, adapted and configured to provide value under the constrains imposed by the 

Q2. What have the authors stated for future works in "Software development in startup companies: a systematic mapping study" ?

While the characterization of startups through recurrent themes presented in this paper can serve as a basis, future work is needed to compile a common startup terminology and a set of definitions. 

Q3. What is the key to improving quality assurance in startups?

providing timeefficient means to develop, execute and maintain acceptance tests is the key to improve quality assurance in startups [68]. 

Q4. What is the main strategy for enhancing performance and chances of success?

To accommodate flexibility of managerial and organizational practices, the empowerment of team members represents the main viable strategy to enhance performance and chances of success [19, 78]. 

Q5. What are examples of startups that evolved into successful businesses?

New software ventures such as Facebook, Linkedin, Spotify, Pinterest, Instagram, and Dropbox, to name a few, are examples of startups that evolved into successful businesses. 

Q6. What is the important factor contributing to academic results being applied in the industry?

One of the most important factors contributing to academic results being applied in the industry is the provision of strong scientific evidence [101, 102]. 

Q7. What is the important architectural aspect in startups?

As reported by Yoffie [76], scalability represents the most important architectural aspect in startups and should be addressed as soon as possible. 

Q8. How did the authors mitigate the threat of the final ranking?

To mitigate this threat, the authors used an automatic spreadsheet to compute the final scores, allowing us to adjust scores and weights, observing the effect of the final ranking in real time. 

Q9. What are the main principles of development in startups?

Development in startups is usually guided by simple principles, trying to avoid architectural constraints that are difficult to overcome as the product and user-base grows. 

Q10. What is the main requirement for startups to have the freedom to choose activities quickly?

Developers should have the freedom to choose activities quickly, stop immediately when results are wrong, fix the approach and learn from previous failures. 

Q11. How many studies are dedicated to software development in startups?

Only 16 studies (37%) are entirely dedicated to software development in startups, whereby 10 studies constitute a weak contribution type. 

Q12. What are the peculiar aspects influencing software development in the market-driven context?

Other peculiar aspects influencing software development in the market-driven context are related to requirements, which are reported to be often “invented by the software company” [33], “rarely documented” [34], and can be validated only after the product is released to market [35, 36]. 

Q13. What factors were used to determine the ranking of the papers?

In order to rank the papers, the authors defined a function incorporating the classification schema, rigor and relevance scores, and two additional factors that characterize the publication type and year. 

Q14. What is the main reason for the lack of open communication in startups?

In addition, open communication remains crucial for startups to handle engineering activities, understanding the progress, code conflicts and competences. 

Q15. What is the risk of excluding relevant primary studies?

The risk of excluding relevant primary studies is further mitigated by the use of multiple databases, which cover the majority of scientific publications in the field. 

Q16. How many primary studies received a low score for scientific rigor?

Seven primary studies (16% of the total), received an average score for industrial relevance (2) but a low score (0) for scientific rigor. 

Q17. How did the authors mitigate the threat of bias in the selection of studies?

To mitigate this threat, the authors defined and documented a rigid protocol for the study selection and, by conducting the selection together and dedicating a reasonable amount of time to review conflicts, mutually adjusting each others’ biases, as suggested by Kitchenham and Charters [4]. 

Q18. What did the authors do to compensate for the limitation of the traditional SMS framework?

With this extension the authors compensate for the SMS’ limitation of not assessing the quality of the mapped studies by developing and using a simple ranking function.