This work is licensed under a Creative Commons Attribution 3.0 License. For more information, see http://creativecommons.org/licenses/by/3.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSE.2016.2630689, IEEE
Transactions on Software Engineering
1
A Survey of App Store Analysis
for Software Engineering
William Martin, Federica Sarro, Yue Jia, Yuanyuan Zhang and Mark Harman
Abstract—App Store Analysis studies information about applications obtained from app stores. App stores provide a wealth of
information derived from users that would not exist had the applications been distributed via previous software deployment methods.
App Store Analysis combines this non-technical information with technical information to learn trends and behaviours within these
forms of software repositories. Findings from App Store Analysis have a direct and actionable impact on the software teams that
develop software for app stores, and have led to techniques for requirements engineering, release planning, software design, security
and testing. This survey describes and compares the areas of research that have been explored thus far, drawing out common
aspects, trends and directions future research should take to address open problems and challenges.
Index Terms—App Store, analysis, mining, API, feature, release planning, requirements engineering, reviews, security, ecosystem
F
1 INTRODUCTION
App stores are a recent phenomenon: Apple’s App Store and
Google Play were launched in 2008, and since then both
have accumulated in excess of 1 million downloadable and
rateable apps. Google announced that there were 1.4 billion
activated Android devices in September 2015 [32]. Mobile
app stores are also extremely lucrative: the set of online
mobile app stores were projected to be worth a combined
25 billion USD in 2015 [152]. The success of app stores has
coincided with the mass consumer adoption of smartphone
devices. Smartphones existed prior to the launch of these
stores, but it was not until 2008 that users could truly exploit
their extra computing power and resulting versatility through
downloadable apps. In-house and even commercial applica-
tions had been available before the launch of app stores, but
app stores had some differences: availability, compatibility,
ease of use, variety, and user-submitted content.
It is the user-submitted content that fundamentally dis-
tinguishes app stores from the ad-hoc commercially available
applications that existed beforehand. As a result, software
engineering researchers have access to large numbers of
software applications together with customer feedback and
commercial performance data, unavailable in previous soft-
ware deployment mechanisms.
Furthermore, through readily available, downloadable
toolkits, users can write their own applications to make use
of a smart device’s hardware. They can subsequently publish
their software in the central app store for users to download
(and possibly pay for). This publication process is subject to
the store’s in-house review and certification policies, but in
general apps and app updates can be made available quickly
(typically within hours/days).
In this paper we provide a survey of literature that per-
forms “App Store Analysis for Software Engineering” between
2000 and November 27, 2015
1
. Our contributions are as
follows: i) We provide formal definitions of apps, stores,
and technical and non-technical attributes, which are used
1. This paper is an updated version of an earlier technical report [157].
category
price
in-app purchases
descriptionname
size
rank of downloads
installs
what’s new
release date
version
platform version
API usage
version control
issues
discussions
number of ratings
rating
reviews
reviewers
Technical Attributes Non-technical Attributes
author
Fig. 1. Example attributes showing mined attributes that are strictly
technical (left) or non-technical (right), and attributes that may be in
either category (centre in box).
for App Store Analysis research. ii) We study the growth
patterns of App Store Analysis literature both overall, and in
each emergent subcategory. iii) We analyse the scale of app
samples used, and discuss how this is likely to progress in
the future. iv) We identify some of the key ideas published in
App Store Analysis, in addition to common aspects, trends
and future directions, to help readers to understand the
progression of the field overall.
1.1 Definitions
The following definitions help to clarify key components of
App Store Analysis literature. We used them to find all the
relevant literature.
App: An item of software that anyone with a suitable plat-
form can install without the need for technical expertise.
App Store: A collection of apps that provides, for each app,
at least one non-technical attribute.
Technical attribute: An attribute that can be obtained solely
from the software.
Non-technical attribute: An attribute that cannot be ob-
tained solely from the software.
This work is licensed under a Creative Commons Attribution 3.0 License. For more information, see http://creativecommons.org/licenses/by/3.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSE.2016.2630689, IEEE
Transactions on Software Engineering
2
Examples of attributes are shown in Figure 1, based on the
data we collected in previous studies [92], [154], [202]. As
our diagram shows, some attributes are distinctly technical
or non-technical in a boolean sense, but others lie in a grey
area, depending on the precise interpretation of what can be
obtained from software alone. Those in the grey box cannot
be considered technical in the strictest sense of the definition,
because they are not guaranteed to be obtainable solely from
the software in all cases. These attributes can be both non-
technical and technical, depending on how they are obtained.
They are attributes that, in some cases, can be provided by
the developer and not the app store, whilst attributes that
are strictly non-technical may only be provided by an app
store. For example, consider the ‘author’ attribute. In the case
of Android software, the author can be obtained solely from
the distributed apk file. However, in the case of a compiled
C binary such as a simple “hello world” program, the author
cannot be obtained directly from the binary file. The ‘author’
attribute therefore belongs in the grey area. We can obtain
the size of the C binary, and so this attribute is technical; we
cannot obtain the price from either of these example files,
and so this is a non-technical attribute.
Our definition of App Store may seem simplistic. However,
at the time of writing, app stores serve as more than just
collections of apps, but enable more developers than ever to
produce and distribute content, and enable a communication
channel between users and developer via reviewing systems.
Therefore, our definition is aimed at inclusivity. In only 7
years since the launch of the two biggest app stores, there
are already over 180 papers devoted to their study, and each
of these stores has well over 1 million apps each. As this rapid
development has shown, the concept of apps and app stores
is very likely to evolve over the coming years. It is our aim to
encompass this evolution as best we can through the stated
definitions, in the hope that future surveys will be able to
build upon this work and the App Store Analysis literature to
come.
1.2 Overview
This survey is structured as follows: Section 2 describes the
process used to find the included literature; Section 3 breaks
down the growth trends in non-technical research compared
with technical-only research, and Section 4 breaks down the
growth of scale of apps used; key ideas in each subfield of
app store analysis are identified in Section 5.
We define the following App Store Analysis subfields,
based on the literature gathered through the process ex-
plained in Section 2: “API Analysis”, which is discussed in
Section 6; “Feature Analysis”, which is discussed in Section 7;
“Release Engineering”, which is discussed in Section 8; “Re-
view Analysis”, which is discussed in Section 9; “Security”,
which is discussed in Section 10; “Store Ecosystem”, which
is discussed in Section 11; and “Size and Effort Prediction”,
which is discussed in Section 12.
Closely related work is discussed in Section 13; guidelines
and recommendations for future app store analysis authors
are outlines in Section 14; we identify potential future direc-
tions in Section 15, and conclude our findings in Section 16.
2 LITERATURE SEARCH
In this section, we describe the process used to find litera-
ture, including our scope, search terms and repositories and
lessons learned for future app store analysis surveyors.
2.1 Scope
App Store Analysis literature encompasses studies that per-
form analysis on a collection of apps mined from an App
Store. We are particularly interested in studies that com-
bine technical with non-technical attributes, as these studies
pioneer the new research opportunities presented by app
stores. However, we also include studies that use app stores
as software repositories, to validate their tools on a set of
real world apps, or by using specific properties such as the
malware verification process apps go through before being
published in the major app stores.
Our survey is not a Systematic Literature Review (SLR).
The area of App Store Analysis is still developing, but has not
reached a level of maturity at which research questions can
be chosen and asked of a well-defined body of literature. Our
study aims to define, collect and curate the disparate liter-
ature, arguing and demonstrating that there does, indeed,
exist a coherent area of research in the field that can be
termed “App Store Analysis for Software Engineering”. We
hope that this will prove to be an enabling study for future
SLRs in this area.
We apply the following inclusion criteria:
i) The paper is related to software engineering, and may
have actionable consequences for software users, developers
or maintainers.
ii) The paper is related to mobile app stores, concerning the
use of collections of apps or non-technical data gathered from
one or more app stores.
We apply the following exclusion criteria:
i) The paper focuses on mobile app development but does
not extend to collections of apps nor to app stores.
ii) The paper uses an arbitrary collection of apps to test a
tool, but it was not mined from an app store, and the study
does not extend to app stores.
2.2 Search Methodology
In order to collect all relevant literature to date that meets
the scope defined in Section 2.1, we perform a systematic
search for the terms defined below, from each repository (also
defined below). Unique papers are collected into a table, and
a decision is made based on the inclusion criteria in three
stages:
Title: We remove publications that are clearly irrelevant from
the title.
Abstract: We inspect the abstract and remove publications
which are clearly irrelevant according to the scope defined
in Section 2.1.
Body: Results are read fully and a judgement is made on
whether the paper a) meets the key requirements on what
is defined as “app store analysis” in our scope, or b) is very
relevant to the field and so should be included as “expanded
literature”, to put the main literature into context. Papers
matching the requirements of a) or b) are included in this
survey.
This work is licensed under a Creative Commons Attribution 3.0 License. For more information, see http://creativecommons.org/licenses/by/3.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSE.2016.2630689, IEEE
Transactions on Software Engineering
3
A summary of the number of papers found through the
search, as well as the number of papers accepted at each stage
of validation, can be found in Table 1. All of the references
for papers discussed in this survey are available in an online
repository [201].
2.2.1 Search Repositories
We performed a search in each of the following repositories
for papers to include in the study: Google Scholar, Scopus,
JSTOR, ACM, IEEE and arXiv.
2.2.2 Ter ms
We searched for the following terms and phrases, to encom-
pass the sub-fields of App Store Analysis that we identify:
“App Store”, mining, API, feature, release, requirements, re-
views, security, and ecosystem. We performed searches for
the following specific queries, where terms joined by an ‘AND’
must appear, and phrases in quotes must appear verbatim:
“app store analysis”
“app store analysis” AND mining
“app store analysis” AND mining AND API
“app store analysis” AND mining AND feature
We performed the following more general searches to ensure
that no relevant literature was missed from the survey:
“app store” AND analysis AND API
“app store” AND analysis AND API AND mine
“app store” AND analysis AND feature AND mine
“app store analysis” AND mining AND requirements
“app store analysis” AND mining AND release
“app store analysis” AND mining AND reviews
“app store analysis” AND mining AND security
“app store analysis” AND mining AND ecosystem
We mitigate the threat of missing papers by conducting
searches for “app store analysis” AND “mining” and also each
of the names of each of the major subfields of App Store
Analysis literature. Since, by our definition, app store analysis
research uses collections of apps, this should encompass
much of the field. We also performed snowballing, which
further helps to mitigate the threat of potentially missing
papers. However, the threat of missing papers is a threat to
the validity of any survey, including this one.
2.3 Snowballing
In addition to the repository searches specified in Section 2.2,
we also perform snowballing [244] on many of the included
studies. To do this we inspect the studies cited by the study,
and the publications that subsequently cited the study, using
Google Scholar and ACM. By performing this process in
addition to repository keyword searching, we reduce the risk
that relevant literature is omitted from this survey.
2.4 Search Results
Search results can be found in Table 1.
We set the time window to start with the year 2000, yet
the earliest reported study is 2010. This is likely because the
App Stores that propelled mobile app usage to become widely
adopted were launched in 2008. Yet, it is interesting that
studies incorporating technical with non-technical app store
information did not emerge until two years later. Papers were
collected until November 27, 2015.
2010 2011 2012 2013 2014 2015
Year
0
5
10
15
20
25
30
35
40
45
50
55
60
65
70
75
Studies
Technical-only
Technical and non-technical
Fig. 2. Histogram showing number of research papers incorporat-
ing non-technical information and technical-only research papers
showing the period from 2010 to November 27, 2015.
2010 2011 2012 2013 2014 2015
Year
0
5
10
15
20
25
30
35
40
45
50
55
60
65
70
75
Studies
API
Feature
Release
Reviews
Security
Store Ecosystem
Prediction
Fig. 3. Histogram of sub-field trends showing the period from 2010 to
November 27, 2015.
An overlap was found between search queries performed,
and thus the total number of discovered papers through
search queries was fewer than suggested by the sum of
the bottom rows in Table 1. Many papers were discovered
through snowballing, which do not appear in the table.
We present a summary of the included literature in Ta-
bles 3 to 9. Histograms depicting the growth of publications
studied on App Store Analysis for software engineering can
be found in Figures 2 to 4, which show the split between
technical-only and technical and non-technical research, the
split between different subfields identified as subsections in
this survey, and the split between scale of studies in terms of
the number of apps used, respectively. A breakdown of these
studies in each sub-field that we identify is also presented
in Figure 5.
2.5 Lessons Learned
As can be seen from Table 1, for some queries, there were
large drops in the number of papers upon inspection of their
This work is licensed under a Creative Commons Attribution 3.0 License. For more information, see http://creativecommons.org/licenses/by/3.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSE.2016.2630689, IEEE
Transactions on Software Engineering
4
TABLE 1
Search query results indicating the number of hits each query generates, the number of these that were available to be inspected, the number of
titles and subsequent abstracts and paper bodies that were accepted as valid. The top boxes indicate more specific queries run in multiple paper
repositories, and the lower boxes indicate the more general queries run only in Google Scholar. In the case of Google Scholar, only the top 1,000
results were accessible to inspect at the time of search.
Specific
Queries
“app
store
analysis”
“app store
analysis”
AND mining
“app store
analysis”
AND mining
AND API
“app store
analysis”
AND mining
AND feature
“app
store
analysis”
“app store
analysis”
AND mining
“app store
analysis”
AND mining
AND API
“app store
analysis”
AND mining
AND feature
Google Scholar IEEE
Hits 35 17 9 13 3 40 13 13
Inspect 35 17 9 13 3 40 13 13
Title 15 13 8 12 3 8 8 8
Abstract 13 13 8 12 3 7 4 4
Body 12 13 8 12 3 5 4 4
ACM JSTOR
Hits 7 1,146 295 231 0 36 4 13
Inspect 7 1,146 295 231 0 36 4 13
Title 4 69 44 31 0 0 0 0
Abstract 3 57 27 22 0 0 0 0
Body 3 44 26 17 0 0 0 0
arXiv Scopus
Hits 0 81 28 10 1 128 21 1
Inspect 0 81 28 10 1 128 21 1
Title 0 4 1 0 1 128 21 1
Abstract 0 4 1 0 0 13 6 0
Body 0 4 1 0 0 11 4 0
General
Queries
“app
store” AND
analysis
AND API
“app store”
AND
analysis
API AND
mine
“app store”
AND
analysis
AND feature
AND mine
“app store
analysis”
AND mining
AND re-
quirements
“app store
analysis”
AND mining
AND release
“app store
analysis”
AND mining
AND reviews
“app store
analysis”
AND mining
AND security
“app store
analysis”
AND
mining AND
ecosystem
Google Scholar
Hits 3,130 409 1040 12 9 15 9 9
Inspect 1,000 409 1,000 12 9 15 9 9
Title 87 35 37 12 9 14 8 9
Abstract 61 23 33 12 9 14 8 9
Body 52 21 32 12 9 14 8 9
2010 2011 2012 2013 2014 2015
Year
0
5
10
15
20
25
30
35
40
45
50
55
60
65
70
75
Studies
10
6
−
10
7
apps
10
5
−
10
6
apps
10
4
−
10
5
apps
10
3
−
10
4
apps
10
2
−
10
3
apps
10
−
10
2
apps
1
−
10
apps
0
apps
Fig. 4. Histogram showing number of research papers grouped
into app quantity ranges each year, showing the period from 2010
to November 27, 2015. Each histogram depicts a range such as 10
2
-
10
3
apps, which means that the studies included used between 10
2
and
10
3
apps.
title or abstract, when performing the more general searches
on Google Scholar: searches for “app store” with many of
the combinatoral words resulted in several thousand papers
Prediction
9
Store Ecosystem
22
Security
41
Reviews
45
Release Eng.
12
Feature
32
API usage
26
Fig. 5. Pie chart showing overall sub-field distribution showing the
period from 2010 to November 27, 2015.
which may have mentioned “app store” only once. We found
that searching for “app store analysis” as a phrase narrowed
the results down a lot, but did miss some relevant papers.
Searches that included “mining” as a keyword did en-
compass much of app store analysis research due to the
This work is licensed under a Creative Commons Attribution 3.0 License. For more information, see http://creativecommons.org/licenses/by/3.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSE.2016.2630689, IEEE
Transactions on Software Engineering
5
TABLE 2
Number of research papers studying each app quantity range from
2010 to November 27, 2015.
No. Apps Range Papers No. Apps Range Papers
0 5 [10
3
, 10
4
) 36
[1, 10) 19 [10
4
, 10
5
) 39
[10, 10
2
) 21 [10
5
, 10
6
) 28
[10
2
, 10
3
) 31 ≥ 10
6
3
focus on collections of apps that meets our app store def-
inition. However, we found that the snowballing technique
was crucial in our literature search, because paper discovery
through many of the paper repositories we used could not
be replied upon to find all relevant papers; in a growing
field terms of reference are not fully stabilized. We therefore
encourage future surveyors to visit the App Store Analysis
paper repository [201], which can assist in the discovery of
app store analysis literature.
3 NON-TECHNICAL RESEARCH
While software engineering deals primarily with code, it is
not confined to deal with strictly technical sources of informa-
tion. We can combine data from multiple (technical and non-
technical) sources, and app stores provide a wealth of such
information. There are 127 of 187 (68%) papers included in
this study that incorporate non-technical information mined
from app stores in order to either infer technical attributes
(such as features), or to extract useful information such as
bug reports and feature requests from users.
The histogram in Figure 2 shows that the number of
studies incorporating non-technical information is growing
year-on-year. We can see from Figure 2 that even including
the boom in technical-only research, there is growth year-on-
year (with the exception of 2014). Using linear regression,
we are able to fit the growth trend with high accuracy
(R
2
= 0.9067, p = 0.003373), which indicates that we can
draw a straight line and predict (with 90% accuracy) the
publications for a given year.
4 SCALE OF STUDIES
In order to discuss the number of apps that are studied by
research papers, we first need to define a set of ranges. We
assign the papers studied to app quantity ranges in ascending
powers of 10, according to the number of apps that they
consider. The ranges that we assign, and the number of
research papers that study them, are shown in Table 2.
The median number of apps used in the considered litera-
ture is 1,679, and the mean is 44,807. This result shows that
half of the papers study fewer than 2,000 apps, but the other
half study a quantity of apps several orders of magnitude
larger. This is reflected in Figure 4, where the range [10
4
, 10
5
)
is shown to grow and in 2015 represents almost half of the
app usage literature.
The histogram for the studies using between 10
4
and 10
5
apps shows growth from 2011 to 2015, and this result is
reflected in the histogram for studies using between 10
5
and
10
6
apps as well, up to 2014. It is important to note that we
did not have complete data for 2015, so this result is subject
to change. Studies using smaller scales of apps show an
TABLE 3
Chronological summary of API-related App Store Analysis
literature showing the authors, publication year, publication venue, and
the number of apps used in the study.
Authors [Ref], Year Venue No. apps
Ruiz et al. [195], 2012 ICPC 4,323
Linares-V
´
asquez et al. [138], 2013 FSE 7,097
Shirazi et al. [196], 2013 EICS 400
Minelli and Lanza [163], 2013 ICSM 20
Minelli and Lanza [164], 2013 CSMR 20
Ruiz et al. [193], 2014 IEEE Soft. 236,245
Hao et al. [91], 2014 MobiSys 3,600
Dering and McDaniel [57], 2014 MILCOM 450,000
Linares-V
´
asquez et al. [140], 2014 MSR 24,379
Ruiz et al. [192], 2014 IEEE Soft. 208,601
Linares-V
´
asquez [137], 2014 ICSE comp. 0
Viennot et al. [226], 2014 SIGMETRICS 1,107,476
Bartel et al. [18], 2014 IEEE Soft. Eng. 1,421
Zhang et al. [250], 2014 WiSec 10,311
Borges and Valente [30], 2015 PeerJ C. S. 396
Bavota et al. [21], 2015 IEEE Soft. Eng. 5,848
Kim et al. [121], 2015 ASE 350
Khalid et al. [114], 2015 IEEE Soft. 10,000
Watanabe et al. [242], 2015 SOUPS 200,000
Zhou et al. [254], 2015 WiSec 36,561
Wan et al. [236], 2015 ICST 398
Wang et al. [237], 2015 ISSTA 105,299
Syer et al. [214], 2015 Soft. Qual. 5
Azad [15], 2015 Masters thesis 950
Wang et al. [238], 2015 UbiComp 7,923
Seneviratne et al. [204], 2015 WiSec 4,114
Mean 93,298
Median 5,086
uncertain change in frequency, indicating that most studies
in the future are likely to continue using over 10
4
apps. We
anticipate larger studies in the future, based on the growth of
App Store Analysis literature, the increasing quantity of apps
studied, and of course the growing app stores themselves.
5 KEY IDEAS TIMELINE
A timeline depicting the key ideas is shown in Figure 6. This
highlights the launch of major app stores studied, as well
as the first studies in each subsection. We include studies
into the timeline that have advanced the field of App Store
Analysis in some way, or introduced influential ideas into
their respective subsection.
6 API ANALYSIS
Papers that extract the API usage from app APKs or source
code, and combine this information with non-technical data
are discussed in this section, and are summarised in Table 3.
All API analysis literature studied apps from the Android plat-
form only. This may be due to the availability of tools which
can be used to decompile the apps and extract their API calls,
which are freely available and can be applied to downloaded
app binaries. It is perhaps surprising that such analyses have
not also been performed on the Apple platform, iOS, since
the store was launched in 2008. This might be because iOS
binaries are only available for the intended platforms, and
cannot be downloaded to, or used from a desktop computer
without an Apple Developer account, which is not free. Even
with such an account, app binaries or source code would be
needed, and neither are freely available due to a) copyright
