Cowboys, Ankle Sprains, and Keepers of Quality:
How Is Video Game Development Different
from Software Development?
Emerson Murphy-Hill
North Carolina State University
Raleigh, North Carolina, U.S.
emerson@csc.ncsu.edu
Thomas Zimmermann and Nachiappan Nagappan
Microsoft Research
Redmond, Washington, U.S.
{tzimmer,nachin}@microsoft.com
ABSTRACT
Video games make up an important part of the software industry,
yet the software engineering community rarely studies video
games. This imbalance is a problem if video game development
differs from general software development, as some game experts
suggest. In this paper we describe a study with 14 interviewees and
364 survey respondents. The study elicited substantial differences
between video game development and other software development.
For example, in game development, “cowboy coders” are necessary
to cope with the continuous interplay between creative desires and
technical constraints. Consequently, game developers are hesitant
to use automated testing because of these tests’ rapid obsolescence
in the face of shifting creative desires of game designers. These
differences between game and non-game development have
implications for research, industry, and practice. For instance, as a
starting point for impacting game development, researchers could
create testing tools that enable game developers to create tests that
assert flexible behavior with little up-front investment.
Categories and Subject Descriptors
D.2.0 [Software Engineering]: General – Standards. K.8.0
[Personal Computing]: General – Games.
General Terms
Human Factors, Management
Keywords
Software engineering, games, practices
1. INTRODUCTION
Games are becoming an increasingly important part of the software
development industry. Beyond simply entertainment, video games
are increasingly being used to train students, soldiers, and medical
professionals [1] [2]. Congruent with their growing importance,
video games’ revenue is increasing as well; video games earned
more than three times the revenue of retail software in 2012 [3].
Despite games’ importance, they are rarely studied in software
engineering research. Of the 116 open and closed source software
projects studied in the last two years at major software engineering
venues, only 3 were games [4]. Of the projects in two major
software engineering corpora, SIR [5] and Qualitus [6], 0% and 3%
are games, respectively. The lack of software engineering research
about games, despite their importance, presents two problems.
First, if non-game software development is indeed different than
game development, past software engineering research will have
little impact on games. By analogy, the medical community faced
significant criticism for over-enrolling men in coronary heart
disease studies. As a result, “procedures and therapies currently
used” for the disease are “developed predominantly or exclusively
for men” [7]. Software engineering researchers’ practice of “under-
enrolling” games in studies may likewise result in tools and
practices that are inapplicable to game development.
Second, if game development is indeed different from “traditional”
software engineering, there are educational and practical impacts.
In his book on game development, Bethke states
Too often game developers hold themselves apart from formal
software development and production methods with the false
rationalization that games are an art, not a science. [8]
If this statement is true, then software engineering educators need
to teach their students different skills for game development than
for developing other types of software. If this statement is false,
then game developers would benefit from adopting the practices of
software engineering that are empirically validated.
So: is game development different from traditional software
engineering, or is it not? Like most questions, the answer is likely
that it is different in some ways but similar in others. Unfortunately,
which way it is similar or different has not been systematically
studied. This paper’s primary contribution is the first broad-based
empirical study to explicitly contrast traditional software
engineering against video game development.
In Section 2, we survey research on game development and discuss
the few empirical studies that do exist. In Section 3, we describe
our interview and survey study methodology, then discuss the
results in Section 4. We discuss limitations to our study in Section
5, the implications in Section 6, and conclude in Section 7.
2. RELATED WORK
Many books exist with prescriptive practices for developing games.
Some describe the developer roles and the high-level process that
developers and organizations should use when creating games [9]
[10] [11] [8] [12]. In the same vein, Blow’s magazine article details
his experiences with the fundamental difficulties in game
development [13]. These works are based on the experience of the
authors and largely do not contextualize game development as a
special type of software engineering. In contrast, our findings are
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise,
or republish, to post on servers or to redistribute to lists, requires prior
specific permission and/or a fee.
ICSE'14, May 31 – June 7, 2014, Hyderabad, India
Copyright 2014 ACM 978-1-4503-2756-5/14/05... $15.00.
based on empirical observations that explicitly focus on the
differences between general software engineering and game
development.
Recently, several researchers have focused on studying the process
of developing games. Ampatzoglou and Stamelos provide an
overview of the intersection of software engineering and games,
noting the dearth of empirical studies [14]. One such study is
Tschang’s qualitative investigation of 65 game development
project postmortems, finding significant differences between game
development and other creative industries [15]. Tschang also
developed a grounded theory of creativity in game development
[16] and a theory of innovation [17]. Baba and Tschang contrast the
spiral model of software development [18] against a new “outward
spiral model” of game development, empirically derived from
business practice manuals and some number of interviews with
Japanese “managers and project team members” [19]. Our work
builds on this work by studying differences between traditional
software engineering and game development.
Like our work, existing work has empirically investigated game
development. Burger-Helmchen and Cohendet interviewed 8 game
developers and discovered how communities of developers and
users interact [20]. Kultima and Alha interviewed 28 game
professionals, finding that they viewed their development process
was organic and uncontrollable [21]. Stacey and Nandhakumar
interviewed 20 developers, finding that predefined phases in
traditional software development models may be harmful to game
development [22]. Callele and colleagues analyzed 50 postmortems
of game development projects and found most requirements
failures occur between the preproduction and production phases
[23]. Kasurinen and colleagues interviewed 27 game developers
and found they expect adaptability in the tools they use [24]. Musil
and colleagues surveyed 13 Austrian game companies, revealing
that the industry largely uses Agile practices [25]. Lewis and
colleagues created a taxonomy of bugs in video games [26]. In
contrast to this prior work, our paper studies broad differences
between game development and traditional software engineering.
Also like our work, some existing research has investigated
differences between game development and traditional software
engineering. Specifically, Petrillo and colleagues analyzed 20
publically-available game postmortems and found that problems
encountered [27] [28] and processes used [29] by game developers
were largely the same as those for traditional software engineers.
One significant limitation to this work is that contributing game
developers may be reticent to report some negative aspects of their
work, because the postmortems were publically available. In
contrast, our work uses anonymized interviews and surveys, which
we believe helped respondents be more candid.
Prior position papers have explicitly compared software
engineering and game development, namely that of Lewis and
Whitehead [30] as well as Kanode and Haddad [31]. In contrast, the
work presented here derives its results from empirical grounding.
3. METHODOLOGY
Our study methodology involved two parts, qualitative interviews
and quantitative surveys, which we describe below. All study
materials can be found at our website.
1
1
http://people.engr.ncsu.edu/ermurph3/experiments/Games.pdf
3.1 Interviews
Protocol. We interviewed developers with experience in both game
development and non-game development. The first author
interviewed developers either in person if they worked in the
Seattle area, or via Skype or phone if they did not. Each interview
was completed in an average of about one hour. The interview had
four parts.
In the first, the interviewer asked a few demographic questions
relating to how much experience the interviewee had.
In the second part, the interviewer asked an open-ended question
about what differences the interviewee noticed between software
development for games versus non-games. This part allowed
interviewees to speak freely about differences without the
interviewer biasing their responses.
In the third and fourth part of the interview, we presented
interviewees with a list of topics to prompt them to discuss topics
that they had not explicitly considered. We gave half of
interviewees the topics from the 10 areas in the Software
Engineering Body of Knowledge (SWEBOK) [32], such as
software maintenance and software testing. We gave the other half
of interviewees Humphrey and colleagues’ list of general work
features from applied psychology [33], such as social support and
problem solving. We chose SWEBOK to ensure that software
engineering topics were discussed, and the general list to make sure
that we covered a breadth of potential differences. The difference
between the third and the fourth part was that in the third,
interviewees chose 2 or 3 topics to discuss, whereas in the fourth,
the interviewer chose 2 or 3 topics. Moreover, in the fourth part, the
interviewer selected topics that been discussed the least in previous
interviews, to ensure even coverage of the topics. As a result, each
topic was discussed at least twice across all interviewees. Finally,
we thanked interviewees and debriefed them by informing them
about what we planned to do with the data.
Participants. We interviewed people with experience with both
game and non-game development by searching LinkedIn,
2
which
contains resumes of professionals. We searched for LinkedIn
members who were part of the “Game Development” group, which
included more than 65 thousand members at the time of the study.
Our initial search results included non-developers, including
designers with experience only in entertainment. We thus added the
“engineer” keyword to our search. We also aimed to focus on
developers who made video games, so we included the following
keywords in our search: PSP, PS1, PS2, PS3, PlayStation, Xbox,
Wii, and GameCube. This left 207 potential candidates to
interview.
We further narrowed our selection of potential interviewees by
manually scanning the search results for several criteria, making
sure that each potential interviewee reported at least 2 years of
game development experience within the last 10 years; at least 2
years of non-game development experience within the last 10 years;
and listed contributing to specific game titles. We performed this
search through each of the three LinkedIn accounts of the authors.
We chose candidates from “2
nd
degree connections”, meaning
associates of associates, because LinkedIn does not allow the
unfiltered viewing of profiles of community members of 3
rd
or
more degree.
2
http://www.linkedin.com
Thirty-eight people fit our criteria, all of whom we contacted by
email or social networking. Because many developers did not
respond immediately, we followed up repeatedly until we had
interviewed enough developers to reach saturation, that is, until we
were not discovering any new differences. We reached saturation
at 14 interviewees. In the remainder of the paper, we label each
interviewee P1 through P14.
Nine interviewees were working on a game at the time of the
interview and five worked on other software. Five interviewees
worked at Microsoft. Thirteen interviewees were male. Below, we
summarize the self-reported game and non-game development
experience data from interviewees:
Games
Non-Games
Median years of development experience
8.5
8.5
Number of interviewees
with “extensive”
experience in…
Programming
10
12
Design
6
5
Management
7
4
Audio/Visual
2
3
Testing
3
5
After recruiting, we found that P13 did not have software
engineering experience but instead worked as a hardware engineer
with software developers, prior to working in games. We included
him in our interviews because we felt his current game role, as a
producer, would provide a valuable perspective. However, because
of P13’s lack of software experience outside of games, we only use
P13’s data to illustrate game development themes brought up by
other interviewees.
Data Analysis. We used a transcription service to transcribe the
audio, then coded the interviews using Qualyzer.
3
We coded
transcripts using the same SWEBOK [32] and general work [33]
topics we used to prompt interviewees.
3.2 Surveys
Protocol. We created a 10-minute survey designed to assess
differences between game and non-game development. Our survey
aimed to quantify the qualitative differences expressed by
interviewees over a range of developers.
We used our results from the interviews to write 84 candidate
statements that asked respondents to rate their agreement with each
statement on a 5-point Likert scale, from Strongly Disagree to
Strongly Agree. For example, one statement was “Creating my
software is challenging.”
We removed statements that we felt were the most ambiguous, were
the most difficult for developers to accurately self-assess, or were
most similar to one another. This reduced our list to 28 final
statements, which we felt would keep the survey sufficiently brief.
The survey also collected demographic information.
Participants. We recruited engineers and testers to participate
because many statements on the survey reflected technical concepts
that engineers and testers would be most qualified to rate.
We recruited three sets of potential respondents within Microsoft:
300 who worked on games (who we will refer to as the “Games”
set), 300 who worked on Microsoft Office (“Office”), and 300 from
across the company but did not work on games or Microsoft Office
(“Other”). We chose these sets in order to contrast responses; if
Games respondents provide significantly different responses than
3
http://qualyzer.bitbucket.org
Office and Other, this provides quantitative evidence to establish a
difference between game and non-game development. The reason
for choosing two types of non-game developers (Office and Other)
was that we were unsure whether high variances in product
differences would overwhelm game versus non-game differences.
Thus, to augment a diverse sample of developers (Other), we also
sampled a more homogenous set from single product (Office).
40% of recruits completed the survey. Below we summarize the
self-reported experience and backgrounds from respondents:
Games
Office
Other
Mean years at Microsoft
4.4
7.1
5.1
Mean years of development
experience
10.7
11.0
8.8
Number of engineers
113
61
82
Number of testers
32
39
37
Data Analysis. We examined distributions of Likert responses for
each of the three participant sets and compared them using a
Wilcoxon rank-sum test. Although we report the full results in
Section 4.3, along the way in Sections 4.1 and 4.2, we link
interviewee comments with survey responses by referring to survey
statements like so: [S1]. We number statements in the order in
which they appeared in the survey, S1 through S28. We annotate
each with whether they are statistically significant, like so:
[S1]
Significant differences between Games and both
Office and Other
that confirm interviewees’
responses
[S1]
Significant differences between Games and either
Office or Other that confirm interviewees’ responses
[S1]
No significant differences
[×S1]
Significant differences between Games and Office
or Other, but opposite of interviewee responses
Other outcomes were theoretically possible, but did not occur. One
such example could be [
×
S1], meaning that both Office and Other
were significantly different from Games, but Other confirmed
interviewee responses while Office opposed them.
4. RESULTS
In this section, we report results based on the interview topics. We
combine several topics into one when interviewees had little to say
about an individual topic. In some cases, we have anonymized parts
of quotes to maintain interviewees’ privacy.
4.1 SWEBOK Topics
4.1.1 Software Requirements
Nearly every interviewee made a strong statement about
differences between game requirements versus requirements in
other software. In essence, games generally have one and only one
requirement – that they are “fun.”
Interviewees noted that functional requirements are better suited for
non-games than games [
S6]. As P13 noted, as a game
developer, you are “designing an experience, an emotional
experience... It’s something supposed to be fun which is very
subjective” and is an “artistic achievement.” Rather than strict
requirements, the game designer “will give you a set of high-level
goals that they want out of a certain feature, but they don’t even
really know what they want” (P3).
Interviewees pointed to several reasons why requirements are so
much more subjective in game development. As P3 said, even with
a clear vision from a designer, when the vision is implemented, it
may not be fun. Another reason is that the consequences of
unfulfilled requirements in a game are less problematic than in
other software; game users move on quickly after a single
incomplete experience, but if a user is using email software and the
email does not get to his boss, the user “could get fired” (P1).
Another reason for requirement differences between games and
non-games is because game user experiences tend to be
significantly different from non-game experiences. P8 gave an
example: in an e-commerce application, a user has a task to
complete that typically takes only a few minutes. In contrast, in
some games people play for hours straight on a daily basis over the
course of months. As a result, the requirement for games is that the
user should be able to stay engaged on multiple timescales, and the
mechanism to achieve that will vary from game to game.
Instead of requirements specifications that may be found in general
software development, guidance for what a game should do comes
from other sources. Game designers with a particular vision are one
source. If a game has a predecessor, game requirements can come
from users, yet the “fun” requirement caveat applies – if a user
wants something, it may not be implemented because it may not
enhance fun. For games that are re-released every year (such as
sports games), another source of guidance can come from previous
iterations; as P8 suggested, game developers may ask
"What did players play two years ago?" and really fixing the
issues that were there some time ago. And also playing
somebody else's games and comparing your own game to
somebody else's, trying to make it better or trying to solve some
of the problems, try to differentiate.
Although usability and user experience is often an important
quality of non-game software, the way users interact with games
means that the user experience requirements for games and non-
games are different. For example,
Game play is more about feel. It’s hard to dissect scientifically.
(P13)
In sum, requirements appear to be more subjective in games. As we
will discuss in subsequent sections, this has several consequences
for the way games are developed, compared to non-game software.
4.1.2 Software Design
Interviewees explained that in games they tended to do less design
as a planning activity for a few reasons.
First, because the “fun” requirement is nebulous, many plans that
are made will not produce a fun game. Thus, participants said that
there is wasted effort [S5] if part of a game turns out not to be fun,
and that waste is multiplied if additional effort was put into design.
For example, according to P11, the game producer
…doesn’t want you [the developer] to go and spend a whole
bunch of time planning out how you’re going to do this thing
that he’s asked for because he might change his mind in a week
or two. Knowing that, he knows deep down that designing is
useless because he’s going to be constantly changing things…
As a consequence of less design-as-planning, interviewees reported
very little up-front thought put into architecture [S8]. As P11 put it,
“there is very little design… on the architecture of games. It’s more
of a ‘we needed to do this yesterday, go do it.’” Interviewees did
not totally discount architecture in games, instead noting that it was
less important in one-off games but more important in game series
where components are reused across releases. While the lifespan of
non-game software similarly has an influence on how much
architecture is designed up-front, the problem appears to be
especially acute for games because games’ lifespans are less
predictable. Paradoxically, game developers may go into
“architectural debt” early on [S3][S4] because there is such a high
probability that parts of their code will be thrown away, yet if the
game is successful and they wish to maintain or extend it, the
architectural debt must be paid down.
Second, interviewees reported less design-as-planning for cultural
reasons. As P5 explains it, “the design process as well is a little bit
different because… creativity tends to be rewarded more than
technical prowess.” Likewise, P11 pointed at the culture as well,
noting that the fundamental difference is that game development is
“a young man’s game and because of that, the whole process of
building games is an immature process.”
4.1.3 Software Construction, Tools, and Methods
A theme that interviewees repeatedly mentioned was code and tool
reuse. Interviewees mentioned that they believed that there was
little code reuse between and within games, compared to non-game
software [S1]. For example, in games:
There’s a lot of hacks and kludges to get things working… I’m
sure you would find tons of duplication of effort, definitely. I’ve
been an audio programmer on [X] different games and I’ve
written [X] different audio engines. (P11)
One reason that there appears to be less code reuse is because
games frequently have a significant emphasis on performance, and
that project-specific performance tuning is necessary:
It’s difficult to find a highly-optimized solution that’s going to
work for your particular game because [your situation is]
specific to the type of experience that you’re trying to create.
That just trickles to everything, the kind of physics that you have
in your game, the kind of visuals you have in your game. (P5)
The above quote implies that another reason why there is less code
reuse in games is because reuse implies similarities between
software, yet games emphasize innovation. P5 echoed this, saying,
The thing that makes video games unique is gamers want unique
experiences… With [general software], you don’t want to
change too much [because of, for example, the] backlash that
Microsoft received every time they move a button or change an
interface.
However, several interviewees noted
that reuse takes place
frequently in games, just in different ways. One way was that code
is recycled between subsequent game releases (P11). Another
source of reuse is in game engines, where multiple games and
multiple companies can reuse a core framework (P2, P9, P11).
Another source of reuse is the reuse of tools. Interviewees
mentioned that tool pipelines are critical for building games, and
that these pipelines may be used within organizations. Whereas
general software development tools, such as refactoring tools,
enjoy widespread availability to programmers, game tools appeared
to be developed more commonly within organizations [S15]. For
example, P5 summarized the tooling differences as:
[In general software development,] you might be building Word
or Excel or something like that [and use] some zip… tool or
something. But you’re building… video games, you are
sometimes building the resource compiling tools or tools that
are intended to extract 3D assets from other software like Maya
or Max and then convert it into a native format that then your
engine can load and render and process. So the tool pipeline is
incredibly important to video game development and it’s
probably I would say almost larger than the game itself.
4.1.4 Software Testing and Quality
Although software quality is important in both games and non-
games, the practice of testing appears to differ significantly. The
reason that quality is important in games is that, as P10 put it:
It’s almost like watching a movie... so that experience for you to
immerse [yourself] in the game experience, it has to not create
anything that would take you out of that immersion.
One significant difference is that there appears to be significantly
less test automation in game development:
In general, that's something that is very heavily done in games
– unit testing, regression testing, all the different types of
software testing you don't really see in video games, either.
Games are tested, but at the game play level.” (P9)
As this quote suggests, rather than using automated, low-level
testing, testing in games tends to be run more often at a high level,
either by a human playing through the game [S16], or as a script
simulating what a human would do [
×
S17]. Traditional software
engineering best practices dictate that neglecting low-level testing,
like unit testing [
S18], is risky [32], so it is worth investigating
why game developers appear to ignore this advice.
One reason that games are difficult to write automated tests
for is that it is so difficult to separate the user interface from the rest
of the game. For example, P4 noted, best testing practice is to
use MVC or one of the other patterns in order to try to separate
things so they're more easily testable. This, in general, is more
difficult in computer game development because of the extent to
which the user interaction is so pivotal… A lot of times, I'll see
developers just throw up their hands and say, "No, I'm … not
gonna worry about unit tests at all." It's much more common in
the game industry than it is in other places.
Another reason that it is difficult to write automated tests for games
is that it is harder to explore the state space in games [
S19]:
[Games tend] to have a large number of states that are user
driven… If you tried to create a test matrix for it, you end up
either having an immense test matrix or you end up restricting
the game design. In many cases, severely. (P4)
Another challenge is simply asserting what the correct behavior is:
If I'm playing a game, and maybe like I shoot this guy and I see
like a visual artifact like bouncing, do I care? (P1)
I could… write unit tests and say this enemy dies in two hits but
it’s not really meaningful because it’s not really that he dies in
two hits that’s so important, it’s that he dies in the right amount
of hits that the game designer thinks is the good amount. (P12)
Yet another challenge to writing tests is the non-determinism that
occurs in games due to multithreading, distributed computing,
artificial intelligence, and randomness injected to increase
gameplay difficulty. As P5 put it,
Definitely maintaining determinism in a sort of multi-player
environment is much more crucial that a single player
[environment]. You can definitely introduce very, very strange
bugs in, say, a game that wasn’t designed to be multi-player and
is multi-threaded as well and is doing a lot of these complicated
AI behaviors and physics.
Interviewees reported that there was also a strategic reason for
doing less automated testing and more human testing: automated
testing is fragile to frequent changes, whereas human testing tends
to be more resilient. In this sense, automated tests reduce agility
.
As P12 put it, “the game designer changes his mind so often that
tiny [test] tweaks happen all over the place.”
Interviewees also stated that another reason human testing is so
common is because it is relatively cheap, because game play testers
are less expensive than software testers:
So the cost is less; so that's the thing. Would you rather have one
guy that can do this automation or have four guys who can
actually go play the game? (P1)
Game testers illustrate Braverman’s sociological notion of
deskilling, where technology enables skilled workers (developers
who can write automated tests) to be replaced by unskilled workers
(play testers) [34]. This deskilled work stands in stark contrast to
that of game programmers, who can be described as doing craft
work, which depends “on special skills [which is marked by] the
lack of standardization of the product” [35].
One of the consequences of lack of test automation is that,
once a bug is reported, it is difficult to diagnose and debug [S20]:
A lot of times the bug reports tend to come back and it’s more
like, ‘oh well I pushed this button and I was in this corner and
the game locked up.’ So you have to kind of go back and try and
reproduce that type of situation. (P5)
4.1.5 Software Maintenance
Similar to game developers’ delay of architectural design,
maintenance also appears to be something that is often delayed in
games later than it would be in non-game software. As P12 put it,
There’s always a feeling in games that you almost don’t really
have to maintain it. In [non-game software], what’s going to
happen is most of your development time is actually going to be
in maintenance, you really have to make sure that the code you
write, the abstractions that you come up with for your code are
clean, that they’re maintainable, that you’ll be able to go in and
make changes as the years go by and presumably your system
stays in operation. With a video game… there’s kind of a sense
that you’re the last one to touch the code.
Also like the up-front design of architecture, there is a tradeoff
between improving maintainability early and the likelihood that
this effort will result in waste because the game will not be a
success (P2, P3). Interviewees also delayed improving
maintainability in games due to lack of management buy-in. As we
will discuss in Section 4.1.7, one reason for less management buy-
in appears to be because managers tend to be non-technical [S21]:
Anytime you’re going to be working on clean code, you have to
have buy-in from management or you have to have an
engineering team that’s willing to tell management to back off
because in the end, you’re going to be sacrificing time to do that.
(P12)
Another reason for lack of maintenance in games, at least from
a programming perspective, is that product releases may entail
changes to content rather than changes to behavior. While non-
games may compete in the marketplace based on new features, that
may not be the case for some games:
