HAL Id: hal-00871001
https://hal.archives-ouvertes.fr/hal-00871001
Submitted on 8 Oct 2013
HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entic research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.
L’archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diusion de documents
scientiques de niveau recherche, publiés ou non,
émanant des établissements d’enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
A Survey of Real-Time Strategy Game AI Research and
Competition in StarCraft
Santiago Ontañon, Gabriel Synnaeve, Alberto Uriarte, Florian Richoux,
David Churchill, Mike Preuss
To cite this version:
Santiago Ontañon, Gabriel Synnaeve, Alberto Uriarte, Florian Richoux, David Churchill, et al.. A
Survey of Real-Time Strategy Game AI Research and Competition in StarCraft. IEEE Transactions
on Computational Intelligence and AI in games, IEEE Computational Intelligence Society, 2013, 5
(4), pp.1-19. �hal-00871001�
TCIAIG VOL. X, NO. Y, MONTH YEAR 1
A Survey of Real-Time Strategy Game AI
Research and Competition in StarCraft
Santiago Onta
˜
n
´
on, Gabriel Synnaeve, Alberto Uriarte, Florian Richoux, David Churchill, Mike Preuss
Abstract—This paper presents an overview of the existing work
on AI for real-time strategy (RTS) games. Specifically, we focus
on the work around the game StarCraft, which has emerged in
the past few years as the unified test-bed for this research. We
describe the specific AI challenges posed by RTS games, and
overview the solutions that have been explored to address them.
Additionally, we also present a summary of the results of the
recent StarCraft AI competitions, describing the architectures
used by the participants. Finally, we conclude with a discussion
emphasizing which problems in the context of RTS game AI have
been solved, and which remain open.
Index Terms—Game AI, Real-Time Strategy, StarCraft, Re-
view1
I. INTRODUCTION
T
HE field of real-time strategy (RTS) game AI has ad-
vanced significantly since Michael Buro’s call for re-
search in this area [1]. Specially, competitions like the “ORTS
RTS Game AI Competition” (held from 2006 to 2009), the
“AIIDE StarCraft AI Competition” (held since 2010), and
the “CIG StarCraft RTS AI Competition” (held since 2011)
have motivated the exploration of many AI approaches in the
context of RTS AI. We will list and classify these approaches,
explain their strengths and weaknesses and conclude on what
is left to achieve human-level RTS AI.
Complex dynamic environments, where neither perfect nor
complete information about the current state or about the
dynamics of the environment are available, pose significant
challenges for artificial intelligence. Road traffic, finance, or
weather forecasts are examples of such large, complex, real-
life dynamic environments. RTS games can be seen as a
simplification of one such real-life environment, with simpler
dynamics in a finite and smaller world, although still complex
enough to study some of the key interesting problems like
decision making under uncertainty or real-time adversarial
planning. Finding efficient techniques for tackling these prob-
lems on RTS games can thus benefit other AI disciplines
and application domains, and also have concrete and direct
applications in the ever growing industry of video games.
Santiago Onta
˜
n
´
on is with the Computer Science Department at Drexel
University, Philadelphia, PA, USA.
Gabriel Synnaeve is with the Laboratory of Cognitive Science and Psy-
cholinguistics (LSCP) of ENS Ulm in Paris, France.
Alberto Uriarte is with the Computer Science Department at Drexel
University, Philadelphia, PA, USA.
Florian Richoux is with the Nantes Atlantic Computer Science Laboratory
(LINA) of the Universit
´
e de Nantes, France.
David Churchill is with the Computing Science Department of the Univer-
sity of Alberta, Edmonton, Canada.
Mike Preuss is with the Department of Computer Science of Technische
Universit
¨
at Dortmund, Germany.
This paper aims to provide a one-stop guide on what is
the state of the art in RTS AI, with a particular emphasis
on the work done in StarCraft. It is organized as follows:
Section II introduces RTS games, in particular the game
StarCraft, and their main AI challenges. Section III reviews
the existing work on tackling these challenges in RTS games.
Section IV analyzes several current state of the art RTS game
playing agents (called bots), selected from the participants to
annual StarCraft AI competitions. Section V presents results
of the recent annual competitions held at the AIIDE and CIG
conferences and a StarCraft bot game ladder
1
. Section VI
compiles open questions in RTS game AI. Finally, the paper
concludes on discussions and perspectives.
II. REAL-TIME STRATEGY GAMES
Real-time Strategy (RTS) is a sub-genre of strategy games
where players need to build an economy (gathering resources
and building a base) and military power (training units and
researching technologies) in order to defeat their opponents
(destroying their army and base). From a theoretical point of
view, the main differences between RTS games and traditional
board games such as Chess are:
• They are simultaneous move games, where more than one
player can issue actions at the same time. Additionally,
these actions are durative, i.e. actions are not instanta-
neous, but take some amount of time to complete.
• RTS games are “real-time”, which actually means is that
each player has a very small amount of time to decide the
next move. Compared to Chess, where players may have
several minutes to decide the next action, in StarCraft, the
game executes at 24 frames per second, which means that
players can act as fast as every 42ms, before the game
state changes.
• Most RTS games are partially observable: players can
only see the part of the map that has been explored. This
is referred to as the fog-of-war.
• Most RTS games are non-deterministic. Some actions
have a chance of success.
• And finally, the complexity of these games, both in
terms of state space size and in terms of number of
actions available at each decision cycle is very large. For
example, the state space of Chess is typically estimated to
be around 10
50
, heads up no-limit Texas holdem poker
around 10
80
, and Go around 10
170
. In comparison, the
state space of StarCraft in a typical map is estimated to
1
An extended tournament, which can potentially go on indefinitely.
TCIAIG VOL. X, NO. Y, MONTH YEAR 2
be many orders of magnitude larger than any of those, as
discussed in the next section.
For those reasons, standard techniques used for playing
classic board games, such as game tree search, cannot be
directly applied to solve RTS games without the definition
of some level of abstraction, or some other simplification.
Interestingly enough, humans seem to be able to deal with
the complexity of RTS games, and are still vastly superior to
computers in these types of games [2]. For those reasons, a
large spectrum of techniques have been attempted to deal with
this domain, as we will describe below. The remainder of this
section is devoted to describe StarCraft as a research testbed,
and on detailing the open challenges in RTS game AI.
A. StarCraft
StarCraft: Broo d War is an immensely popular RTS game
released in 1998 by Blizzard Entertainment. StarCraft is set in
a science-fiction based universe where the player must choose
one of the three races: Terran, Protoss or Zerg. One of the
most remarkable aspects of StarCraft is that the three races
are extremely well balanced:
• Terrans provide units that are versatile and flexible giving
a balanced option between Protoss and Zergs.
• Protoss units have lengthy and expensive manufacturing
processes, but they are strong and resistant. These con-
ditions make players follow a strategy of quality over
quantity.
• Zergs, the insectoid race, units are cheap and weak. They
can be produced fast, encouraging players to overwhelm
their opponents with sheer numbers.
Figure 1 shows a screenshot of StarCraft showing a player
playing the Terran race. In order to win a StarCraft game,
players must first gather resources (minerals and Vespene gas).
As resources become available, players need to allocate them
for creating more buildings (which reinforce the economy, and
allow players to create units or unlock stronger units), research
new technologies (in order to use new unit abilities or improve
the units) and train attack units. Units must be distributed to
accomplish different tasks such as reconnaissance, defense and
attack. While performing all of those tasks, players also need
to strategically understand the geometry of the map at hand,
in order to decide where to place new buildings (concentrate
in a single area, or expand to different areas) or where to
set defensive outposts. Finally, when offensive units of two
players meet, each player must quickly maneuver each of
the units in order to fight a battle, which requires quick and
reactive control of each of the units.
A typical StarCraft map is defined as a rectangular grid,
where the width × height of the map is measured in the
number of 32 × 32 squares of pixels, also known as build
tiles. However, the resolution of walkable areas is in squares of
8 × 8 pixels, also known as walk tiles. The typical dimensions
for maps range from 64 × 64 to 256 × 256 build tiles. Each
player can control up to 200 units (plus an unlimited number
of buildings). Moreover, each different race contains between
30 to 35 different types of units and buildings, most of them
with a significant number of special abilities. All these factors
Fig. 1. A screenshot of StarCraft: Brood War.
together make StarCraft a significant challenge, in which
humans are still much better than computers. For instance,
in the game ladder iCCup
2
where users are ranked by their
current point totals (E being the lowest possible rank, and
A
+
and Olympic being the second highest and highest ranks,
respectively), the best StarCraft AI bots are ranked between D
and D
+
, where average amateur players are ranked between
C
+
and B. For comparison, StarCraft professional players are
usually ranked between A
−
and A
+
.
From a theoretical point of view, the state space of a
StarCraft game for a given map is enormous. For example,
consider a 128 × 128 map. At any given moment there might
be between 50 to 400 units in the map, each of which might
have a complex internal state (remaining energy and hit-
points, action being executed, etc.). This quickly leads to
an immense number of possible states (way beyond the size
of smaller games, such as Chess or Go). For example, just
considering the location of each unit (with 128 × 128 possible
positions per unit), and 400 units, gives us an initial number
of 16384
400
≈ 10
1685
. If we add the other factors playing a
role in the game, we obtain even larger numbers.
Another way to measure the complexity of the game is by
looking at the branching factor, b, and the depth of the game,
d, as proposed in [3], with a total game complexity of b
d
.
In Chess, b ≈ 35 and d ≈ 80. In more complex games,
like Go, b ≈ 30 to 300, and d ≈ 150 to 200. In order
to determine the branching factor in StarCraft when an AI
plays it, we must have in mind, that the AI can issue actions
simultaneously to as many units in the game as desired. Thus,
considering that, in a typical game, a player controls between
50 to 200 units, the branching factor would be between u
50
and u
200
, where u is the average number of actions each unit
can execute. Estimating the value of u is not easy, since the
number of actions a unit can execute is highly dependent on
the context. Let us make the following assumptions: 1) at most
16 enemy units will be in range of a friendly unit (larger values
are possible, but unlikely), 2) when an AI plays StarCraft,
it only makes sense to consider movement in the 8 cardinal
directions per unit (instead of assuming that the player can
issue a “move” command to anywhere in the map at any point
2
http://www.iccup.com/StarCraft/
TCIAIG VOL. X, NO. Y, MONTH YEAR 3
in time), 3) for “build” actions, we consider that SCVs (Terran
worker units) only build in their current location (otherwise, if
they need to move, we consider that as first issuing a “move”
action, and then a “build”), and 4) let’s consider only the
Terran race. With those assumptions, units in StarCraft can
execute between 1 (units like “Supply Depots”, whose only
action is to be “idle”) to 43 actions (Terran “Ghosts”), with
typical values around 20 to 30. Now, if we have in mind
that actions have cool-down times, and thus not all units
can execute all of the actions at every frame, we can take a
conservative estimation of about 10 possible actions per unit
per game frame. This results in a conservative estimate for the
branching factor between b ∈ [10
50
, 10
200
], only considering
units (ignoring the actions buildings can execute). Now, to
compute d, we simply consider the fact that typical games
last for about 25 minutes, which results in d ≈ 36000 (25
minutes × 60 seconds × 24 frames per second).
B. Challenges in RTS Game AI
Early research in AI for RTS games [1] identified the
following six challenges:
• Resource management
• Decision making under uncertainty
• Spatial and temporal reasoning
• Collaboration (between multiple AIs)
• Opponent modeling and learning
• Adversarial real-time planning
While there has been a significant work in many, others
have been untouched (e.g. collaboration). Moreover, recent
research in this area has identified several additional research
challenges, such as how to exploit the massive amounts of
existing domain knowledge (strategies, build-orders, replays,
and so on). Below, we describe current challenges in RTS
Game AI, grouped in six main different areas.
1) Planning: As mentioned above, the size of the state
space in RTS games is much larger than that of traditional
board games such as Chess or Go. Additionally, the number
of actions that can be executed at a given instant of time is also
much larger. Thus, standard adversarial planning approaches,
such as game tree search are not directly applicable. As we
elaborate later, planning in RTS games can be seen as having
multiple levels of abstraction: at a higher level, players need
long-term planning capabilities, in order to develop a strong
economy in the game; at a low level, individual units need to
be moved in coordination to fight battles taking into account
the terrain and the opponent. Techniques that can address these
large planning problems by either sampling, or hierarchical
decomposition do not yet exist.
2) Learning: Given the difficulties in playing RTS games
by directly using adversarial planning techniques, many re-
search groups have turned attention to learning techniques.
We can distinguish three types of learning problems in RTS
games:
• Prior learning: How can we exploit available data, such
as existing replays, or information about specific maps for
learning appropriate strategies before hand? A significant
amount of work has gone in this direction.
• In-game learning: How can bots deploy online learning
techniques that allow them to improve their game play
while playing a game? These techniques might include
reinforcement learning techniques, but also opponent
modeling. The main problem again is the fact that the
state space is too large and the fact that RTS games are
partially observable.
• Inter-game learning: What can be learned from one game
that can be used to increase the chances of victory in the
next game? Some work has used simple game-theoretical
solutions to select amongst a pool of predefined strategies,
but the general problem remains unsolved.
3) Uncertainty: Adversarial planning under uncertainty in
domains of the size of RTS games is still an unsolved chal-
lenge. In RTS games, there are two main kinds of uncertainty.
First, the game is partially observable, and players cannot
observe the whole game map (like in Chess), but need to
scout in order to see what the opponent is doing. This type of
uncertainty can be lowered by good scouting, and knowledge
representation (to infer what is possible given what has been
seen). Second, there is also uncertainty arising from the fact
that the games are adversarial, and a player cannot predict
the actions that the opponent(s) will execute. For this type
of uncertainty, the AI, as the human player, can only build a
sensible model of what the opponent is likely to do.
4) Spatial and Temporal Reasoning: Spatial reasoning is
related to each aspect of terrain exploitation. It is involved
in tasks such as building placement or base expansion. In
the former, the player needs to carefully consider building
positioning into its own bases to both protect them by creating
a wall against invasions and to avoid bad configurations where
large units could be stuck. In base expansion, the player has to
choose good available locations to build a new base, regarding
its own position and opponent’s bases. Finally, spatial reason-
ing is key to tactical reasoning: players need to decide where
to place units for battle, favoring, for instance, engagements
when the opponent’s units are lead into a bottleneck.
Another example of spatial reasoning in StarCraft is that
it is always an advantage to have own units on high ground
while the enemy is on low ground, since units on low ground
have no vision onto the high ground.
Analogously, temporal reasoning is key in tactical or strate-
gic reasoning. For example, timing attacks and retreats to gain
an advantage. At a higher strategic level, players need to rea-
son about when to perform long-term impact economic actions
such as upgrades, building construction, strategy switching,
etc. all taking into account that the effects of these actions are
not immediate, but longer term.
5) Domain Knowledge Exploitation: In traditional board
games such as Chess, researchers have exploited the large
amounts of existing domain knowledge to create good evalu-
ation functions to be used by alpha-beta search algorithms,
extensive opening books, or end-game tables. In the case
of RTS games, it is still unclear how the significantly large
amount of domain knowledge (in the forms or strategy guides,
replays, etc.) can be exploited by bots. Most work in this
area has focused on two main directions: on the one hand,
researchers are finding ways in which to hard-code existing
TCIAIG VOL. X, NO. Y, MONTH YEAR 4
strategies into bots, so that bots only need to decide which
strategies to deploy, instead of having to solve the complete
problem of deciding which actions to execute by each individ-
ual unit at each time step. One the other hand, large datasets
of replays have been created [4], [5], from where strategies,
trends or plans have been tried to learn. However, StarCraft
games are quite complex, and how to automatically learn from
such datasets is still an open problem.
6) Task Decomposition: For all the previous reasons, most
existing approaches to play games as StarCraft work by
decomposing the problem of playing an RTS game into a
collection of smaller problems, to be solved independently.
Specifically, a common subdivision is:
• Strategy: corresponds to the high-level decision making
process. This is the highest level of abstraction for the
game comprehension. Finding an efficient strategy or
counter-strategy against a given opponent is key in RTS
games. It concerns the whole set of units and buildings
a player owns.
• Tactics: are the implementation of the current strategy.
It implies army and building positioning, movements,
timing, and so on. Tactics concerns a group of units.
• Reactive control: is the implementation of tactics. This
consists in moving, targeting, firing, fleeing, hit-and-
run techniques (also knows as “kiting”) during battle.
Reactive control focuses on a specific unit.
• Terrain analysis: consists in the analysis of regions com-
posing the map: choke-points, minerals and gas emplace-
ments, low and high walkable grounds, islands, etc.
• Intelligence gathering: corresponds to information col-
lected about the opponent. Because of the fog-of-war,
players must regularly send scouts to localize and spy
enemy bases.
In comparison, when humans play StarCraft, they typically
divide their decision making in a very different way. The
StarCraft community typically talks about two tasks:
• Micro: is the ability to control units individually (roughly
corresponding to Reactive Control above, and part of
Tactics). A good micro player usually keeps their units
alive over a longer period of time.
• Macro: is the ability to produce units and to expand
at the appropriate times to keep your production of
units flowing (roughly corresponding to everything but
Reactive Control and part of Tactics above). A good
macro player usually has the larger army.
The reader can find a good presentation of task decom-
position for AIs playing RTS in [6]. Although the previous
task decomposition is common, a significant challenge is on
designing architectures so that the individual AI techniques
that address each of those tasks can communicate and effec-
tively work together, resolving conflicts, prioritizing resources
between them, etc. Section IV provides an overview of the
task decompositions that state-of-the-art bots use. Moreover,
we would like to point out that the task decomposition above
is not the only possible approach. Some systems, such as
IMAI [7], divide gameplay into much smaller tasks, which are
then assigned resources depending on the expected benefits of
Strategy
Tactics
Reactive control
~1 sec
~30 sec
~3 min
direct knowledge
Spatial reasoning
player's intentions
partial observations
mean
plan term
Temporal reasoning
numb
3
Fig. 2. RTS AI levels of abstraction and theirs properties: uncertainty (coming
from partial observation and from not knowing the intentions of the opponent)
is higher for higher abstraction levels. Timings on the right correspond to an
estimate of the duration of a behavior switch in StarCraft. Spatial and temporal
reasoning are indicated for the levels at which greedy solutions are not enough.
achieving each task.
III. EXISTING WORK ON RTS GAME AI
Systems that play RTS games need to address most, if
not all, the aforementioned problems together. Therefore, it
is hard to classify existing work on RTS AI as addressing the
different problems above. For that reason, we will divide it
according to three levels of abstraction: strategy (which loosely
corresponds to “macro”), tactics and reactive control (which
loosely corresponds to “micro”).
Figure 2 graphically illustrates how strategy, tactics and
reactive control are three points in a continuum scale where
strategy corresponds to decisions making processes that affect
long spans of time (several minutes in the case of StarCraft),
reactive control corresponds to low-level second-by-second
decisions, and tactics sit in the middle. Also, strategic deci-
sions reason about the whole game at once, whereas tactical
or reactive control decisions are localized, and affect only
specific groups of units. Typically, strategic decisions constrain
future tactical decisions, which in turn condition reactive
control. Moreover, information gathered while performing
reactive control, can cause reconsideration of the tactics being
employed; which could trigger further strategic reasoning.
Following this idea, we consider strategy to be everything
related to the technology trees, build-order
3
, upgrades, and
army composition. It is the most deliberative level, as a player
selects and performs a strategy with future stances (aggressive,
defensive, economy, technology) and tactics in mind. We
consider tactics to be everything related to confrontations
between groups of units. Tactical reasoning involves both
spatial (exploiting the terrain) and temporal (army movements)
reasoning, constrained on the possible types of attacks by the
army composition of the player and their opponent. Finally,
reactive control describes how the player controls individual
units to maximize their efficiency in real-time. The main
difference between tactics and reactive control is that tactical
3
The build-order is the specific sequence in which buildings of different
types will be constructed at the beginning of a game, and completely
determines the long-term strategy of a player.