The challenge of designing scientific discovery games
Seth Cooper
1
, Adrien Treuille
3
, Janos Barbero
1
, Andrew Leaver-Fay
4
, Kathleen Tuite
1
, Firas
Khatib
2
, Alex Cho Snyder
1
, Michael Beenen
1
, David Salesin
1,5
, David Baker
2
, Zoran Popovi
´
c
1
,
and >57,000 Foldit players
6
1
Center for Game Science
Department of Computer Science & Engineering
University of Washington
{scooper,jbarbero,ktuite,axchos,beenen34,zoran}@cs.washington.edu
2
Department of Biochemistry
University of Washington
{firas,dabaker}@u.washington.edu
3
Department of Computer Science
Carnegie Mellon University
treuille@cs.cmu.edu
4
Department of Biochemistry
University of North Carolina
aleaverfay@gmail.com
5
Adobe Systems
salesin@adobe.com
6
Worldwide
ABSTRACT
Incorporating the individual and collective problem solv-
ing skills of non-experts into the scientific discovery pro-
cess could potentially accelerate the advancement of science.
This paper discusses the design process used for Foldit, a
multiplayer online biochemistry game that presents players
with computationally difficult protein folding problems in
the form of puzzles, allowing ordinary players to gain ex-
pertise and help solve these problems. The principle chal-
lenge of designing such scientific discovery games is harness-
ing the enormous collective problem-solving potential of the
game playing population, who have not been previously in-
troduced to the specific problem, or, often, the entire scien-
tific discipline. To address this challenge, we took an itera-
tive approach to designing the game, incorporating feedback
from players and biochemical experts alike. Feedback was
gathered both before and after releasing the game, to create
the rules, interactions, and visualizations in Foldit that max-
imize contributions from game players. We present several
examples of how this approach guided the game’s design,
and allowed us to improve both the quality of the gameplay
and the application of player problem-solving.
Categories and Subject Descriptors
D.2.2 [Software Engineering]: Design Tools and Tech-
niques; K.8.0 [Personal Computing]: General—games
1. INTRODUCTION
Games have recently been used to aid science by leveraging
human image-recognition abilities, for example, to locate
celestial objects [11]. This paper introduces a more gen-
eral class of scientific discovery games that focus on lever-
aging human problem solving ability to solve computation-
ally difficult scientific problems. A scientific discovery game
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise, to
republish, to post on servers or to redistribute to lists, requires prior specific
permission and/or a fee.
FDG 2010, June 19-21, Monterey, CA, USA
Copyright 2010 ACM 978-1-60558-937-4/10/06... $10.00
translates a class of computationally difficult scientific prob-
lems into puzzles, and provides a game-like mechanism for
non-expert players to help solve these problems. Many tra-
ditional aspects of game design apply to scientific discov-
ery games, including the design of introductory levels to
draw newcomers and explain game mechanics, the use of a
client-server architecture for competition and collaboration,
and the requirement that the game be fun. However, un-
like games whose goal is entertainment or education, scien-
tific discovery games introduce a unique challenge: enabling
non-expert natural problem solvers to advance a specific sci-
entific domain. This challenge influences all aspects of the
game design. First, visualization and graphics need to pro-
mote human ability to see complex solutions and convey
accurate scientific information while remaining accessible to
beginners. Second, interaction design must optimize for nat-
ural interactions suitable for the human exploration process,
while still respecting scientific constraints. Finally, the scor-
ing mechanism needs to be informative enough to promote
multiple human strategies, while remaining true to the lat-
est models of the underlying scientific phenomenon. Perhaps
the most distinguishing feature and the greatest difficulty of
design for this type of game is that the solution to the sci-
entific problem, and thus the solution to the corresponding
puzzles, is unknown. Since we do not know the solution a
priori, we cannot design the game with specific solutions in
mind.
To explore this space, we focused on human ability to rea-
son about 3D structures and on the biochemistry domain,
where many problems tend to be structural. We developed
Foldit, a biochemical discovery game. In this paper, we
discuss Foldit’s initial focus on protein structure prediction
– determining a protein’s shape given its sequence of con-
stituent amino acids. Protein structure prediction involves
finding favorable interactions that form when the protein’s
chemical groups come into contact – essentially a 3D jigsaw
puzzle. We believe that humans’ innate spatial reasoning
ability makes it possible for non-experts to make useful con-
tributions to this problem. We leverage expert knowledge to
shape the rules of the game, thus enabling a much larger pool
of non-experts to make make discoveries within this frame-
work. Over the first two years since Foldit’s public release
in May 2008, we have run roughly 600 structure prediction
Solution analysis
Game clients
Scientists Players Infrastructure
Aggregate
solutions
Open problems
Puzzles
Puzzles
and updates
Solutions
Servers
Web
Database
Figure 1: Overview of our architecture for scientific discovery games. The biochemistry team provides
structure prediction and design problems for the server. These problems become puzzles and are sent to
each player’s client. Players collaborate and compete to solve these problems and upload their solutions to
the server, where they are aggregated and sent back to the biochemistry team for analysis. This analysis can
then be used to improve the design of the game and puzzles.
puzzles and had over 57,000 players from a wide variety of
backgrounds participate.
The rest of this paper describes our experience designing
Foldit, with a special emphasis on the unique challenges
posed by making biochemistry problems accessible to any-
one. The creation of Foldit was a challenging and multidis-
ciplinary project, drawing together computer science, art,
game design and biochemistry. Moreover, we did not know
ahead of time which parts of the problem players would be
best at solving, or which in-game manipulation tools they
would use most effectively. The only way to find out was
to have people play Foldit. In order to deal with these and
other uncertainties, we took an iterative approach both be-
fore and after releasing the game to the public. We have
continually evolved the gameplay in response to massive
gameplay traces, player feedback and expert analysis, and
continue even now with this iterative process as we add fea-
tures and expand the set of biochemical problems to which
the Foldit community can contribute.
2. RELATED WORK
Games are often designed with an iterative approach, which
involves designing, testing, and evaluating repeatedly until
the player’s experience meets some criteria [10]. For most
games, the main criterion for the player’s experience is sim-
ply to have fun. Player feedback and playtesting are an inte-
gral part of the process, and there are a number of methods
of gathering and incorporating this information from play-
ers [1]. We have also continued the design process after the
game’s release, to incorporate data gathered from the play-
ers in a continual process of evolutionary redesigning [12].
Our work differs from the standard iterative approach in
that the game design space is constrained to conform with
existing physical models, and we include the input of scien-
tific experts in the evaluation of the game.
Recently, there has been much interest in using games as a
means of motivating people to perform tasks that are cur-
rently difficult for computers. Games such as the ESP game
[23] and Peekaboom [24] use human image-recognition abil-
ity to produce labeled images from gameplay. Image recog-
nition has also been used for finding particular features of in-
terest in scientific data, such as looking for signs of interstel-
lar dust [25], measuring and aligning features on a planet’s
surface [15], and classifying galaxy shapes [11]. Most such
work is heavily image-based, and these projects have been
successful in motivating players to sift through large image
sets, which would otherwise be a mundane task. Some games
have taken a slightly different approach, such as looking for
solutions to graph based problems [7]. Our work is different
because it leverages a deeper human problem solving ability
to create novel scientific results.
More generally, all serious games have a purpose beyond
entertainment that ranges from fitness and health [26], to
training [4], to social change [22]. In our work, the main goal
is to generate useful scientific discoveries; however, other
aspects of game design, such as the requirement that the
game be fun, contribute to achieving this goal, as the results
rely on players playing the game.
There have been many purely computational approaches to
protein structure prediction, including distributed comput-
ing projects such as Rosetta@home [21] (built on top of the
BOINC distributed computing interface [2]). Atomic simu-
lations of proteins have also been performed by distributed
computing [9, 16].
3. OVERVIEW
3.1 Background
Predicting protein structures computationally is a central
goal for computational biochemists because so much can be
understood about a protein’s function once its structure is
known, and because it is so challenging to observe a protein’s
structure directly. Proteins are central to biochemistry be-
cause they are the primary chemical for almost all cellular
processes. DNA, a perhaps more widely recognized cellu-
lar chemical, derives its entire purpose in encoding protein
sequences.
DNA encodes a protein by describing the linear sequence of
amino acids that compose the protein. Cells translate a se-
quence of DNA into a sequence of amino acids, and then the
resulting amino acid chain (the protein) folds into a unique,
compact structure – often called the protein’s native struc-
ture. It is well known that sequence determines structure
[3].
The native structure is one that is lowest in free energy –
it has the most favorable set of chemical interactions. Some
interactions involving the backbone – the repeating pattern
of atoms that connects all the amino acids in the chain – oc-
cur so frequently that the structures they form have special
names. These so called secondary structures include tightly
wound helices and extended sheets. The remaining inter-
actions involve amino acid sidechains, which stick out from
the backbone and differentiate the various amino acids.
Foldit is built on top of the Rosetta molecular modeling suite
which has proven useful at a wide variety of protein model-
ing tasks [20, 5, 17, 13]. The suite contains an energy func-
tion which captures the interaction energies between protein
elements, as well as a set of structural optimization subrou-
tines. For protein structure prediction, structures closer to
the native structure will have a lower energy than structures
further away from it. Foldit uses this state-of-the-art energy
function to compute player’s scores, and also takes advan-
tage of the optimization routines Rosetta makes available.
3.2 Architecture
Here we give an overview of the architecture of Foldit, which
can be seen at a high level in Figure 1. Foldit generally uses a
client-server architecture. Each user downloads and installs
the client, which then communicates with a central server to
send information about the local player and get information
about other players.
Scientists post problems to the server; in the case of Foldit,
these are protein structures for which the players are meant
to find the native structures. An initial protein structure
is associated with metadata such as a title and description,
and parametrization such as which energy function terms
to use. We call these puzzles, and they are posted on the
server for a fixed amount of time (usually a week). While
a puzzle is active, players can download it and interactively
reshape the protein to try to achieve the best score. This
often requires significant changes to the puzzle structures,
which are given in various partially-folded states, and in
some cases need to be completely refolded from a straight
line. Players’ structures, or solutions, are reported back to
the server, and players are ranked against other players who
are playing the same puzzle. Players can form groups with
which to share their solutions through the server, allowing
them to work together to find even better solutions than they
could working alone. When one player shares a solution by
Scientists
Game dev
team
Result solutions
Playtesting
Usage data
Puzzle problems
Expert feedback
Solution analysis
Prototypes
Game updates
Prototypes
Players
Figure 2: An overview of the interactions between
the three groups.
uploading it to the server, other players in the same group
are able to see it and download it. The social aspect of the
game is supported by in-game chat, a website with forums,
and a player-created wiki. At the close of a puzzle, the
solution data is aggregated, and presented to the scientists
for analysis.
The game is designed to be flexible, and the client allows
automatic updating so that we can continually evolve the
gameplay. The puzzle posting cycle and automatic updates
allow us to respond to not only player feedback, but also
to expert analysis, as we introduce and refine gameplay ele-
ments.
3.3 Iterative Strategy
In order to arrive at the current state of Foldit, we took an
iterative approach to the game’s design. Given the complex-
ity of this undertaking, we realized that it was unlikely that
all our initial decisions would be the best. There are three
major groups relevant to our approach: 1) the scientific ex-
perts whose problems the game is meant to help solve; 2)
the players; and 3) the game development team. The devel-
opment team must incorporate feedback from the players to
make sure the game is understandable and fun, and from the
experts to make sure that the results produced will be use-
ful to them. An overview of the interactions between these
three groups is given in Figure 2.
During the game’s initial development, the development team
and experts must work together closely to determine an ini-
tial direction. This involves defining what problems to ap-
proach, what the fundamental gameplay mechanics needed
are, and what the desired results are. Once possible games
have been prototyped, player feedback can begin to be incor-
porated. Early playtesting helps to uncover what elements
of the problem are fun and which can be most confusing
and difficult to understand. This can help to both focus the
gameplay and narrow the scope of the game to where players
will most likely be able to contribute.
After making the game available to the public, a large amount
of data and feedback can become available to help improve
the game. As in a traditional game, data on gameplay can
Launch
CASP8 begins
Lock and unlock all
Multiple chat rooms, news in
game
Group sharing
Rebuild added, pie menus
First CASP8 target
Constraints
Rebuild and tweak improved
Player wiki
Note mode
Soloist and evolver scores
separated
Band options, screenshot
sharing through chat
Quicksaves
In game notifications, tweak
improved
Signup in game
Feedback tracker
Development blog
Beta previews of upcoming
releases
CASP8 results
Achievements, level changes
Autosave recent best, share
with self
Events in intro puzzles
Behavior tab
Shake improved
Exploration map, draggable
panels
5/1/08 6/30/08 8/29/08 10/28/08 12/27/08 2/25/09 4/26/09 6/25/09 8/24/09 10/23/09
Figure 3: Selected events from the game’s evolution over time, with screenshots from from before release
(top) and the current version (bottom).
be gathered from players for an objective analysis of what
players are doing, and feedback from the player community
is extremely useful in determining new features. However,
in a scientific discovery game, as scientists post puzzles and
player solutions are analyzed, this analysis must then be in-
corporated in the design of the game, progressing towards
ever better results.
Following this pattern, Foldit has evolved significantly since
its initial release. A timeline of significant events in the
evolution of the game are given in Figure 3.
4. DESIGN CHALLENGES
4.1 Visualizations
The visualizations in a scientific discovery game must achieve
several purposes in order to allow players to apply their
problem-solving skills. They must reflect and illuminate
the natural rules of the system , in a way that makes state
of the system evident to the player and directs them to where
their contribution will be most useful. At the same time, the
visualizations need to manage and hide the complexity
of the system, so that players are not immediately over-
whelmed by information. They must be approachable by
players who have no knowledge of the scientific problem
at hand. Thus, they should look inviting and fun, and not
bring back memories of high school textbooks. Ideally, they
should be customizable, because as with other aspects of
the game, it is not clear from the outset what the best vi-
sualization will be, and different players may have different
preferences.
In order to make the visualization of Foldit reflect and illu-
minate the fundamental properties of proteins, we worked
with experts to distill simple rules upon which to base them.
The first rule is to avoid clashes. Clashes occur when atoms
are unrealistically close to each other, causing a large repul-
sive force. These can be prevented by keeping the atoms
from overlapping, and are represented by spiky, rotating
spheres that float between the overlapping atoms. The sec-
ond rule is to fill voids, or empty spaces in the protein. Pack-
ing the protein tightly will remove voids. Voids are repre-
sented as bubble-like objects that pop when they come in
contact with the protein. Clashes and voids appear red, as
natural proteins should not generally have any. The third
rule is to bury exposed hydrophobics. Hydrophobics are
sidechains whose chemical properties are such that it is fa-
vorable for them to be on the interior of the protein. Ex-
posed hydrophobics are represented as small, pulsing spheres
that move along their sidechain. These are drawn in yellow,
rather than red, because natural proteins may have some
exposed hydrophobics. The fourth rule is to maintain and
create hydrogen bonds, which form between particular pairs
of atoms and hold the protein together. Hydrogen bonds
appear as undulating bars between the bonded atoms, and
are drawn in blue, because they are good.
Due to the spatial nature of the problem, the visualization
of the protein closely matches the actual geometry of the
protein. To make the overall structure stand out, sheets,
helices, and loops are stylized, similar to many expert vi-
sualization tools [8]. Sheets appear with a zig-zag pattern
that will form hydrogen bonds when properly fit together.
Color also plays a large role in the visualization of the pro-
tein. The backbone color reflects the score of the protein
in a particular region – going from red in poor scoring re-
gions to green in good scoring regions – so players can see
where they can gain the most points. The sidechains are
colored by hydrophobicity, so players can quickly see if they
are extending them in the preferred direction. By coloring
backbone and sidechain independently we can display more
information while not introducing too much visual clutter.
Foldit takes a number of approaches to manage and hide
the complexity of huge networks of interconnected atoms
that make up a protein. Many unimportant details are hid-
den. Hydrogen atoms, which are plentiful on the protein
but do not add a lot of structural information, are hidden.
However, hidden information will reappear if it becomes im-
portant to the player: sidechains can disappear entirely to
make the overall structure of the protein’s backbone clearer,
but will reappear if they are causing a problem, such as if
they are involved in a clash. Many actual clashes them-
selves are also hidden: only the worst clash is shown on a
per-amino acid basis. This prevents the player from being
overwhelmed by the number of clashes if the protein is com-
pressed too tightly.
Figure 4: Flow through introductory levels. First, the player selects an available level from the menu. While
playing the level, text bubbles pop up to guide the player. When the goal score is reached, a short reward
animation is played.
To make the game approachable, we gave the protein itself
a bright, cartoonish look. Many pieces of the visualizations
move playfully around the protein. There are a wide variety
of visualization options available in the game as well, such as
alternative colorings and geometries for the protein. These
can be accessed through a special menu option that is turned
off by default. This approach allows more advanced players
the ability to customize their view in the view options menu,
but keeps things simple for newcomers.
Visualizations such as voids and exposed hydrophobics can
be computationally expensive to compute. To keep the game
interactive, we compute such visualizations in a separate
thread, which will update the visualization after a delay.
4.2 Interactions
The interactions in a scientific discovery game must also
achieve several purposes. They must respect the constraints
of the system required. However, they must also be suf-
ficient to explore the space of solutions enough to be able
to solve the problem. They should also be as intuitive and
fun as possible.
To ensure that the player interactions respect the con-
straints of protein folding, we developed a number of tools
for players to use based on the powerful set of optimizations
offered by Rosetta. By using Rosetta as a model for our
interactions, we could ensure that they would result in plau-
sible for protein structures. However, these optimizations
only formed the basis for the moves, and they all needed to
be adapted for interactive and intuitive use by players. The
primary method of interaction in Foldit is directly manipu-
lating the protein through pulling, by clicking and dragging
the mouse. Depending on the location of the pull, this per-
forms various Rosetta-based optimizations with the player’s
pull as a soft constraint. There are also buttons to launch au-
tomatic algorithms for continuous energy minimization (wig-
gle); discrete sidechain energy optimization (shake); frag-
ment insertion (rebuild); and the ability to rigidly rotate,
translate, and shift sections of the protein (tweak ). Players
are able to achieve fine-grained control over these optimiza-
tion through two methods. First, freezing, which prevents
parts of the protein from moving, and second, bands, which
can connect amino acids and pull on them independently
of the player. When making large restructuring operations,
the repulsion force between atoms can overpower what the
player is trying to do and make it more difficult to interact
with the protein. To get around this, we have added a be-
havior menu, with a slider that allows player to adjust the
strength of the repulsion during interactions.
There are additional modes for interaction that define what
the mouse buttons do when the user clicks on the protein.
The primary mode is the pull mode, described above. The
structure mode allows the player to redefine the secondary
structure labeling of the protein. This is done by directly
assigning from a menu, or dragging existing labels across
the protein. The note mode allows players to add their own
notes and remarks to sections of the protein. These can be
used by an individual or to communicate between players
sharing solutions.
In order to confirm that the interactions available in Foldit
were sufficient to explore enough protein structures to
allow the players to make a discovery, we ran several puzzles
in which the native structure was visible as a guide. With
this native guide, players were able to use the tools in Foldit
to get close to the native structure. The fact that players
were able to do this suggested that the interactions would be
sufficient to reach the native structure on unknown proteins
as well.
Further, to encourage players to use the available interac-
tions to explore the space in new ways, we added an explo-
ration map. The exploration map plots every Foldit solution
for a puzzle based on its score and how different it is from the
puzzle’s starting structure. The map gives players a rough
idea of the solution landscape: the different areas other play-
ers are exploring, and the scores they found there. Players
might be exploring a new region on the map that initially
gives a worse score, but by working hard in this new unex-
plored region, they might find a better shape and get the
highest score.
In order to make the interactions more intuitive and fun,
we followed the concept of touchability – being able to di-
rectly interact with the protein as though you could actu-
ally touch it. Before embracing this concept, our designs
only manipulated the protein though indirect sliders, but-
tons, and plots. However, we soon changed the design to
cause thing to occur by clicking on the protein itself. While
the major optimizations are still launched by buttons, ac-
tions like pulling, attaching bands, freezing, tweaking, and
others are performed directly on the protein. This also led
![Table 1: Rankings of predictions in CASP8 from the DBAKER team for targets which included Foldit solutions. Rankings for the remaining DBAKER prediction, which did not use Foldit, are also given. Rankings were based on the GDT TS metric[27].](/figures/table-1-rankings-of-predictions-in-casp8-from-the-dbaker-2j678dbm.webp)
