IEEE TRANSACTIONS ON COMPUTATIONAL INTELLIGENCE AND AI IN GAMES, VOL. 5, NO. 2, JUNE 2013 77
Guest Editorial: Brain/Neuronal–Computer Game
Interfaces and Interaction
I. BRAIN–COMPUTER GAMES INTERACTION
T
HERE have been many attempts to define the word
“game.” The essential element
s of a good game are play,
nontrivial goals/ch allen ges and rules, and games often involve
a pretended or virtual reality. Nontrivi a l gameplay r equi res
a challenge, but not so much o
f a challenge that the player
becomes disinterested. Games, in general, have been around
since ancient times to entertain us. Since the first electronic and
video games appeared in
the 1940s and 1950 s, there has been
an increasing demand for enhancements to existing gam es and
new ways of interacting with computer games. More recently,
games have been used to
engage and stimulate us cognitively,
to help us learn, and to help us recover from illness (e.g.,
edutainment and serious games). Buttons, mice, joysticks,
joypads, and handh
eld devices have been the main human in-
terfaces with games for many years, then came steering wheels,
gear sticks, electronic musical instruments, and all sorts of
peripherals tha
t represent real-life objects. Today, brainwave-
or electroencephalogram (EEG)-controlled game controllers
add new options to satisfy the continual demand for new ways
to interact wi
th games, following trends such as the Nintendo®
Wii, Microsoft® Kinect, and Playstation® Move, which are
based o n accelerometers and motion capture.
EEG-based gam
e interactions are controlled through
brain–computer interface (B CI) technology, wh ich requ ires so-
phisticated signal processing to produce a relatively inaccurate
and unstable
control signal that provides a low comm unication
bandwidth with only a few degrees of freedom . E xtractin g
a reliable control signal from nonstationary brainw aves is a
challenge
being addressed by many researchers. Producing
paradigms for training users to produce brain activity that is
easily translated into a control is another key focus of BCI
research
. Another challenge is to develop games and game
control strategies that can be operated usin g unstable and
limited control sig nals to e x ploit the rich dynamics available in
brainw
aves. It is, therefore, important to engage those involved
in game development to help develop new paradigm s, not
only for enabling nonmu scu lar game interaction but also for
advan
cing the field of BCI in general.
Brainwave-controlled computer games have been researched
since the 1990s, with an emphasis on using video games to im-
prove
the user’s performance in BCI experiments and to main-
tain motivation, as w ell as to modulate brain activity, with the
aim of using that ability to interact with and control technology
and t
o comm unicate, movement-free. More recently, entertain-
ment and gaming have become a popular application focus fo r
BCI researchers, and gam e developers h ave begun to engage in
the
challenge that such a field propo ses. As a result, BCI games
are becoming increasingly more advanced, incorporating 3-D
environments, multiple user objectives, and hybrid control sys-
te
ms, inclu din g both conventional input devices and multiple
BCI techniques.
Dig
ital Object Identifier 10.1109/TCIAIG.2013.2264736
Brain-controlled video gam es can be used to train users to in-
tentionally m od ulate their brainw aves, to name just a few: 1) to
enable communication for those with severe movement impair-
ment; 2) to enable those who are p hysically impaired to enjo y
video games; 3) for rehabilita
tion (e.g., poststroke rehab to en -
courage brain repair); and 4) for gamers, to augment and im-
prove the game playing exp erience. The challenge is to train
the users in parad igms that are moderately challenging (e.g .,
beyond simple cursor control), such that pro gressing from a
training paradigm to a real-world communication/control situa-
tion significantly impact
s the subject performance. Video games
provide just such a learning environment, where increased cog-
nitive load can be controlled by modulating the amount of sta-
tionary or moving objects/matter, which may or m ay not need
to be attend ed to by the BCI user. Controllable distraction to the
user can help the user learn to cope with such distractions when
using BCI in the real wo
rld.
While, to date, there has been successful research into
brain–computer game interaction (BCGI), the algorithms and
techniques developed are limited in scope and may not utilize
all available data in the app rop ri ate contexts, e.g., optimizing
for g enre-specifi c games. While the importance of computer
games, the challenge
, and the competition provide key ingre-
dients for m otivating and engaging the users as they learn to
controlaBCI,itmustbeemphasized that brainwave-controlled
games need to be d eveloped to suit the end purpose or applica-
tion. For entertainment, this is obvious: keep the users engaged,
excited, challenged (but not too much), and immersed, where
gamers must feel th
ey are in control of the BCI. This may
involve tailoring the difficulty to suit the players’ ability, which
is acceptable for gaming when positive feedback is important,
but may not be desirable for training when the end application
requires precise and accurate selection, such as com mands for
assistive robotics or com mu nication . For ex amp le, w hen a BC I
is used as an alte
rnative communication device, the objective
is to maximize the probability of int erpreting the user’s intent
correctly. T her e fore, the games should m aintain a person’s
motivation to perfor m better, try harde r to produce the righ t
brain activity and activate the correct area of brain, and provide
feedback to the user to enable learning. However, if the BCI
is aimed at indu
cing neuroplastic changes in specific cortical
areas, i.e., in stroke rehabilitation (another application which
involves games and BCI), the objective is not only to provide
accurate feedback but also to encourage the user to activate
regions of co rt ex or t o produce oscillatory activities that do no t
necessarily provide optimal cont rol s ignals in the context of
BCI.
This special issue was, therefore, solicited to gain insights
into new biosignal processing algorithms, tested in gaming ap-
plications, which exploit BCI and neural signals to enhance
gameplay experience and player motivation, be the players able-
bodied or physically impaired.
1943-068X/$31.00 © 2013 IEEE
78 IEEE TRANSACTIONS ON COMPUTATIONAL INTELLIGENCE AND AI IN GAMES, VOL. 5, NO. 2, JUNE 2013
II. THE PAPERS
In this special issue, a snapshot of the current trends in BCI-
controlled computer games is presented across 11 manuscripts.
Marshall et al. provide a review of the field to date, how it
has grown over the past 20 years, and what types of BCI con-
trol strategies are suited to particular game genres, for the first
time categorizing games used with BCIs into genres. The ap-
propriateness of game genres (a category of games character-
ized by a particular set of gameplay challenges) and the associ-
ated gameplay challenges for different BCI paradigms are eval-
uated. Gameplay mechanics employed across a range of BCI
games are reviewed and evaluated in terms of the BCI control
strategy’s suitability, considering the genre and gameplay
me-
chanics employed. A number of recommendations for the field
relating to genre-specific BCI-game development and assessing
user performance are also provided for BCI-game developers
.
Interestingly, it was fo und that the action game genre was the
most popular, even though action games tend to require fast re-
sponses, while motor imagery was the most used BCI ap
proach
for games, even though motor-imagery-based BCI no rmally re-
quires training, which would be unusu al for general gaming.
The breadth of topics covered in this special issue
correlate
with the findings of the r eview by M arshal l et al.,wheremotor
imagery, steady-state visual evoked potentials (SSVEPs), and
P300-based game interfaces are shown to be th
e most popular,
with aspects of how biofeedback, user states, and emotions are
also exploited to enhance gameplay.
Chumerin et al. evaluated the use of steady-s
tate visual
evoked potentials in a m aze game on a consumer grade EEG
device (the Emo tiv EPOC) and a traditional EEG cap. The
game was then tested using th e consumer grad
e headset with
a broad audience (53 persons) in a real-world setting. Most
players enjoyed the game and had good control over it, yet a
percentage of players found the s tim u
li difficult to concentrate
upon. Recomm endations for the BCI-game design, the fitting
of consumer grad e EEG headsets on participants, and the use
of SSVEP s t imuli in games are present
ed, based on the findings
from the study.
Legeny et al. examine a context-dependent approach for an
SSVEP-based BCI-g ame controller
. The controller uses two
kinds of behavior alteration. Command s ma y be added and
removed if their use is i rrelevant to the context or the actions
resulting from th eir activation
, and may be weighted depending
on the likeliness of the actual users’ intention. The controller
was integrated in a test spaceship shooting game for a pilot
study using 12 subjects. Prel
iminary results obtained confirmed
thepossiblebenefits in te rms of a context-dependent controller,
workload reduction, and performance i mprovement.
Leeb et al. describe a multimo
dal approach using an asyn-
chronous B C I in parallel with a man ual joystick con trol signal,
while playing a game in virtual reality. The subject controls
a penguin character slidin
g down a mountain slope, in w hich
steering the game character left or right was achieved with a joy-
stick, whereas making the character jump was achieved using
foot motor imagery. The BC
I was built upon the so-called brai n
switch, which allowed for discrete and asynchronou s actions.
Results from 14 subjects showed that the use of a secondary
motor task (in this cas
e, joystick control) did not deteriorate the
BCI performance during the game. These findings show that
BCImaybeusedinamultimodalorhybridBCIimplementa-
tion in which a user can perform two tasks in parallel. These are
encouraging results, suggesting that BC I can indeed be used as
an additional control in computer games.
Thurlings et al.
studied a different aspect of multitasking,
in particular, dual-attentional tasks for event-related potential
(ERP)-based BCI, investigating if and to what extent ERPs and
ERP–BCI performan ce are affected in a dual-task situatio n and
if these effects are a function of the level of difficulty of a
concurrent task. These two tasks consist in attending to tac-
tile stimuli (for ERP-based BCI control) and performing a vi-
sual memory task. The study showed that when users are re-
quired to perform these two tasks simultaneously, the resu
lting
ERP-based BCI performances drop significantly. While they are
still higher than chance, they become lower than what wo uld be
necessary for effective control.
Overall, the s tud ies on multitasking may suggest that using
spontaneous BC I, such as motor-imagery-based BCI, can be
used in addition to other mo tor control commands ( e.
g., a joy-
stick), but that ERP-based BCI may not be used in addition to
other attentional tasks, hence constraining the types of games in
which ERP-based BCI can be used.
A more specific look at BCI games was provided by
Kaplan et al. in a review of BCI-controlled games, based on
the P300 evoked potential. The shortcomings o
f the P300 BCI
in gaming applications are reviewed, and it is outlined how
solutions for ov ercoming these shortco mings already exist in
several different games. Problems such as st
atic stereotyped
stimuli, goal selection c ontro l instead of process control, re-
peated stereotyped mental actions required to co ntrol a single
action in the game, and the syn chronous prot
ocols associated
with P300 are reviewed. Solutions for these problems are found
in existing BCI games, as well as recom m endations for making
future P300 BCI games more practical.
Kos’myna and Tarpin-Bernard present an evaluation of a
multimodal combination of BCI paradigms and eye tracking
with consumer grade h ardw are in a game.
The p aper evaluates
three combina tions of BCI and eye tracking, used in the context
of a simple puzzle game. The SSVEP, motor imagery, and eye
tracking are used in several diffe
rent combinat ion s to identify
the extent to which the paradigms impact the playability of the
game. Th e paper presents pr e liminary results that indicate th a t
BCI interaction is tiring and imp
recise, yet may be suited as
an optional and complem e nt ary m od alit y to oth er inter actio n
techniques. The combination of the eye tracker and SSVEP was
found to be the most well-roun
ded and natural com bination.
In terms of practical focus for BCGI, Scherer et al. propose
the use of games to enhance the user experience when col-
lecting behavioral data for r
esearch. The rationale is that exper-
imental paradigms used to collect behavioral trials from indi-
viduals are data centered a nd not user centered, resultin g in t he
experimental paradigms th
at are generally demanding for the
user/participant, and not always motivating or engaging. An ap-
proach involv ing the use of the K inect motion tracking sensor in
a game-based paradigm for
noninvasive EEG-based functional
motor m apping is proposed to alleviate this pro blem by m aking
the data collection experience more interesting to the user. Re-
sults from an experime
ntal study with able-bodied participants
IEEE TRANSACTIONS ON COMPUTATIONAL INTELLIGENCE AND AI IN GAMES, VOL. 5, NO. 2, JUNE 2013 79
playing a virtual ball game suggest that the Kinect sensor is
useful for isolating specific mo vements during the interaction
with the game, and that the computed EEG patterns for hand
and feet movements are in agreement with results described in
the literature.
Berta et al. provide an EEG and phy siolo gical signal analysis
for assessing flow in games. The paper defines flow in games as
a measure of keeping the player fu lly imme rsed and engaged i n
the process of activity within the gam e. The evaluation of flow
involves a four-electrode EEG, using the low beta frequen c y
bands for discriminating am on g g aming conditions. U sin g
simple signals from the peripheral nervous system, three levels
of user states were branded using a support vector machine
classifier. The user states were identified, using three lev
els of
a simple plane battle game, identifying states of boredom, flow,
and anxiety. The paper argues that a personalized system could
be im plemented in a consum er context, allowing for more
flowing gameplay in consumer games.
Van de Laar et al. investigate w heth er th e incorporation of
BCI into the popular game World of Warcraft affects t
he user
experience. A BCI control channel based on alpha band power
is used to control the shape and function o f the avatar in the
game. The character within AlphaWoW has two forms:
an elf
and a bear. The elf form allows them to attack enem ies from a
safe range and the bear form allows them to attack from close
range (the bear is also more resistant to atta
cks). The “shape
shifting” in the game is controlled via the user’s parietal alpha
activity. This study suggests that the use of BCI control can be
as much fun and natural to use as convention
al controls, even if
the player’s control is limited.
Finally, in the paper by Bonnet et al.,weseeafirst step t owar d
multibrain games, that is, games where two (
or more) players
compete or collaborate to steer a ball in the direction of goal-
posts left an d right on the s creen. Steering is done using moto r
imagery. In the collaborative versio
n, players make a joint effort
to steer the ball i n the same direction. In the competitive version,
their efforts are compared, and the ball goes in the direction of
the goalposts indicated by the stron
gest motor imagery co ntrol.
The p aper makes clear that multibrain control of a BCI game is
possible and enjoyable for the players.
III. D
ISCUSSION/CONCLUSION
In recent years, many proof-of-concept investigations have
shown that BCI can be used to control computers, therefore
they can be used in computer games. This special issue presents
a variety of examples representing the latest developments i n
BCI-based game designs and outlines progress in the field, in-
cluding designs and studies of more complex and more practical
BCI-based gam es, beyond sim ple proof-of-concept investiga-
tions. The papers presented in this special issu e have attempted
to study issues related to multitasking with BCI control, mu lti -
player BCI gaming, BCI-game design constraints, natural and
efficient integration of the BCI system and its lim itations in the
game, use of commercial EEG devices, bio/neurofeedback for
adaptive/passive gaming, BCI-game performance assessment,
and BCIs and B C I game categorization by genre and suitabi lit y
to genre design, among other interesting aspects that, together,
render this special issue special.
The p rogress outlined herein will no doubt increase the in-
terest in BCGI and make BCI-based gaming a mainstream tech-
nology of the future. This technology will offer not only enter-
tainment but will also enhance many of t he applicatio ns that
are linked with BCI and may provide assistive enabling tech-
nologies to the physically impaired, as well as provide inter-
esting and challenging activities enabling users to learn how to
modulate brain activity more proficiently. Naturally, there are
still a number of research problems that need to be solved to
increase the market penetration of BCI games. These include:
completely suppressing BCI calibration or camouflaging it in
the gam e design and story, as well as identifying th e kind of BCI
controls that are the most efficient in a gaming context and/or
the most enjoyable for the players. This also includes finding
seamless ways to train play ers during the course of the game. As
Marshall et al. suggested, to learn how to best tackle such chal-
lenges, as BCI games evolve and further studies are co ndu cted,
it w ill be important for all investigators to consider and report
the many different variables that dictate perform ance. For ex-
ample, the player’s level of BCI c on tro l proficiency (measured
as game performance and as BCI performance), the number of
sessions a user has und ertaken, types of control strategies, B CI
setup, including the number of electrodes used, types of assis-
tance within t he game and game AI, and game d istractions and
environments, along with other variables, should be reported
consistently. This will allo w assessm e nt of prog ress in th e field
on an ong oin g basis and the develo pm e nt of a clearer picture
of the best practices and best designs for BCGI. There are ex-
citing research problems ahead that BCI-based game designers
will h ave to address, affording researchers fun with computer
games while serious and beneficial research is being conducted.
D
AMIEN COYLE, Guest Editor
Intelligent Systems Research Centre
Faculty of Computing and Engineering
University of Ulster
Londonderry, BT48 7JL, U.K.
dh.coyle@ulster.ac.uk
J
OSE PRINCIPE, Guest Editor
Computational NeuroEngineering Laboratory
Department of Electrical & Computer Engineering
University of Florida
Gainesville, FL 32611-6130 USA
principe@cnel.ufl .edu
F
ABIEN LOTTE,GuestEditor
Inria Bordeaux Sud-O uest
LaBRI Potioc
Talence Cedex, 33405 France
fabien.lotte@inria.fr
A
NTON N IJHOLT, Guest Editor
Universiteit Twente
Enschede, 750 0AE The Netherland s
A.Nijholt@utwente.nl
80 IEEE TRANSACTIONS ON COMPUTATIONAL INTELLIGENCE AND AI IN GAMES, VOL. 5, NO. 2, JUNE 2013
Damien Coyle (S’04–M’05–SM’12) received a first class degree in computing and electro
nic en-
gineering and the P h.D. degree in intelligent systems engineering from the Universi
ty of Ulster,
Londonderry, U.K., in 2002 and 2006, respectively.
Since 2006, he has been a Lecturer/Senior Lecturer at the School of Computing and Inte
lligent
Systems and a mem ber of the Intelligent Systems Research Centre, University of Ulst
er, where
he is a fou nding member of the Brain–Co mp uter Interface (BCI) Team and th
e Computational
Neuroscience Research Team (CNRT). His research interests include br
ain–computer interfaces,
computational intelligence, computational neuroscience, neuroim
aging, and biomedical signal pro-
cessing. He has coauthored several journal articles and book chapte
rs in these areas.
Dr. Coyle is the 2008 recipient of the IEEE Computatio nal Intelligenc
e Soc iety ’s Outstanding
Doctoral Dissertation Aw ard and the 2011 recipient of the Internati
onal Neural Network Society’s
Young Investigator of the Year Awar d. He received the University of
Ulster ’s Distinguished Re-
search Fellowship award in 2011 and the Royal Academ y of Eng
ineering/The Leverhulme Trust
Senior Research Fellowship in 2013. He is an active volunt
eer in the IEEE Computational Intelligence Society.
Jose Principe (M’83–SM’90–F’00) received the
Engenheiro degree from the University of Porto,
Porto, Portugal, in 1972, an d the M.Sc. and Ph.
D. degrees in electrical engineering from the Uni-
versity of Florida, Gainesville, FL, USA, in
1975 an d 1979, respectively. He h old s Honoris Causa
degrees from the U niversita Mediterranea,
Reggio Calabria, Italy; Universidade do Maranhao,
Maranhao, Brazil; and Aalto Universit
y, Espoo, Finland.
He has been a D isting uished Professor of
electrical and biomedical engineering with the Uni-
versity of Florida since 2002 . H e is B el
lSouth Professor and Founding Director of the Univer-
sity of Florida Computational Neuro-E
ngineering Laboratory (CNEL). H e joined the University
of Florida in 1987, after an eight-yea
r appointment as Professor with the University of Aveiro,
Aveiro, P ortugal. He chaired 78 Ph.D
. and 61 Master student committees, and he is author of more
than 600 refereed publications (
5 books, 7 edited books, 19 book chapters, 201 j ournal papers, and
427 conference proceedings). H
e holds 22 patents and has su bm itted seven more.
Dr. Principe has been a Fellow of t
he A IMBE since 2006, and the IAMBE since 2012. He serv ed
as President o f the Internation
al Neural Network Society in 2004, as Editor-in-Chief of the IEEE T
RANSACTIONS OF BIOMEDICAL
ENGINEERIN G from 2001 to 2007, a
nd as a member of the Advisory Science Board of the FDA from 2001 to 2004. He is currently
the Founding Editor in Chief of
the IEEE Reviews in B iomedical Engineering. He has been heavily involved in conference organ-
ization and several IEEE soci
ety administrative committees. He is a recipient of the INNS Gabor Award, t he IEEE Engineering
in Medicine and Biology Soc
iety Career Achievement Award, and the IEEE Computational I ntelligence Society Neural Network
Pioneer Award.
IEEE TRANSACTIONS ON COMPUTATIONAL INTELLIGENCE AND AI IN GAMES, VOL. 5, NO. 2, JUNE 2013 81
Fabien Lotte received the M.Sc. and M.Eng. degrees and Ph.D. degree in computer sciences from
the National Institute of Applied Sciences (INSA), Rennes, France, in 2005 and 2008 respectively.
As a Ph.D. candidate, he was a member of the BUNRAKU team at the Inria Rennes Bretagne-
Atlantique and a m e m ber of the OpenViBE project dedicated to brain–computer interfaces and
virtual reality. He was supervised by Dr. A. Lecu yer and Pr. B. Arnaldi. In 2009 and 2010, he was
a Research Fellow at the I nstitute for I nfocom m Research (I2R), Singapore, working in the Brain-
Computer Interface Laboratory, led by Dr. C. Guan. Since January 2011, he has been a R esearch
Scientist (with tenure) at Inria B ordeaux Sud -O uest, Talence Cedex, France, in the Potioc t eam.
His research interests include brain–computer interfaces, v irtual reality and 3-D interaction, pattern
recognition, and signal processing .
Dr. L otte received both the Ph.D. dissertation award in 2009 from the French Association for
Pattern Recognition (AFRIF) and the Ph.D. dissertation award in 2009 (second prize) from the
French Association for Information Sciences and Technologies (ASTI).
Anton Nijholt stu died mathem a tics and computer science at the Technical University of Delft,
Delft, The Netherlands and received the Ph.D. degree in theoretical computer science from the
Vrije Universiteit, Amsterdam, The N etherlan ds, in 1980.
He is a Professor of Computer Science in the Human Media Inter action group, University of
Twente, Enschede, The Netherlands. He held positions at v arious universities in and outside The
Netherlands. His main research interests are entertainm ent computing, multim odal interaction, af-
fective co mp uting, and brain–computer interfacing. He has hundreds of scientific publications,
including (edited) boo ks on the history of computing, language processing , and brain–computer
interfacing. In preparation are books on “playful interaction” and “entertainment for the whole
world.”
Prof. N ijholt has served as Program Chair and G eneral Chair for conferences on affective com-
puting and entertainment computing an d has been Guest Editor for various j ournals.