Neurophysiological methods for monitoring brain activity in serious games and virtual environments: a review

TL;DR: The opportunities offered and challenges posed by neuroscientific methods when capturing user feedback and using the data to create greater user adaptivity in game are explored.

Abstract: The use of serious games and virtual environments for learning is increasing worldwide. These technologies have the potential to collect live data from users through game play and can be combined with neuroscientific methods such as EEG, fNIRS and fMRI. The several learning processes triggered by serious games are associated with specific patterns of activation that distributed in time and space over different neural networks. This paper explores the opportunities offered and challenges posed by neuroscientific methods when capturing user feedback and using the data to create greater user adaptivity in game. Existing neuroscientific studies examining cortical correlates of game-based learning do not form a common or homogenous field. In contrast, they often have disparate research questions and are represented through a broad range of study designs and game genres. In this paper, the range of studies and applications of neuroscientific methods in game-based learning are reviewed.

Summary

Keywords:

• Neurophysiological methods; brain; serious games; games; virtual environments; virtual reality; NIRS; near infrared spectroscopy; EEG; electroencephalography; fMRI; functional magnetic resonance imaging; neuroscience; learning; game based learning.
• Neuroscientific methods can be used to collect brain activity and then be used as feedback to the user, allowing the game to adapt to the user’s interactions and inputs.
• An extensive examination of the scientific literature was undertaken.
• Regarding the assessment of the efficacy of game-based approaches, action video games have been shown to enhance a wide variety of perceptual skills such as visual selective attention (Green and Bavelier, 2003) and spatial attention and mental rotation (Feng, Spence and Pratt, 2007).
• The issue of how user inputs can be fed back into the game adaptivity is an emerging line of game-related research not only to provide efficacy but also as an adaptive element in HCI and games.

2.1. EEG

• With EEG, electrical activity of the brain can be recorded non-invasively at the scalp surface, which reflects the summed potential of ionic currents across membranes of single cells, thus a direct method of measuring brain activity.
• These stimuli can elicit ERPs in the EEG.
• General effects of interaction with virtual environments on nervous system activity have already been explored in the 1990s.
• Furthermore, EEG oscillations are useful for assessing attention, concentration, fatigue, and interest during playing games (Yamada, 1998).
• By collaborative problem solving they learn in the first place to integrate and coordinate their collective knowledge.

2.2. fMRI

• FMRI is a popular non-invasive neuroscientific method.
• The information gained from fMRI can be used, for example, to improve the effectiveness of serious games or how games affect the player’s brain.
• Klasen and colleagues (2012) argued that the activation in sensory and motor networks could underpin the central role of simulation for flow experience.
• Williams and colleagues (2005) identified a number of clusters of additional activity in the joint attention settings in comparison to non-joint attention conditions.
• Shams and Seitz (2008) advocated multisensory teaching approaches as mirroring more closely evolved learning processes, suggesting unisensory approaches are sub-optimal and that their selection is based in practicality rather than pedagogy.

2.3. fNIRS

• FNIRS can be used to explore the functional activation of the human cerebral cortex.
• As a result, deoxygenated haemoglobin decreases, whereas oxygenated haemoglobin increases in the active brain region.
• This circumstance is the major source of the BOLD contrast as measured with fMRI (Huettel, Song, & McCarthy, 2009; Telkemeyer et al., 2011).
• FNIRS applications in game-based learning: Many game studies used fNIRS to examine haemodynamic changes in frontal brain areas during playing video games.
• The majority found a decrease in oxygenated-haemoglobin during gaming which might result from attention demand or task load from the video games (Izzetoglu et al., 2004; Matsuda and Hiraki, 2006; Nagamitsu et al., 2006).

2.4. Pros and cons of different neurophysiological methods

• Different neurophysiological methods can be used for monitoring and assessing cognitive processes in serious games and virtual environments.
• Compared to fMRI, which is locally bounded to the installation site, fNIRS and EEG are more flexible and portable systems.
• Finally, it should be noted that the costs of the different neurophysiological methods are quite different.
• If examining brain responses to fast events of a game, the EEG with its high temporal resolution should be the method of choice.

3. Discussion

• One of the issues emerging from their review and on-going pilot studies is that there is a lack of a common methodological approach in the use and validity assessment of neuroscientific methods in serious games or games in general.
• Many decisions needs to be taken before conducting neurophysiological measurements, therefore it is essential to have a clear experimental design and hypotheses for psychophysiological studies.
• Especially for explorative studies, a simple game design with only few variables should be used to be able to clearly identify which brain processes are involved or which skills are needed during gaming.
• Study designs under such a paradigm should focus on defining and understanding expertise at a task established through an understanding of the prior knowledge or measured performance of subjects.
• An increased presence experience in a VE should foster the transfer of knowledge acquired in the virtual environment to corresponding real world behaviour (Slater et al., 1996).

4. Summary and Conclusions

• The paper summarises some of the leading scientific research studies in the emerging field of neuroscience applied to gaming where EEG, fMRI and fNIRS are used to investigate the learner‘s brain activity during game play.
• The findings indicate that there is a need to focus study designs in a particular way , as addressed in the discussion, to ensure that these are effective, focused on direct task performance and thereby narrowing the variables.
• The review indicates that there are key benefits in using these neuroscientific techniques for developing serious games, particularly to provide better user feedback and allow designers to get a better understanding of the neurophysiological outcomes of the learner during learning periods and tasks.
• The state of the art clearly shows the potential for using these devices.
• Yet, there are ways to reach a balance between both things.

Figures

Figure 1: Visualization of the planning process of an experiment and summing up relevant points from the discussion

Table 1: Neuroscientific studies for monitoring brain activity in serious games and virtual environments

Full text

MURDOCH RESEARCH REPOSITORY
This is the author’s final version of the work, as accepted for publication
following peer review but without the publisher’s layout or pagination.
http://dx.doi.org/10.1504/IJTEL.2014.060022
Ninaus, M., Kober, S.E., Friedrich, E.V.C., Dunwell, I., de Freitas,
S., Arnab, S., Ott, M., Kravcik, M., Lim, T., Louchart, S., Bellotti,
F., Hannemann, A., Thin, A.G., Berta, R., Wood, G. and Neuper,
C. (2014) Neurophysiological methods for monitoring brain
activity in serious games and virtual environments: a review.
International Journal of Technology Enhanced Learning,
6 (1). pp. 78-103.
http://researchrepository.murdoch.edu.au/25902/
© 2013 British Educational Research Association.
It is posted here for your personal use. No further distribution is permitted.

Neurophysiological methods for monitoring brain activity in serious games and virtual
environments: A review.
Authors: Manuel Ninaus, Silvia Erika Kober, Elisabeth V.C. Friedrich, Ian Dunwell, Sara de
Freitas, Sylvester Arnab, Michela Ott, Milos Kravcik, Theodore Lim, Sandy Louchart,
Francesco Bellotti, Anna Hannemann, Alasdair G. Thin, Riccardo Berta, Guilherme Wood,
Christa Neuper
Manuel Ninaus:
M.Sc., PhD-Student
Department of Psychology, Section Neuropsychology
University of Graz
Universitätsplatz 2/III
8010 Graz
Austria
manuel.ninaus@uni-graz.at
Manuel Ninaus is Research Assistant and PhD-candidate at the Department of Psychology (Section
Neuropsychology) at the University of Graz, Austria. He received his MSc in Psychology from
University of Graz, Austria in 2012. Since 2010 he works as Research Assistant at the Department of
Psychology and at the University of Graz and is involved in national and international scientific
projects. He is currently involved in the EU GALA Network of Excellence (www.galanoe.eu). His
research topics are neuronal plasticity through learning, EEG-based neurofeedback and auditory
mirror neurons. Furthermore Manuel Ninaus has many years of experience with different
neurophysiological methods such as EEG, NIRS and fMRI.
Silvia Erika Kober:
Dr., Postdoctoral researcher
Department of Psychology, Section Neuropsychology
University of Graz
Universitätsplatz 2/III
8010 Graz
Austria
silvia.kober@uni-graz.at
Silvia Erika Kober is a postdoctoral researcher at the Department of Psychology (Section
Neuropsychology) at the University of Graz, Austria. Born in Graz, Austria, she received her MSc in
Psychology from University of Graz, Austria in 2009 and obtained the Ph.D. degree from the
University of Graz, Austria in 2012. Since 2007 she works as Research Assistant at the Department of
Psychology (Section Neuropsychology) at the University of Graz and is involved in national and
international scientific projects. Her research topics are EEG/NIRS-based brain-computer
communication and neurofeedback, interacting in virtual realities, virtual reality as rehabilitation tool
and spatial cognition.
Elisabeth V.C. Friedrich:
Dr., Postdoctoral researcher
Department of Psychology, Section Neuropsychology

University of Graz
Universitätsplatz 2/III
8010 Graz
Austria
elisabeth.friedrich@uni-graz.at
Elisabeth V.C. Friedrich studied Psychology at the University of Graz, Austria, and conducted her
master thesis at the Laboratory of Neural Injury and Repair, Wadsworth Center, Albany, NY. She has
received her doctorate in natural science in 2012 at the University of Graz, Austria. Her main research
interest is brain-computer interface (BCI). She explored different mental tasks to control an EEG-
based BCI, the impact of distraction on user performance as well as improvements of BCI usability
for severely motor impaired individuals. She is currently involved in the EU GALA Network of
Excellence (www.galanoe.eu).
Ian Dunwell:
Dr., Postdoctoral researcher
The Serious Games Institute
Coventry University Technology Park
Innovation Village
Cheetah Road
Coventry
CV1 2TL
United Kingdom
IDunwell@cad.coventry.ac.uk
Ian Dunwell is a senior researcher at the Serious Games Institute. He holds a BSc in Physics from
Imperial College London and a PhD in Computer Science from the University of Hull. He has
responsibility for a wide portfolio of health-related Serious Game research and development projects.
He is also the Technical Lead for the Applied Research Group.
Sara de Freitas:
Prof., Director of Research at the Serious Games Institute
The Serious Games Institute
Coventry University Technology Park
Innovation Village
Cheetah Road
Coventry
CV1 2TL
United Kingdom
s.defreitas@coventry.ac.uk
Sara de Freitas is director of research at the Serious Games Institute, Professor of Virtual
Environments and Fellow of the Royal Society of Arts. Sara leads the applied research team at the
Serious Games Institute and the Serious Games and Virtual Worlds Applied Research Group at
Coventry University.
Sylvester Arnab:
Senior researcher, PhD
The Serious Games Institute
Coventry University Technology Park
Innovation Village

Cheetah Road
Coventry
CV1 2TL
United Kingdom
s.arnab@coventry.ac.uk
Sylvester Arnab is a senior researcher at the SGI. He holds a doctorate from The University of
Warwick and he is coordinating the SGI's contribution to the EU Funded Games and Learning
Alliance (GALA).
Michela Ott:
Dr., Researcher
Istituto Tecnologie Didattiche (ITD)
Consiglio Nazionale delle Ricerche (CNR)
Via De Marini, 6
16149 Genova
Italy
ott@itd.cnr.it
Michela Ott is a senior researcher at the Institute for Educational Technology of Italian National
Research Council. She carries out research in the fields of: cognitive processes underpinning learning,
educational use of software tools (with a special focus on Serious Games), special education, e-
inclusion, learning design, distance education. She is the author of more than 150 scientific
publications. She is also the author of educational software packages and hypermedia systems. She
has participated in, led and coordinated a number of national and international research projects in the
field of Educational Technology. At present she is the coordinator of the MAGICAL (MAking Games
in CollaborAtion for Learning- KA3 LLP project), is involved in the eSG LLP project (Stimulating
Entrepreneurship through Serious games-Erasmus FEXI) and in three European networks: GaLA
(Game and Learning Alliance); V-MuST.net (Virtual Museum Transnational Network) and ETNA
(European Thematic Network on Assistive Information and Communication Technologies).
Milos Kravcik:
Dr., Research Associate
Informatik 5 – Information Systems
RWTH Aachen University
Ahornstr. 55
52056 Aachen
Germany
kravcik@dbis.rwth-aachen.de
Milos Kravcik has a diploma degree in computer science and a doctoral degree in applied informatics
from the Comenius University in Slovakia. He has been dealing with Technology Enhanced Learning
(TEL) since 1988 in various national and international projects, later also at the Fraunhofer Institute
for Applied Information Technology in Germany and at the Open University in the Netherlands.
Since 2010 he works as a Research Fellow at the RWTH Aachen University and his main research
interests include personalized learning environments and self-regulated learning. He co-organized
several TEL doctoral schools and serves also as executive peer-reviewer or editorial board member
for several journals related to learning technologies.
Theodore Lim:

Dr., Lecturer
School of Engineering & Physical Sciences; Mechanical Engineering
NS G.04, Heriot-Watt University
Edinburgh
EH14 4AS
United Kingdom
t.lim@hw.ac.uk
Theodore Lim is an active member within the Digital Tools Group; part of the EPSRC-funded
Innovative Manufacturing Research Centre (IMRC) at Heriot-Watt University (www.smi.hw.ac.uk).
As both an academic and researcher with considerable industrial experience, he has been instrumental
in the research, analysis and development of virtual engineering environments in a variety of product
engineering domains and now focuses his work on the acquisition of engineering knowledge
information management systems within all aspect of product engineering; with a particular emphasis
on conceptual design. He has also implemented game-based learning methods in design and
manufacturing taught courses. With over 40 international publications, a book and the successful
commercialisation of his novel feature recognition algorithms, he is now applying his knowledge and
expertise to the domain of serious games, game ware and computational biometrics for next
generation engineering applications.
Sandy Louchart:
Dr, Lecturer
School of Mathematical & Computer Sciences; Computer Science
EM1.38, Heriot-Watt University
Edinburgh
EH14 4AS
United Kingdom
s.louchart@hw.ac.uk
Sandy Louchart is a lecturer in Human Computer Interaction and Interaction Design in the School of
Mathematical and Computer Sciences (MACS) at Heriot-Watt University. His PhD, awarded by the
University of Salford in 2007, explored the domain of Interactive Storytelling (IS) via the
development of the Emergent Narrative concept. The work was conducted in the domains of Artificial
Intelligence, Synthetic Characters and Interactive Narratives and involved the design of Autonomous
Synthetic Characters and their affect-based action/selection mechanisms to simulate characterisation
within an interactive drama scenario. The research led to the development of a novel approach to
Synthetic Character action/selection mechanisms featuring projection-based affective planning so as
to select dramatically intense actions and events for interaction; the Double Appraisal action/selection
mechanism. His work has been published internationally in Intelligent Agents, Virtual Reality, Game
and Interactive Storytelling Journals and Conferences.
Francesco Bellotti:
Assistant Professor, Dr.
ELIOS Lab; Department of Naval, Electrical, Electronic and Telecommunications Engineering
(DITEN)
University of Genoa
Via Opera Pia 11a
16145 Genova
Italy
franz@elios.unige.it

Frequently asked questions

Q1. What are the contributions in this paper?

This paper explores the opportunities offered and challenges posed by neuroscientific methods when capturing user feedback and using the data to create greater user adaptivity in-game. In contrast, they often have disparate research questions and are represented through a broad range of study designs and game genres. In this article, the range of studies and applications of neuroscientific methods in gamebased learning are reviewed. 

Q2. What are the future works in this paper?

Their ambition is to extend the scope of how neurofeedback is currently used in games, as real time data that can be used to support scaffolded learning and also to allow us to understand more about how game play supports effective learning. Future work will therefore focus upon user studies where simple and complex learning activities are tested using neurophysiological correlates according to direct task performance with different neurophysiological methods. However, the authors strongly argue that it is hard time to study a rigorous, generalizable method for capturing neuroscientific user data from a player and store it in a real-time executable user model, also extending existing models of feedback ( e. g. Dunwell, de Freitas, and Jarvis, 2011 ). Further future research will focus on testing how feedback can be used in real time to make the learner experience more coherent and immersive, as well as recording levels of presence and engagement. 

Q3. What is the main lesson for serious game design?

The main lesson for serious game design here is that feedback to the learner needs to be formative in games and that multisensory environment can be more effective as a design strategy for effective learning with games. 

Q4. What is the challenge to neurophysiological measurements?

A particular challenge to neurophysiological measurements is confirming their measured outputs provide true proxies for learning efficacy. 

Q5. What is the role of the researcher in the network of excellence for serious games?

Within the EU-founded Network of Excellence for Serious Games (GALA, www.galanoe.eu), she works on serious gaming in context of neuroscience, community nurturing and interactive storytelling. 

Q6. What is the main argument for a study design under a serious game paradigm?

Study designs under such a paradigm should focus on defining and understanding expertise at a task established through an understanding of the prior knowledge or measured performance of subjects. 

Q7. What is the main purpose of fNIRS?

fNIRS applications in game-based learning: Many game studies used fNIRS to examine haemodynamic changes in frontal brain areas during playing video games. 

Q8. What is the role of the parietal brain in the presence experience?

Baumgartner and colleagues (2006) and Kober, Kurzmann and Neuper (2012) concluded that parietal brain areas might play an important role in the presence experience because these areas are involved in generating an egocentric (body-centred) representation of space (Maguire et al., 1998; Maguire, Burgess and O’Keefe, 1999). 

Q9. What is his role in the Erasmus Lifelong Learning project?

He is in the coordinating team of Games and Learning Alliance (GaLA), the FP7 Network of Excellence on serious games within technology enhanced learning, and of eSG, an Erasmus Lifelong Learning project aimed at promoting entrepreneurship in higher education through serious games. 

Q10. What is the main argument for a study design for serious games?

Serious games can form a particularly effective basis for this, as game design lends itself easily to the definition and assessment of simple cognitive tasks, as well as their deployment in an experimental context, with a wide range of user performance data collection. 

Q11. What can be used to improve the effectiveness of serious games?

The information gained from fMRI can be used, for example, to improve the effectiveness of serious games or how games affect the player’s brain. 

Q12. What is the method of enhancing game adaptivity?

In the context of using neuroscientific methods to enhance game adaptivity to the user, again the high temporal resolution of EEG is the most advantageous, since there is no time delay when the electrical activity of the brain is fed back to the gamer in real time during interacting with a game. 

Q13. How can the authors identify what cognitive processes or factors are involved in one specific task?

By theory-based analysing the subjects’ task with psychometric methods, it is possible to identify what cognitive processes or factors are involved in one specific task. 

Q14. What can be used for monitoring and assessing cognitive processes in serious games and virtual environments?

Different neurophysiological methods can be used for monitoring and assessing cognitive processes in serious games and virtual environments. 

Q15. What is the difference between fNIRS and fMRI?

fNIRS allows for the recording of changes in the BOLD response with a much higher temporal resolution than fMRI but at the costs of lower spatial resolution and no sensitivity to haemodynamic activity in deep brain regions. 

Q16. What is the common type of relationship between oxygenation levels and task difficulty?

An inverted U-type relationship where oxygenation levels increased with increasing task difficulty until it became too difficult and then they started to decline was observed. 

Q17. What is the main argument for the study design of serious games?

The authors thus argue that effective study design for serious games should avoid such dilemmas by focussing specifically on direct task performance as a correlate of device measurements. 

Q18. What is the main challenge to generalizing these assertions?

Generalizing these assertions, however, is challenging, given the task-specificity of many studies, and the complexity of the underlying physiological system.