There is a consensus that serious games have a significant potential as a tool for instruction. However, their effectiveness in terms of learning outcomes is still understudied mainly due to the complexity involved in assessing intangible measures. A systematic approach--based on established principles and guidelines--is necessary to enhance the design of serious games, and many studies lack a rigorous assessment. An important aspect in the evaluation of serious games, like other educational tools, is user performance assessment. This is an important area of exploration because serious games are intended to evaluate the learning progress as well as the outcomes. This also emphasizes the importance of providing appropriate feedback to the player. Moreover, performance assessment enables adaptivity and personalization to meet individual needs in various aspects, such as learning styles, information provision rates, feedback, and so forth. This paper first reviews related literature regarding the educational effectiveness of serious games. It then discusses how to assess the learning impact of serious games and methods for competence and skill assessment. Finally, it suggests two major directions for future research: characterization of the player's activity and better integration of assessment in games.

https://downloads.hindawi.com/journals/ahci/2013/136864.pdf

Assessment in and of serious games: an overview

In the domain of computer games, research into the interaction between player and game has centred on 'enjoyment', often drawing in particular on optimal experience research and Csikszentmihalyi's 'Flow theory'. Flow is a well-established construct for examining experience in any setting and its application to game-play is intuitive. Nevertheless, it's not immediately obvious how to translate between the flow construct and an operative description of game-play. Previous research has attempted this translation through analogy. In this article we propose a practical, integrated approach for analysis of the mechanics and aesthetics of game-play, which helps develop deeper insights into the capacity for flow within games. The relationship between player and game, characterized by learning and enjoyment, is central to our analysis. We begin by framing that relationship within Cowley's user-system-experience (USE) model, and expand this into an information systems framework, which enables a practical mapping of flow onto game-play. We believe this approach enhances our understanding of a player's interaction with a game and provides useful insights for games' researchers seeking to devise mechanisms to adapt game-play to individual players.

/pdf/toward-an-understanding-of-flow-in-video-games-2bzvemnjeu.pdf

Toward an understanding of flow in video games

Educational researchers have recognized Augmented Reality (AR) as a technology with great potential to impact affective and cognitive learning outcomes. However, very little work has been carried out to substantiate these claims. The purpose of this study was to assess to which extent an AR learning application affects learners' level of enjoyment and learning effectiveness. The study followed an experimental/control group design using the type of the application (AR-based, web-based) as independent variable. 64 high school students were randomly assigned to the experimental or control group to learn the basic principles of electromagnetism. The participants' knowledge acquisition was evaluated by comparing pre- and post-tests. The participants' level overall-state perception on flow was measured with the Flow State Scale and their flow states were monitored throughout the learning activity. Finally, participants' perceptions of benefits and difficulties of using the augmented reality application in this study were qualitatively identified. The results showed that the augmented reality approach was more effective in promoting students' knowledge of electromagnetic concepts and phenomena. The analysis also indicated that the augmented reality application led participants to reach higher flow experience levels than those achieved by users of the web-based application. However, not all the factors seem to have influence on learners' flow state, this study found that they were limited to: concentration, distorted sense of time, sense of control, clearer direct feedback, and autotelic experience. A deeper analysis of the flow process showed that neither of the groups reported being in flow in those tasks that were very easy or too difficult. However, for those tasks that were not perceived as difficult and included visualization clues, the experimental group showed higher levels of flow that the control group. The study suggests that augmented reality can be exploited as an effective learning environment for learning the basic principles of electromagnetism at high school provided that learning designers strike a careful balance between AR support and task difficulty.

/pdf/experimenting-with-electromagnetism-using-augmented-reality-4i2qn6fb5d.pdf

Experimenting with electromagnetism using augmented reality: Impact on flow student experience and educational effectiveness

Background: Cognitive tasks are typically viewed as effortful, frustrating, and repetitive, which often leads to participant disengagement. This, in turn, may negatively impact data quality and/or reduce intervention effects. However, gamification may provide a possible solution. If game design features can be incorporated into cognitive tasks without undermining their scientific value, then data quality, intervention effects, and participant engagement may be improved. Objectives: This systematic review aims to explore and evaluate the ways in which gamification has already been used for cognitive training and assessment purposes. We hope to answer 3 questions: (1) Why have researchers opted to use gamification? (2) What domains has gamification been applied in? (3) How successful has gamification been in cognitive research thus far? Methods: We systematically searched several Web-based databases, searching the titles, abstracts, and keywords of database entries using the search strategy (gamif* OR game OR games) AND (cognit* OR engag* OR behavi* OR health* OR attention OR motiv*). Searches included papers published in English between January 2007 and October 2015. Results: Our review identified 33 relevant studies, covering 31 gamified cognitive tasks used across a range of disorders and cognitive domains. We identified 7 reasons for researchers opting to gamify their cognitive training and testing. We found that working memory and general executive functions were common targets for both gamified assessment and training. Gamified tests were typically validated successfully, although mixed-domain measurement was a problem. Gamified training appears to be highly engaging and does boost participant motivation, but mixed effects of gamification on task performance were reported. Conclusions: Heterogeneous study designs and typically small sample sizes highlight the need for further research in both gamified training and testing. Nevertheless, careful application of gamification can provide a way to develop engaging and yet scientifically valid cognitive assessments, and it is likely worthwhile to continue to develop gamified cognitive tasks in the future. [JMIR Serious Games 2016;4(2):e11]

/pdf/gamification-of-cognitive-assessment-and-cognitive-training-3q7s2iz6xt.pdf

Gamification of Cognitive Assessment and Cognitive Training: A Systematic Review of Applications and Efficacy

This article reviews the psychophysiological method in game research. Psychophysiological measurements provide an objective, continuous, real-time, noninvasive, precise and sensitive way to assess the game experience. However, the best results require controlled experiments with careful monitoring of variables, large enough sample sizes and expertise in electrical signal processing. We briefly explain the theory behind the method and present the most useful measures: electromyography (EMG), electrodermal activity (EDA), electroencephalography (EEG) and cardiac measures. We review previous studies that have used psychophysiological measures in game research and illustrate some future directions. Our article covers several research lines using the psychophysiological method in game studies, and offers a comprehensive list of references for those interested in the field.

/pdf/a-review-of-the-use-of-psychophysiological-methods-in-game-35olmo9t4p.pdf

A review of the use of psychophysiological methods in game research

Learning principles and interaction design for 'Green My Place': A massively multiplayer serious game

In a study on learning in serious games, 45 players were tested for topic-comprehension by a questionnaire administered before and after solo-playing of the serious game Peacemaker (Impact Games 2007), during which their psychophysiological signals were measured. Play lasted for 1 h, with a break at half time. The questionnaire was divided into two parts, with fixed and open questions respectively. We use the Bloom taxonomy to distinguish levels of difficulty in demonstrated learning - with the first five levels assigned to fixed questions - and gain scores to measure actual value of demonstrated learning. We present the analysis of the psychophysiology recorded during game play and its relationship to learning scores. The Heart Rate Variability (HRV) (an indicator of mental workload) and interaction between HRV and electromyography of Orbicularis Oculi (an indicator of positive affect) significantly predicted the learning results at certain levels of difficulty. Results indicate that increased working-memory related mental workload in support of on-task attention aids learning at these levels.

Cardiovascular physiology predicts learning effects in a serious game activity

This paper details the central ideas to date, from a PhD entitled ‘Player Profiling for Adaptive Artificial Intelligence in Computer and Video Games’. Computer and videogames differ from other web and productivity software in that games are much more highly interactive and immersive experiences. Whereas usability and user modelling for other software may be based on productivity alone, games require an additional factor that takes account of the quality of the user experience in playing a game. In order to describe that experience we describe a model of User, System and Experience (USE) in which the primary construct for evaluation of a player’s experience will be the Experience Fluctuation Model (EFM), taken from Flow theory. We illustrate with a straightforward example how this system may be automated in real-time within a commercial game.

Ben Cowley

Papers

Toward an understanding of flow in video games

A review of the use of psychophysiological methods in game research

Learning principles and interaction design for 'Green My Place': A massively multiplayer serious game

Cardiovascular physiology predicts learning effects in a serious game activity

User-system-experience model for user centered design in computer games