Review Article: Multimodal interaction: A review

Brain-computer interfaces (BCIs) and basic computer games have been interconnected since BCI development began, exploiting gameplay elements as a means of enhancing performance in BCI training protocols and entertaining and challenging participants while training to use a BCI. By providing the BCI user with an entertaining environment, researchers hope to assist users in becoming more proficient at controlling a BCI system. BCIs have been used to enrich the experience of abled-bodied and physically impaired users in various computer applications, in particular, computer games. BCI games have been reviewed previously, yet a critical evaluation of “gameplay” within BCI games has not been undertaken. Gameplay is a key aspect of any computer game and encompasses the challenges presented to the player, the actions made available to the player by the game designer to overcome the challenges and the interaction mechanism in the game. Here, the appropriateness of game genres (a category of games characterized by a particular set of gameplay challenges) and the associated gameplay challenges for different BCI paradigms is evaluated. The gameplay mechanics employed across a range of BCI games are reviewed and evaluated in terms of the BCI control strategy's suitability, considering the genre and gameplay mechanics employed. A number of recommendations for the field relating to genre-specific BCI-games development and assessing user performance are also provided for BCI game developers.

Games, Gameplay, and BCI: The State of the Art

Brain-computer interfaces (BCIs) have been explored in the field of neuroengineering to investigate how the brain can use these systems to control external devices. We review the principles and approaches we have taken to develop a sensorimotor rhythm electroencephalography (EEG)-based brain-computer interface (BCI). The methods include developing BCI systems incorporating the control of physical devices to increase user engagement, improving BCI systems by inversely mapping scalp-recorded EEG signals to the cortical source domain, integrating BCI with noninvasive neuromodulation strategies to improve learning, and incorporating mind-body awareness training to enhance BCI learning and performance. The challenges and merits of these strategies are discussed, together with recent findings. Our work indicates that the sensorimotor-rhythm-based noninvasive BCI has the potential to provide communication and control capabilities as an alternative to physiological motor pathways.

Noninvasive Brain-Computer Interfaces Based on Sensorimotor Rhythms

Transferring brain-computer interfaces beyond the laboratory

In their early days, brain–computer interfaces (BCIs) were only considered as control channel for end users with severe motor impairments such as people in the locked-in state. But, thanks to the multidisciplinary progress achieved over the last decade, the range of BCI applications has been substantially enlarged. Indeed, today BCI technology cannot only translate brain signals directly into control signals, but also can combine such kind of artificial output with a natural muscle-based output. Thus, the integration of multiple biological signals for real-time interaction holds the promise to enhance a much larger population than originally thought end users with preserved residual functions who could benefit from new generations of assistive technologies. A BCI system that combines a BCI with other physiological or technical signals is known as hybrid BCI (hBCI). In this work, we review the work of a large scale integrated project funded by the European commission which was dedicated to develop practical hybrid BCIs and introduce them in various fields of applications. This article presents an hBCI framework, which was used in studies with nonimpaired as well as end users with motor impairments.

Towards Noninvasive Hybrid Brain–Computer Interfaces: Framework, Practice, Clinical Application, and Beyond

In this paper, we describe a multimodal brain-computer interface (BCI) experiment, situated in a highly immersive CAVE. A subject sitting in the virtual environment controls the main character of a virtual reality game: a penguin that slides down a snowy mountain slope. While the subject can trigger a jump action via the BCI, additional steering with a game controller as a secondary task was tested. Our experiment profits from the game as an attractive task where the subject is motivated to get a higher score with a better BCI performance. A BCI based on the so-called brain switch was applied, which allows discrete asynchronous actions. Fourteen subjects participated, of which 50% achieved the required performance to test the penguin game. Comparing the BCI performance during the training and the game showed that a transfer of skills is possible, in spite of the changes in visual complexity and task demand. Finally and most importantly, our results showed that the use of a secondary motor task, in our case the joystick control, did not deteriorate the BCI performance during the game. Through these findings, we conclude that our chosen approach is a suitable multimodal or hybrid BCI implementation, in which the user can even perform other tasks in parallel.

/pdf/thinking-penguin-multimodal-brain-computer-interface-control-21741znozc.pdf

M. Lancelle

Papers

Thinking Penguin: Multimodal Brain–Computer Interface Control of a VR Game