The Mood Game - How to Use the Player's Affective State in a Shoot'em up Avoiding Frustration and Boredom

TL;DR: A game logic that adapts the playing difficulty based on the player's emotional state is developed that automatically adjusts the enemy spawn rate and enemy behavior, the amount of obstacles, the number and type of power ups and the game speed to provide a smooth game play for different player skills.

Abstract: In this demo paper, we present a shoot'em up game similar to Space Invaders called the "Mood Game" that incorporates players' affective state into the game mechanics in order to enhance the gaming experience and avoid undesired emotions like frustration and boredom. By tracking emotions through facial expressions combined with self-evaluation, keystrokes and performance measures, we have developed a game logic that adapts the playing difficulty based on the player's emotional state. The implemented algorithm automatically adjusts the enemy spawn rate and enemy behavior, the amount of obstacles, the number and type of power ups and the game speed to provide a smooth game play for different player skills. The effects of our dynamic game balancing mechanism will be tested in future work.

Summary

1 INTRODUCTION

• Besides its relevance in various areas of life like reading, factory working, medical working and sports [4], Flow is also a fundamental aspect in player centered game design.
• This could happen before the game starts or at any point during the game.
• The idea behind DGB is to enhance user satisfaction and the gaming experience by automatically adjusting the difficulty level in real-time based on the player’s ability and emotional state [14].
• If the user is bored, the game increases the challenge by handicaps.

2 METHODS Game Design

• The authors developed a 2D shoot’em up game from top down perspective alike to the classic Space Invaders.
• The objective of the game is to move across the screen, shoot descending hostile shuttles, preventing them from reaching the bottom of the screen and not getting hit by any obstacles (stones, hostile shuttles and their laser beam).
• The authors created an algorithm based on different metrics that makes the game react to the user’s emotional state (see section 2).
• The first describes the immediate adjustment of the game difficulty within a level.
• Self-reporting requests are exclusively made before or after a level not to interrupt the gaming experience.

Implementation

• The authors have implemented the game using the Unity 3D game engine1 and deployed it for windows computers.
• //www.affectiva.com/product/emotion-sdk/ metrics for stress detection [11], also known as 2https.
• Unnecessary keystrokes, i.e. firing the laser beam without ammunition or during recharge time, are interpreted as arousal and integrated into the DGB algorithm.
• Thus, the emotional state will be calculated as follows: EmotionScore = 1/3[0.6(se) + 0.3(f e) + 0.1(ks)] (1) As the most significant metric, self-reporting has the strongest impact for the result, i.e. each parameter is weighted by relative percentages due to their significance.
• Finally, a score larger than 10 leads to increasing the difficulty.

Materials

• This game is built to play on computers with a regular keyboard as input device and a webcam for detecting facial expressions.
• The software itself does not have any special hardware requirements.
• Indeed, the authors do not need external sensors and additional hardware components to detect user’s emotions.

Conclusion

• The presented game is at an early stage of development and provides first rudimentary approaches to dynamically adjusting the gameplay depending on the user’s emotional state.
• The authors have yet to evaluate their application concerning game experience, emotion detection accuracy and overall impression.
• So far, the authors have conducted small play testing sessions with interviews afterwards.
• Based on the results of the game prototype to date, the authors see possible difficulties with the current approach, as facial expressions and keystrokes may not be accurate enough to entirely capture the emotions experienced during a game.
• The authors would like to continue the approach to detect user’s emotions unobtrusively without the use of external sensors, like heart rate monitors or electrodermal activity (EDA) sensors thus not disturbing game experience.

Figures

Figure 4: DGB workflow

Figure 3: Graphical visualization of emotional state

Figure 1: Game Objects: enemies, obstacles, powerUps and the player

Figure 2: Game over screen with self-reporting icons

Figure 5: Tutorial screenwith given instructions for the user

Full text

The Mood Game - How to Use the Player’s Aective
State in a Shoot’em up Avoiding Frustration and
Boredom
David Halbhuber
Media Informatics Group
University of Regensburg
Regensburg, Germany
david.halbhuber
@stud.uni-regensburg.de
Jakob Fehle
Media Informatics Group
University of Regensburg
Regensburg, Germany
jakob.fehle
@stud.uni-regensburg.de
Alexander Kalus
Media Informatics Group
University of Regensburg
Regensburg, Germany
alexander.kalus
@stud.uni-regensburg.de
Konstantin Seitz
Media Informatics Group
University of Regensburg
Regensburg, Germany
konstantin.seitz
@stud.uni-regensburg.de
Martin Kocur
Media Informatics Group
University of Regensburg
Regensburg, Germany
martin.kocur@ur.de
Thomas Schmidt
Media Informatics Group
University of Regensburg
Regensburg, Germany
thomas.schmidt@ur.de
Christian Wol
Media Informatics Group
University of Regensburg
Regensburg, Germany
christian.wol@ur.de
ABSTRACT
In this demo paper, we present a shoot’em up game similar
to Space Invaders called the "Mood Game" that incorporates
players’ aective state into the game mechanics in order to
enhance the gaming experience and avoid undesired emo-
tions like frustration and boredom. By tracking emotions
through facial expressions combined with self-evaluation,
keystrokes and performance measures, we have developed
a game logic that adapts the playing diculty based on the
player’s emotional state. The implemented algorithm auto-
matically adjusts the enemy spawn rate and enemy behavior,
the amount of obstacles, the number and type of power ups
and the game speed to provide a smooth game play for dier-
ent player skills. The eects of our dynamic game balancing
mechanism will be tested in future work.
Permission to make digital or hard copies of part or all of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for prot or commercial advantage and that copies
bear this notice and the full citation on the rst page. Copyrights for third-
party components of this work must be honored. For all other uses, contact
the owner/author(s).
MuC ’19, September 8–11, 2019, Hamburg, Germany
© 2019 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-7198-8/19/09.
https://doi.org/10.1145/3340764.3345369
CCS CONCEPTS
• Human-centered computing→
User interface program-
ming.
KEYWORDS
aective computing, gaming, dynamic game balancing, game
engineering, space invaders, emotion detection
ACM Reference Format:
David Halbhuber, Jakob Fehle, Alexander Kalus, Konstantin Seitz,
Martin Kocur, Thomas Schmidt, and Christian Wol. 2019. The
Mood Game - How to Use the Player’s Aective State in a Shoot’em
upAvoidingFrustrationandBoredom.InMensch undComputer2019
(MuC ’19), September 8–11, 2019, Hamburg, Germany. ACM, New
York, NY, USA, 4 pages. https://doi.org/10.1145/3340764.3345369
1 INTRODUCTION
First described by Mihaly Csikszentmihalyi [
3
], the psycho-
logical eect of being in the zone, called Flow, is a mental
state of being highly focused and thus feeling the strongest
immersion with a high level of enjoyment and fulllment.
Besides its relevance in various areas of life like reading,
factory working, medical working and sports [
4
], Flow is
also a fundamental aspect in player centered game design. It
is a central goal for every game designer to put the player
into a "state in which people are so involved in an activity
that nothing else seems to matter; the experience itself is so
Demos
867

MuC ’19, September 8–11, 2019, Hamburg, Germany Halbhuber et al.
enjoyable that people will do it even at great cost, for the
sheer sake of doing it" [
2
]. The question is how can we create
an appropriate game design to trigger a sense of ow?
One important concept is the Flow Zone [
1
]. Described
as the balance of a game’s challenge and the player’s skill,
the game will keep the player in the Flow Zone if the pre-
sented challenge is neither too hard nor too easy related to
the player’s ability. If the challenge is beyond the players’
skills, they tend to feel overwhelmed. If the action is not chal-
lenging, the player is not engaged. Both causes the player
to feel frustrated or bored and leads to quitting the game
and poor retention. The most common approach in game
balance design to avoid undesirable emotions like frustration
and boredom is to let the user manually select the diculty
level on a scale from "easy" over "medium" to "hard". This
could happen before the game starts or at any point during
the game. As technology and thus games grow in complex-
ity, video games evolved from relatively small interactive
applications like Pong to complex software products that
reach out a wide range of players worldwide, each of them
with dierent skills. Thus, static diculty adjustment is not
enough [
8
] and a new research eld evolved: Dynamic game
balancing (DGB).
The idea behind DGB is to enhance user satisfaction and
the gaming experience by automatically adjusting the dif-
culty level in real-time based on the player’s ability and
emotional state [
14
]. There are several methods applied in
DGB to track both parameters. They cover genre-specic
performance measures derived from in-game behavior like
damage-taken, opponent damage, power up usage, levels
completed [
5
] and inventory tracking [
7
], complex proba-
bilistic models for player and diculty progression based
on player behavior [
13
] as well as methods from aective
computing, like determining the emotional state by actions,
physiological signals and facial expressions [
9
]. Based on the
latter approaches, we present a 2D shoot’em up game with
an unobtrusive DGB system that tracks the player’s emo-
tional state by combining dierent metrics to adapt the game
play. If the user is getting frustrated, the game counteracts
by reducing the enemy spawn rate or providing new power
ups. If the user is bored, the game increases the challenge by
handicaps. That’s why we call it the "Mood Game"!
2 METHODS
Game Design
We developed a 2D shoot’em up game from top down per-
spective alike to the classic Space Invaders. A collection of
the avatar, agents and game items is illustrated in gure 1.
The objective of the game is to move across the screen, shoot
descending hostile shuttles, preventing them from reaching
the bottom of the screen and not getting hit by any obstacles
Figure 1: Game Objects: enemies, obstacles, powerUps and
the player
(stones, hostile shuttles and their laser beam). The players
control a space shuttle with two degrees of freedom (moving
up and down, moving left and right) and uses a mounted
laser cannon to defend themselves against the enemies (see
gure 5). As the game progresses, the player will face a linear
diculty enhancement with new enemies, obstacles and end
bosses. The common approach would be a pre-processed
linear diculty adjustment from level to level. However, we
created an algorithm based on dierent metrics that makes
the game react to the user’s emotional state (see section 2).
If the user shows frustration or boredom and hence crosses
the Flow Zone boundaries, the game counteracts in dierent
ways in addition to a linear diculty enhancement based on
gaming progress. The enemy spawn rate, the enemy speed,
the type of power ups and the player speed are manipu-
lated. We distinguish between immediate and level-based
balancing. The rst describes the immediate adjustment of
the game diculty within a level. For instance, if the player
is highly frustrated the "Clearwave" power up is spawned
Figure 2: Game over screen with self-reporting icons
Demos
868

The Mood Game MuC ’19, September 8–11, 2019, Hamburg, Germany
at once to destroy all enemies within the viewport. To pre-
vent cheating by intentionally express frustration to receive
the "Clearwave", we included a 20 second timeout for this
item after spawning. Furthermore, the enemy speed is ad-
justed immediately as well, but only before spawning and
the speed remains constant throughout lifetime to prevent
abrupt motion changes. In contrast, level-based balancing
reacts level to level. Before starting a level, the users report
their current emotional state from negative over neutral to
positive by selecting the appropriate icon and provides di-
rect insights about their current emotional state (see gure
2). Self-reporting requests are exclusively made before or
after a level not to interrupt the gaming experience. Conse-
quently, we do not depend solely on objective data, but also
include subjective self-reports to enhance the probability of
detecting the correct user’s emotional state. Furthermore, we
visualize the progress of the emotional state over time with a
graph on a second screen (see gure 3). We can use this data
to evaluate the emotional state of the user as a function of
the current game scenario to nd out how the gaming events
inuence the player’s emotions. We do not provide the user
with live data to distract them from playing the game.
Figure 3: Graphical visualization of emotional state
Implementation
We have implemented the game using the Unity 3D game
engine
1
and deployed it for windows computers. We have
created a DGB mechanism that calculates an aective score
from facial expressions (fe), keystrokes (ks) and player’s self
- evaluation (se). Based on the circumplex emotion model
[
12
], we focused on valence and arousal, however, we can
also detect the six basic human emotions anger, disgust, fear,
happiness, sadness and surprise as dened in Ekman’s emo-
tion model [
6
]. To do so, we use the Aectiva Emotion SDK
2
which is able to recognize up to seven emotions based on
facial expressions (fe). The Aectiva SDK processes regular
webcam recordings by means of computer vision methods
and returns probability scores for the emotional state. Ad-
ditionally, we track keystroke dynamics (ks) as potential
1
https://unity.com/
2
https://www.aectiva.com/product/emotion-sdk/
metrics for stress detection [
11
]. Unnecessary keystrokes, i.e.
ring the laser beam without ammunition or during recharge
time, are interpreted as arousal and integrated into the DGB
algorithm. The last parameter for calculating an aective
state is the user’s self-evaluation (se) before and after levels.
Thus, the emotional state will be calculated as follows:
Emoti onScore = 1/3[0.6(se) + 0.3(f e) + 0.1(ks)] (1)
Asthemost signicantmetric, self-reporting hasthe strongest
impact for the result, i.e. each parameter is weighted by rela-
tive percentages due to their signicance. Hence, equation 1
gives an emotion score between 0 and 15. We dened thresh-
olds to infer the emotional state. If the score is between 0 and
5 as an indicator for negative emotion, the game decreases
the challenge. A score of 5 up to 10 means a neutral aective
state with a regular linear diculty adjustment. Finally, a
score larger than 10 leads to increasing the diculty. Figure
4 shows the DGB system workow.
Figure 4: DGB workow
Materials
This game is built to play on computers with a regular key-
board as input device and a webcam for detecting facial
expressions. The software itself does not have any special
hardware requirements. Indeed, we do not need external sen-
sors and additional hardware components to detect user’s
emotions.
Demos
869

MuC ’19, September 8–11, 2019, Hamburg, Germany Halbhuber et al.
Figure 5: Tutorial screen with given instructions for the user
Conclusion
The presented game is at an early stage of development and
provides rst rudimentary approaches to dynamically adjust-
ing the gameplay depending on the user’s emotional state.
We have yet to evaluate our application concerning game
experience, emotion detection accuracy and overall impres-
sion. So far, we have conducted small play testing sessions
with interviews afterwards. The participants did not notice
the DGB mechanism, hence we have implemented an un-
obtrusive method to track players’ emotional state in the
rst place. Note that the accuracy of the emotion detection
algorithm has not been tested yet. Based on the results of
the game prototype to date, we see possible diculties with
the current approach, as facial expressions and keystrokes
may not be accurate enough to entirely capture the emotions
experienced during a game. To underpin the automatically
measured emotions, we attach increased signicance to self-
evaluation when calculating the emotion score. However, we
plan to integrate additional metrics going beyond the smi-
ley icons. As studies show, in-game performance measures
could be promising to identify the player’s skills and adapt
the game diculty to the skill level [
15
]. Furthermore, incor-
porating approved methods from aective computing, like
physiological feedback should be considered as well: Never-
mind
3
for example, is a horror game which uses biofeedback
to enhance the gaming experience. The gameplay is dynami-
cally adjusted based on mental stress derived from heart rate
and emotional states detected by facial expressions through
eyetracking [
10
]. However, we would like to continue the
approach to detect user’s emotions unobtrusively without
the use of external sensors, like heart rate monitors or elec-
trodermal activity (EDA) sensors thus not disturbing game
experience.
3
https://nevermindgame.com/
REFERENCES
[1]
Jenova Chen. 2007. Flow in Games (and Everything else). Com-
mun. ACM 50, 4 (April 2007), 31–34. https://doi.org/10.1145/1232743.
1232769
[2]
Mihaly Csikszentmihalyi. 1990. Flow: The Psychology of Optimal Expe-
rience.
[3]
Mihaly Csikszentmihalyi. 1997. Finding ow: The psychology of en-
gagement with everyday life. Basic Books.
[4]
Mihaly Csikszentmihalyi. 1998. Finding Flow: The Psychology of En-
gagement With Everyday Life. –144.
[5]
Anders Drachen, Magy Seif El-Nasr, and Alessandro Canossa. 2013.
Game analytics–the basics. In Game analytics. Springer, 13–40.
[6]
Paul Ekman. 1999. Basic emotions. Handbook of cognition and emotion
(1999), 45–60.
[7]
Robin Hunicke and Vernell Chapman. 2004. AI for Dynamic Diculty
Adjustment in Games.
[8]
Raph Koster. 2013. Theory of Fun for Game Design (2nd ed.). O’Reilly
Media, Inc.
[9]
Irene Kotsia, Stefanos Zafeiriou, and Spiros Fotopoulos. 2013. Aective
Gaming: A Comprehensive Survey. IEEE Computer Society Conference
on Computer Vision and Pattern Recognition Workshops, 663–670. https:
//doi.org/10.1109/CVPRW.2013.100
[10]
Erin Reynolds. 2013. Nevermind: Creating an Entertaining
Biofeedback-enhanced Game Experience to Train Users in Stress Man-
agement. In ACM SIGGRAPH 2013 Posters (SIGGRAPH ’13). ACM, New
York, NY, USA, Article 77, 1 pages. https://doi.org/10.1145/2503385.
2503469
[11]
Manuel Rodrigues, Sérgio Gonçalves, Davide Carneiro, Paulo Novais,
and Florentino Fdez-Riverola. 2013. Keystrokes and Clicks: Measuring
Stress on E-learning Students. InManagement Intelligent Systems, Jorge
Casillas, Francisco J. Martínez-López, Rosa Vicari, and Fernando De la
Prieta (Eds.). Springer International Publishing, Heidelberg, 119–126.
[12]
James Russell. 1980. A Circumplex Model of Aect. Journal of
Personality and Social Psychology 39 (12 1980), 1161–1178. https:
//doi.org/10.1037/h0077714
[13]
Su Xue, Meng Wu, John Kolen, Navid Aghdaie, and Kazi A. Zaman.
2017. Dynamic Diculty Adjustment for Maximized Engagement in
Digital Games. In Proceedings of the 26th International Conference on
World Wide Web Companion (WWW ’17 Companion). International
World Wide Web Conferences Steering Committee, Republic and Can-
ton of Geneva, Switzerland, 465–471. https://doi.org/10.1145/3041021.
3054170
[14]
Mohammad Zohaib. 2018. Dynamic Diculty Adjustment (DDA) in
Computer Games: A Review. Advances in Human-Computer Interaction
2018 (11 2018), 1–12. https://doi.org/10.1155/2018/5681652
[15]
Alexander Zook, Stephen Lee-Urban, Michael R.Drinkwater, and Mark
Riedl. 2012. Skill-based Mission Generation: A Data-driven Temporal
Player Modeling Approach. https://doi.org/10.1145/2538528.2538534
Demos
870