Showing papers by "Julian Togelius published in 2017"

Neuroevolution in Games: State of the Art and Open Challenges

Abstract: This paper surveys research on applying neuroevolution (NE) to games. In neuroevolution, artificial neural networks are trained through evolutionary algorithms, taking inspiration from the way biological brains evolved. We analyze the application of NE in games along five different axes, which are the role NE is chosen to play in a game, the different types of neural networks used, the way these networks are evolved, how the fitness is determined and what type of input the network receives. The paper also highlights important open research challenges in the field.

Procedural Content Generation via Machine Learning (PCGML)

Abstract: This survey explores Procedural Content Generation via Machine Learning (PCGML), defined as the generation of game content using machine learning models trained on existing content. As the importance of PCG for game development increases, researchers explore new avenues for generating high-quality content with or without human involvement; this paper addresses the relatively new paradigm of using machine learning (in contrast with search-based, solver-based, and constructive methods). We focus on what is most often considered functional game content such as platformer levels, game maps, interactive fiction stories, and cards in collectible card games, as opposed to cosmetic content such as sprites and sound effects. In addition to using PCG for autonomous generation, co-creativity, mixed-initiative design, and compression, PCGML is suited for repair, critique, and content analysis because of its focus on modeling existing content. We discuss various data sources and representations that affect the resulting generated content. Multiple PCGML methods are covered, including neural networks, long short-term memory (LSTM) networks, autoencoders, and deep convolutional networks; Markov models, $n$-grams, and multi-dimensional Markov chains; clustering; and matrix factorization. Finally, we discuss open problems in the application of PCGML, including learning from small datasets, lack of training data, multi-layered learning, style-transfer, parameter tuning, and PCG as a game mechanic.

AI-based playtesting of contemporary board games

Abstract: Ticket to Ride is a popular contemporary board game for two to four players, featuring a number of expansions with additional maps and tweaks to the core game mechanics. In this paper, four different game-playing agents that embody different playing styles are defined and used to analyze Ticket to Ride. Different playing styles are shown to be effective depending on the map and rule variation, and also depending on how many players play the game. The performance profiles of the different agents can be used to characterize maps and identify the most similar maps in the space of playstyles. Further analysis of the automatically played games reveal which cities on the map are most desirable, and that the relative attractiveness of cities is remarkably consistent across numbers of players. Finally, the automated analysis also reveals two classes of failures states, where the agents find states which are not covered by the game rules; this is akin to finding bugs in the rules. We see the analysis performed here as a possible template for AI-based playtesting of contemporary board games.

General video game rule generation

Abstract: We introduce the General Video Game Rule Generation problem, and the eponymous software framework which will be used in a new track of the General Video Game AI (GVGAI) competition. The problem is, given a game level as input, to generate the rules of a game that fits that level. This can be seen as the inverse of the General Video Game Level Generation problem. Conceptualizing these two problems as separate helps breaking the very hard problem of generating complete games into smaller, more manageable subproblems. The proposed framework builds on the GVGAI software and thus asks the rule generator for rules defined in the Video Game Description Language. We describe the API, and three different rule generators: a random, a constructive and a search- based generator. Early results indicate that the constructive generator generates playable and somewhat interesting game rules but has a limited expressive range, whereas the search- based generator generates remarkably diverse rulesets, but with an uneven quality.

Autoencoder-augmented Neuroevolution for Visual Doom Playing

Abstract: Neuroevolution has proven effective at many reinforcement learning tasks, but does not seem to scale well to high-dimensional controller representations, which are needed for tasks where the input is raw pixel data. We propose a novel method where we train an autoencoder to create a comparatively low-dimensional representation of the environment observation, and then use CMA-ES to train neural network controllers acting on this input data. As the behavior of the agent changes the nature of the input data, the autoencoder training progresses throughout evolution. We test this method in the VizDoom environment built on the classic FPS Doom, where it performs well on a health-pack gathering task.

Autoencoder-augmented neuroevolution for visual doom playing

Abstract: Neuroevolution has proven effective at many re-inforcement learning tasks, including tasks with incomplete information and delayed rewards, but does not seem to scale well to high-dimensional controller representations, which are needed for tasks where the input is raw pixel data. We propose a novel method where we train an autoencoder to create a comparatively low-dimensional representation of the environment observation, and then use CMA-ES to train neural network controllers acting on this input data. As the behavior of the agent changes the nature of the input data, the autoencoder training progresses throughout evolution. We test this method in the VizDoom environment built on the classic FPS Doom, where it performs well on a health-pack gathering task.

Affective evolutionary music composition with MetaCompose

Abstract: This paper describes the MetaCompose music generator, a compositional, extensible framework for affective music composition. In this context ‘affective’ refers to the music generator’s ability to express emotional information. The main purpose of MetaCompose is to create music in real-time that can express different mood-states, which we achieve through a unique combination of a graph traversal-based chord sequence generator, a search-based melody generator, a pattern-based accompaniment generator, and a theory for mood expression. Melody generation uses a novel evolutionary technique combining FI-2POP with multi-objective optimization. This allows us to explore a Pareto front of diverse solutions that are creatively equivalent under the terms of a multi-criteria objective function. Two quantitative user studies were performed to evaluate the system: one focusing on the music generation technique, and the other that explores valence expression, via the introduction of dissonances. The results of these studies demonstrate (i) that each part of the generation system improves the perceived quality of the music produced, and (ii) how valence expression via dissonance produces the perceived affective state. This system, which can reliably generate affect-expressive music, can subsequently be integrated in any kind of interactive application (e.g., games) to create an adaptive and dynamic soundtrack.

Depth in strategic games

Abstract: This paper explores the question of whether it’s possible to discover a well-defined property of game systems that corresponds to what game designers and players mean by the term “depth.” We propose a measurable property of a game’s formal system, which we call ‘d’, that corresponds to the capacity of a game to absorb dedicated problem-solving attention and allow for sustained, long-term learning. To define this property we develop a formal model that measures how susceptible a game is to partial solutions under conditions of steadily increasing computational resources. We then sketch out several directions for using the model to investigate questions about the structural properties of games that produce these effects.

Evolving Game Skill-Depth using General Video Game AI Agents

Abstract: Most games have, or can be generalised to have, a number of parameters that may be varied in order to provide instances of games that lead to very different player experiences. The space of possible parameter settings can be seen as a search space, and we can therefore use a Random Mutation Hill Climbing algorithm or other search methods to find the parameter settings that induce the best games. One of the hardest parts of this approach is defining a suitable fitness function. In this paper we explore the possibility of using one of a growing set of General Video Game AI agents to perform automatic play-testing. This enables a very general approach to game evaluation based on estimating the skill-depth of a game. Agent-based play-testing is computationally expensive, so we compare two simple but efficient optimisation algorithms: the Random Mutation Hill-Climber and the Multi-Armed Bandit Random Mutation Hill-Climber. For the test game we use a space-battle game in order to provide a suitable balance between simulation speed and potential skill-depth. Results show that both algorithms are able to rapidly evolve game versions with significant skill-depth, but that choosing a suitable resampling number is essential in order to combat the effects of noise.

DeepMasterPrint: Generating Fingerprints for Presentation Attacks

Abstract: We present two related methods for creating MasterPrints, synthetic fingerprints that are capable of spoofing multiple people’s fingerprints. These methods achieve results that advance the state-of-the-art for single MasterPrint attack accuracy while being the first methods capable of creating MasterPrints at the image level. Both of the methods presented in this paper start with training a Generative Adversarial Network (GAN) on a set of real fingerprint images. The generator network is then used to search for fingerprints that maximize the probability of matching with most subjects in a dataset. The first method uses evolutionary search in the space of latent variables, and the second method uses gradient-based optimization. The unique combination of evolution and GANs is able to design a MasterPrint that a commercial fingerprint system matches to 23% of all users in a strict security setting, and 77% of all users at a looser security setting.

Evolving Game Skill-Depth using General Video Game AI agents

Abstract: Most games have, or can be generalised to have, a number of parameters that may be varied in order to provide instances of games that lead to very different player experiences. The space of possible parameter settings can be seen as a search space, and we can therefore use a Random Mutation Hill Climbing algorithm or other search methods to find the parameter settings that induce the best games. One of the hardest parts of this approach is defining a suitable fitness function. In this paper we explore the possibility of using one of a growing set of General Video Game AI agents to perform automatic play-testing. This enables a very general approach to game evaluation based on estimating the skill-depth of a game. Agent-based play-testing is computationally expensive, so we compare two simple but efficient optimisation algorithms: the Random Mutation Hill-Climber and the Multi-Armed Bandit Random Mutation Hill-Climber. For the test game we use a space-battle game in order to provide a suitable balance between simulation speed and potential skill-depth. Results show that both algorithms are able to rapidly evolve game versions with significant skill-depth, but that choosing a suitable resampling number is essential in order to combat the effects of noise.

"Press Space to Fire": Automatic Video Game Tutorial Generation.

Abstract: We propose the problem of tutorial generation for games, i.e. to generate tutorials which can teach players to play games, as an AI problem. This problem can be approached in several ways, including generating natural language descriptions of game rules, generating instructive game levels, and generating demonstrations of how to play a game using agents that play in a human-like manner. We further argue that the General Video Game AI framework provides a useful testbed for addressing this problem.

AI as evaluator: Search driven playtesting of modern board games

Abstract: This paper presents a demonstration of how AI can be useful in the game design and development process of a modern board game. By using an artificial intelligence algorithm to play a substantial amount of matches of the Ticket to Ride board game and collecting data, we can analyze several features of the gameplay as well as of the game board. Results revealed loopholes in the game’s rules and pointed towards trends in how the game is played. We are then led to the conclusion that large scale simulation utilizing artificial intelligence can offer valuable information regarding modern board games and their designs that would ordinarily be prohibitively expensive or time-consuming to discover manually.

EvoCommander: A Novel Game Based on Evolving and Switching Between Artificial Brains

Abstract: Neuroevolution [i.e., evolving artificial neural networks (ANNs) through evolutionary algorithms] has shown promise in evolving agents and robot controllers, which display complex behaviors and can adapt to their environments. These properties are also relevant to video games, since they can increase their longevity and replayability. However, the design of most current games precludes the use of any techniques which might yield unpredictable or even open-ended results. This paper describes the game EvoCommander, with the goal to further demonstrate the potential of neuroevolution in games. In EvoCommander the player incrementally evolves an arsenal of ANN-controlled behaviors (e.g., ranged attack, flee, etc.) for a simple robot that has to battle other player and computer controlled robots. The game introduces the novel game mechanic of “brain switching,” selecting which evolved neural network is active at any point during battle. Results from playtests indicate that brain switching is a promising new game mechanic, leading to players employing interesting different strategies when training their robots and when controlling them in battle.

SeekWhence a retrospective analysis tool for general game design

Abstract: This paper describes the design of SeekWhence, a retrospective analysis tool for gameplay session. SeekWhence is a new addition to the Cicero AI-assisted game design tool, which is built on top of the Video Game Description Language (VGDL) and the General Video Game Framework (GVG-AI). With SeekWhence, designers can prototype their games and record gameplay sessions simulated by agents or human players. They can go back and forth on every frame of the recorded session, analyzing it step by step and import it into their current project to edit it. This paper explains the technical details of SeekWhence and gives examples of its usage.

Evolving game-specific UCB alternatives for general video game playing

Abstract: At the core of the most popular version of the Monte Carlo Tree Search (MCTS) algorithm is the UCB1 (Upper Confidence Bound) equation. This equation decides which node to explore next, and therefore shapes the behavior of the search process. If the UCB1 equation is replaced with another equation, the behavior of the MCTS algorithm changes, which might increase its performance on certain problems (and decrease it on others). In this paper, we use genetic programming to evolve replacements to the UCB1 equation targeted at playing individual games in the General Video Game AI (GVGAI) Framework. Each equation is evolved to maximize playing strength in a single game, but is then also tested on all other games in our test set. For every game included in the experiments, we found a UCB replacement that performs significantly better than standard UCB1. Additionally, evolved UCB replacements also tend to improve performance in some GVGAI games for which they are not evolved, showing that improvements generalize across games to clusters of games with similar game mechanics or algorithmic performance. Such an evolved portfolio of UCB variations could be useful for a hyper-heuristic game-playing agent, allowing it to select the most appropriate heuristics for particular games or problems in general.

DeepMasterPrints: Generating MasterPrints for Dictionary Attacks via Latent Variable Evolution

Abstract: Recent research has demonstrated the vulnerability of fingerprint recognition systems to dictionary attacks based on MasterPrints. MasterPrints are real or synthetic fingerprints that can fortuitously match with a large number of fingerprints thereby undermining the security afforded by fingerprint systems. Previous work by Roy et al. generated synthetic MasterPrints at the feature-level. In this work we generate complete image-level MasterPrints known as DeepMasterPrints, whose attack accuracy is found to be much superior than that of previous methods. The proposed method, referred to as Latent Variable Evolution, is based on training a Generative Adversarial Network on a set of real fingerprint images. Stochastic search in the form of the Covariance Matrix Adaptation Evolution Strategy is then used to search for latent input variables to the generator network that can maximize the number of impostor matches as assessed by a fingerprint recognizer. Experiments convey the efficacy of the proposed method in generating DeepMasterPrints. The underlying method is likely to have broad applications in fingerprint security as well as fingerprint synthesis.

Can you feel it?: evaluation of affective expression in music generated by MetaCompose

Abstract: This paper describes an evaluation conducted on the MetaCompose music generator, which is based on evolutionary computation and uses a hybrid evolutionary technique that combines FI-2POP and multi-objective optimization. The main objective of MetaCompose is to create music in real-time that can express different mood-states. The experiment presented here aims to evaluate: (i) if the perceived mood experienced by the participants of a music score matches intended mood the system is trying to express and (ii) if participants can identify transitions in the mood expression that occur mid-piece. Music clips including transitions and with static affective states were produced by MetaCompose and a quantitative user study was performed. Participants were tasked with annotating the perceived mood and moreover were asked to annotate in real-time changes in valence. The data collected confirms the hypothesis that people can recognize changes in music mood and that MetaCompose can express perceptibly different levels of arousal. In regards to valence we observe that, while it is mainly perceived as expected, changes in arousal seems to also influence perceived valence, suggesting that one or more of the music features MetaCompose associates with arousal has some effect on valence as well.

Showdown AI competition

Abstract: We present the Showdown AI Competition, a game-based AI competition built around a clone of the popular game Pokemon. This is a game of turn-based team battle, where the objective is to defeat an opponent team using clever combinations of creatures and their abilities. The gameplay is reminiscent of computer role-playing game battles and collectible card games. The game has characteristics, such as the combination of turn- based gameplay and partial observability, that are unusual in current game-based AI competitions and therefore offers a fresh challenge.

Deep Learning for Video Game Playing

Abstract: In this article, we review recent Deep Learning advances in the context of how they have been applied to play different types of video games such as first-person shooters, arcade games, and real-time strategy games. We analyze the unique requirements that different game genres pose to a deep learning system and highlight important open challenges in the context of applying these machine learning methods to video games, such as general game playing, dealing with extremely large decision spaces and sparse rewards.

Predictive Physics Simulation in Game Mechanics

Abstract: Computers can now simulate simple game physics systems hundreds of times faster than real-time, which enables real-time prediction and visualization of the effects of player actions. Predictive simulation is traditionally used as part of planning and game AI algorithms; we argue that it presents untapped potential for game mechanics and interfaces. We explore this notion through 1) deriving a four-quadrant design space model based on game design and human motor control literature, and 2) developing and evaluating six novel prototypes that demonstrate the potential and challenges of each quadrant. Our work highlights opportunities in enabling direct control of complex simulated characters, and in transforming real-time action into turn-based puzzles. Based on our results, adding predictive simulation to existing game mechanics is less promising, as it may feel alienating or make a game too easy. However, the approach may still be useful for game designers, for example, as it allows one to test designs beyond one's playing skill.

Ethical considerations for player modeling

Abstract: In this paper we discuss some of the ethical challenges that may arise from player modeling. Player modeling is used in modern games e.g. to enable various kinds of game play, to optimize games for specific players, and to maximize the monetization of games. In this paper, we propose that applying player modeling implies serious ethical questions, since it impacts how players spend their leisure time and money, affects their social relations, and changes computer games as ethical artifacts. We source categories of ethical issues in the application of artificial intelligence (AI) from work on AI ethics and using these we provide several specific examples of ethical issues in player modeling. Building from the examples, we suggest establishing a framework for understanding ethical issues in player modeling and we propose a number of methodological approaches to address the identified chal-

DeepMasterPrint: Fingerprint Spoofing via Latent Variable Evolution

Abstract: Recent research has demonstrated the vulnerability of fingerprint recognition systems to dictionary attacks based on MasterPrints. MasterPrints are real or synthetic fingerprints that can fortuitously match with a large number of fingerprints thereby undermining the security afforded by fingerprint systems. Previous work by Roy et al. generated synthetic MasterPrints at the feature-level. In this work we generate complete image-level MasterPrints known as DeepMasterPrints, whose attack accuracy is found to be much superior than that of previous methods. The proposed method, referred to as Latent Variable Evolution, is based on training a Generative Adversarial Network on a set of real fingerprint images. Stochastic search in the form of the Covariance Matrix Adaptation Evolution Strategy is then used to search for latent input variables to the generator network that can maximize the number of impostor matches as assessed by a fingerprint recognizer. Experiments convey the efficacy of the proposed method in generating DeepMasterPrints. The underlying method is likely to have broad applications in fingerprint security as well as fingerprint synthesis.

A comparative sequence analysis of career paths among knowledge workers in a multinational bank

Abstract: This study examined two metrics for measuring the distance between sequences (Euclid and OMSpell) and creating distance matrices combined with two types of clustering methods (AGNES and PAM) to analyze the career path clusters of knowledge workers. A regression tree of covariates and career path clusters was used to predict advancement rates. The results indicated that the metric which focused on subsequences (OMSpell) worked best for both clustering methods. Less time as a knowledge worker was associated with greater advancement. Implications for boundaryless careers and social capital formation are discussed.

Multi-objective Adaptation of a Parameterized GVGAI Agent Towards Several Games

Abstract: This paper proposes a benchmark for multi-objective optimization based on video game playing. The challenge is to optimize an agent to perform well on several different games, where each objective score corresponds to the performance on a different game. The benchmark is inspired from the quest for general intelligence in the form of general game playing, and builds on the General Video Game AI GVGAI framework. As it is based on game-playing, this benchmark incorporates salient aspects of game-playing problems such as discontinuous feedback and a non-trivial amount of stochasticity. We argue that the proposed benchmark thus provides a different challenge from many other benchmarks for multi-objective optimization algorithms currently available. We also provide initial results on categorizing the space offered by this benchmark and applying a standard multi-objective optimization algorithm to it.

Marahel: A Language for Constructive Level Generation.

Abstract: Marahel is a language and framework for constructive generation of 2D tile-based game levels. It is developed with the dual aim of making it easier to build level generators for game developers, and to help solving the general level generation problem by creating a generator space that can be searched using evolution. We describe the different sections of the level generators, and show examples of generated maps from 5 different generators. We analyze their expressive range on three dimensions: percentage of empty space, number of isolated elements, and cell-wise entropy of empty space. The results show that generators that have starkly different output from each other can easily be defined in Marahel.

Primal-improv: Towards co-evolutionary musical improvisation

Abstract: This paper describes a work in progress on co-evolving Artificial Neural Networks (ANNs) for music improvisation. Using this neuro-evolutionary approach the ANNs adapt to the changes in the human player's music as input, while still maintaining some of the structure of the musical piece previously evolved. The system is called Primal-Improv and evolves modules that are composed of two ANNs, one controlling pitch and one controlling rhythm. The results of a quantitative study show that, by only introducing simple rules as fitness functions, the system is able to generate more interesting arrangements than ANNs evolved without a specific objective. The emerging and interesting musical patterns that are produced by the evolved ANNs hint at the promising potential of the system.