The game of Go has long been viewed as the most challenging of classic games for artificial intelligence owing to its enormous search space and the difficulty of evaluating board positions and moves. Here we introduce a new approach to computer Go that uses ‘value networks’ to evaluate board positions and ‘policy networks’ to select moves. These deep neural networks are trained by a novel combination of supervised learning from human expert games, and reinforcement learning from games of self-play. Without any lookahead search, the neural networks play Go at the level of stateof-the-art Monte Carlo tree search programs that simulate thousands of random games of self-play. We also introduce a new search algorithm that combines Monte Carlo simulation with value and policy networks. Using this search algorithm, our program AlphaGo achieved a 99.8% winning rate against other Go programs, and defeated the human European Go champion by 5 games to 0. This is the first time that a computer program has defeated a human professional player in the full-sized game of Go, a feat previously thought to be at least a decade away.

/pdf/mastering-the-game-of-go-with-deep-neural-networks-and-tree-3viuxo6uxm.pdf

Mastering the game of Go with deep neural networks and tree search

For large state-space Markovian Decision Problems Monte-Carlo planning is one of the few viable approaches to find near-optimal solutions. In this paper we introduce a new algorithm, UCT, that applies bandit ideas to guide Monte-Carlo planning. In finite-horizon or discounted MDPs the algorithm is shown to be consistent and finite sample bounds are derived on the estimation error due to sampling. Experimental results show that in several domains, UCT is significantly more efficient than its alternatives.

/pdf/bandit-based-monte-carlo-planning-4s6smd0jbe.pdf

Bandit based monte-carlo planning

Monte Carlo tree search (MCTS) is a recently proposed search method that combines the precision of tree search with the generality of random sampling. It has received considerable interest due to its spectacular success in the difficult problem of computer Go, but has also proved beneficial in a range of other domains. This paper is a survey of the literature to date, intended to provide a snapshot of the state of the art after the first five years of MCTS research. We outline the core algorithm's derivation, impart some structure on the many variations and enhancements that have been proposed, and summarize the results from the key game and nongame domains to which MCTS methods have been applied. A number of open research questions indicate that the field is ripe for future work.

/pdf/a-survey-of-monte-carlo-tree-search-methods-2mj9olkdni.pdf

A Survey of Monte Carlo Tree Search Methods

A Monte-Carlo evaluation consists in estimating a position by averaging the outcome of several random continuations. The method can serve as an evaluation function at the leaves of a min-max tree. This paper presents a new framework to combine tree search with Monte-Carlo evaluation, that does not separate between a min-max phase and a Monte-Carlo phase. Instead of backing-up the min-max value close to the root, and the average value at some depth, a more general backup operator is defined that progressively changes from averaging to minmax as the number of simulations grows. This approach provides a finegrained control of the tree growth, at the level of individual simulations, and allows efficient selectivity. The resulting algorithm was implemented in a 9 × 9 Go-playing program, Crazy Stone, that won the 10th KGS computer-Go tournament.

/pdf/efficient-selectivity-and-backup-operators-in-monte-carlo-6t5e5ij712.pdf

Efficient selectivity and backup operators in Monte-Carlo tree search

AI and Games Introduction What Is AI? Model of Game AI Algorithms, Data Structures, and Representations On the Website Layout of the Book Game AI The Complexity Fallacy The Kind of AI in Games Speed and Memory The AI Engine Techniques Movement The Basics of Movement Algorithms Kinematic Movement Algorithms Steering Behaviors Combining Steering Behaviors Predicting Physics Jumping Coordinated Movement Motor Control Movement in the Third Dimension Pathfinding The Pathfinding Graph Dijkstra A* World Representations Improving on A* Hierarchical Pathfinding Other Ideas in Pathfinding Continuous Time Pathfinding Movement Planning Decision Making Overview of Decision Making Decision Trees State Machines Behavior Trees Fuzzy Logic Markov Systems Goal-Oriented Behavior Rule-Based Systems Blackboard Architectures Scripting Action Execution Tactical and Strategic AI Waypoint Tactics Tactical Analyses Tactical Pathfinding Coordinated Action Learning Learning Basics Parameter Modification Action Prediction Decision Learning Naive Bayes Classifiers Decision Tree Learning Reinforcement Learning Artificial Neural Networks Board Games Game Theory Minimaxing Transposition Tables and Memory Memory-Enhanced Test Algorithms Opening Books and Other Set Plays Further Optimizations Turn-Based Strategy Games Supporting Technologies Execution Management Scheduling Anytime Algorithms Level of Detail World Interfacing Communication Getting Knowledge Efficiently Event Managers Polling Stations Sense Management Tools and Content Creation Knowledge for Pathfinding and Waypoint Tactics Knowledge for Movement Knowledge for Decision Making The Toolchain Designing Game AI Designing Game AI The Design Shooters Driving Real-Time Strategy Sports Turn-Based Strategy Games AI-Based Game Genres Teaching Characters Flocking and Herding Games Appendix Books, Periodicals, and Papers Games

Artificial Intelligence for Games

We describe two Go programs, Olga and Oleg, developed by a Monte-Carlo approach that is simpler than Bruegmann’s (1993) approach. Our method is based on Abramson (1990). We performed experiments, to assess ideas on (1) progressive pruning, (2) all moves as first heuristic, (3) temperature, (4) simulated annealing, and (5) depth-two tree search within the Monte-Carlo framework. Progressive pruning and the all moves as first heuristic are good speed-up enhancements that do not deteriorate the level of the program too much. Then, using a constant temperature is an adequate and simple heuristic that is about as good as simulated annealing. The depth-two heuristic gives deceptive results at the moment. The results of our Monte-Carlo programs against knowledge-based programs on 9x9 boards are promising. Finally, the ever-increasing power of computers lead us to think that Monte-Carlo approaches are worth considering for computer Go in the future.

/pdf/monte-carlo-go-developments-3e1l41gr6u.pdf

Monte-carlo go developments

We present a way to integrate search and Monte-Carlo methods in the game of Go. Our program uses search to find the status of tactical goals, builds groups, selects interesting goals, and computes statistics on the realization of tactical goals during the random games. The mean score of the random games where a selected tactical goal has been reached and the mean score of the random games where it has failed are computed. They are used to evaluate the selected goals. Experimental results attest that combining search and Monte-Carlo significantly improves the playing level.

Combining Tactical Search and Monte-Carlo in the Game of Go.

Search for transitive connections

We present a new method for taking advantage of the relative independence between parts of a single-player game. We describe an implementation for improving the search in a solitaire card game called Gaps. Considering the basic techniques, we show that a simple variant of Gaps can be solved by a straightforward depth-first search (DFS); turning to variants with a larger search space, we give an approximation of the winning chances using iterative sampling. Our new method was designed to make a complete search; it improves on DFS by grouping several positions in a block, and searching only on the boundaries of the blocks. A block is defined as a product of independent sequences. We describe precisely how to detect interactions between sequences and how to deal with them. The resulting algorithm may run ten times faster than DFS, depending on the degree of independence between the subgames.

/pdf/searching-with-analysis-of-dependencies-in-a-solitaire-card-lbbkoadg8w.pdf

Searching with Analysis of Dependencies in a Solitaire Card Game

It is known that in chess, random positions are harder to memorize for humans. We here reproduce these experiments in the Asian game of Go, in which computers are much weaker than humans. We survey families of positions, discussing the relative strength of humans and computers, and then experiment random positions. The result is that computers are at the best amateur level for random positions. We also provide a protocol for generating interesting random positions (avoiding unfair situations).

https://hal.inria.fr/inria-00625815/document

Bernard Helmstetter

Papers

Monte-carlo go developments

Combining Tactical Search and Monte-Carlo in the Game of Go.

Search for transitive connections

Searching with Analysis of Dependencies in a Solitaire Card Game

Random positions in Go