Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning.

/pdf/reinforcement-learning-an-introduction-rzxgej9p17.pdf

Reinforcement Learning: An Introduction

Reinforcement learning offers to robotics a framework and set of tools for the design of sophisticated and hard-to-engineer behaviors. Conversely, the challenges of robotic problems provide both inspiration, impact, and validation for developments in reinforcement learning. The relationship between disciplines has sufficient promise to be likened to that between physics and mathematics. In this article, we attempt to strengthen the links between the two research communities by providing a survey of work in reinforcement learning for behavior generation in robots. We highlight both key challenges in robot reinforcement learning as well as notable successes. We discuss how contributions tamed the complexity of the domain and study the role of algorithms, representations, and prior knowledge in achieving these successes. As a result, a particular focus of our paper lies on the choice between model-based and model-free as well as between value-function-based and policy-search methods. By analyzing a simple problem in some detail we demonstrate how reinforcement learning approaches may be profitably applied, and we note throughout open questions and the tremendous potential for future research.

/pdf/reinforcement-learning-in-robotics-a-survey-4w4397qx9i.pdf

Reinforcement learning in robotics: A survey

The reinforcement learning paradigm is a popular way to address problems that have only limited environmental feedback, rather than correctly labeled examples, as is common in other machine learning contexts. While significant progress has been made to improve learning in a single task, the idea of transfer learning has only recently been applied to reinforcement learning tasks. The core idea of transfer is that experience gained in learning to perform one task can help improve learning performance in a related, but different, task. In this article we present a framework that classifies transfer learning methods in terms of their capabilities and goals, and then use it to survey the existing literature, as well as to suggest future directions for transfer learning work.

/pdf/transfer-learning-for-reinforcement-learning-domains-a-4d00ui6nyo.pdf

Transfer Learning for Reinforcement Learning Domains: A Survey

Reinforcement Learning in Robotics: A Survey.

https://is.tuebingen.mpg.de/fileadmin/user_upload/files/publications/Neural-Netw-2008-21-682_4867[0].pdf

2008 Special Issue: Reinforcement learning of motor skills with policy gradients

We explore the use of learning schemes in training and adapting performance on simple coordination tasks. The tasks are 1-D pole balancing. Several programs incorporating learning have already achieved this (1, S, 8): the problem is to move a cart along a short piece of track to at to keep a pole balanced on its end; the pole is hinged to the cart at its bottom, and the cart is moved either to the left or to the right by a force of constant magnitude. The form of the task considered here, after (3), involves a genuinely difficult credit-assignment problem. We use a learning scheme previously developed and analysed (1, 7) to achieve performance through reinforcement, and extend it to include changing and new requirements. For example, the length or mast of the pole can change, the bias of the force, its strength, and so on; and the system can be tasked to avoid certain regions altogether. In this way we explore the learning system's ability to adapt to changes and to profit from a selected training sequence, both of which are of obvious utility in practical robotics applications.

The results described here were obtained using a computer simulation of the pole-balancing problem. A movie will be shown of the performance of the system under the various requirements and tasks.

/pdf/training-and-tracking-in-robotics-2xtvdcxq50.pdf

Training and tracking in robotics

In the Real-Time Learning Laboratory at GTE Laboratories, machine learning algorithms are being implemented on hardware testbeds. A modified connectionist actor-critic system has been applied to a ball balancing task. The system learns to balance a ball on a beam in less than 5 min and maintains the balance. A ball can roll along a few inches of a track on a flat metal beam, which an electric motor can rotate. A computer learning system running on a PC senses the position of the ball and the angular position of the beam. The system learns to prevent the ball from reaching either end of the beam. The system has shown to be robust through sensor noise and mechanical changes; it has also generated many interesting questions for future research. >

Real-time learning: a ball on a beam

Some new directions for adaptive control theory in robotics

The topic of this annotated bibliography is connectionism, a field of computer science that has enjoyed a vast resurgence in the last ten years. Properly speaking, connectionism should be regarded as part of Artificial Intelligence, or AI, and up to some years ago it was usually so treated. The earlier entries below will make that clear. Another term for connectionism is Neural Networks, and it is being widely used, especially among the new efforts that are arising, including start-up companies.

Selected bibliography on connectionism

At GTE Laboratories, we are advancing the theory of connectionist learning architectures for real-time control while exploring their relationships to animal learning models, applications in manufacturing quality control, and VLSI implementations. We seek connectionist-network architectures with improved convergence rate and scaling properties, as assessed on simulated and actual control problems. Our primary focus is on extensions to reinforcement learning. These include adaptive critics, feature/representation adaptation in multilayer networks, hybrid connectionist/conventional controllers, and modular networks for hierarchical control. We are also extending methods for system identification, or model learning, to include internal models learned using temporal-differences. We propose the integration of reinforcement and model learning based on their relationships to dynamic programming. We are working to resolve how connectionist systems should serve as a total systems concept or as tools in a larger architecture.

Oliver G. Selfridge

Papers

Training and tracking in robotics

Real-time learning: a ball on a beam

Some new directions for adaptive control theory in robotics

Selected bibliography on connectionism

Connectionist Learning Control at GTE Laboratories