Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning.

/pdf/reinforcement-learning-an-introduction-rzxgej9p17.pdf

Reinforcement Learning: An Introduction

Machine Learning is the study of methods for programming computers to learn. Computers are applied to a wide range of tasks, and for most of these it is relatively easy for programmers to design and implement the necessary software. However, there are many tasks for which this is difficult or impossible. These can be divided into four general categories. First, there are problems for which there exist no human experts. For example, in modern automated manufacturing facilities, there is a need to predict machine failures before they occur by analyzing sensor readings. Because the machines are new, there are no human experts who can be interviewed by a programmer to provide the knowledge necessary to build a computer system. A machine learning system can study recorded data and subsequent machine failures and learn prediction rules. Second, there are problems where human experts exist, but where they are unable to explain their expertise. This is the case in many perceptual tasks, such as speech recognition, hand-writing recognition, and natural language understanding. Virtually all humans exhibit expert-level abilities on these tasks, but none of them can describe the detailed steps that they follow as they perform them. Fortunately, humans can provide machines with examples of the inputs and correct outputs for these tasks, so machine learning algorithms can learn to map the inputs to the outputs. Third, there are problems where phenomena are changing rapidly. In finance, for example, people would like to predict the future behavior of the stock market, of consumer purchases, or of exchange rates. These behaviors change frequently, so that even if a programmer could construct a good predictive computer program, it would need to be rewritten frequently. A learning program can relieve the programmer of this burden by constantly modifying and tuning a set of learned prediction rules. Fourth, there are applications that need to be customized for each computer user separately. Consider, for example, a program to filter unwanted electronic mail messages. Different users will need different filters. It is unreasonable to expect each user to program his or her own rules, and it is infeasible to provide every user with a software engineer to keep the rules up-to-date. A machine learning system can learn which mail messages the user rejects and maintain the filtering rules automatically. Machine learning addresses many of the same research questions as the fields of statistics, data mining, and psychology, but with differences of emphasis. Statistics focuses on understanding the phenomena that have generated the data, often with the goal of testing different hypotheses about those phenomena. Data mining seeks to find patterns in the data that are understandable by people. Psychological studies of human learning aspire to understand the mechanisms underlying the various learning behaviors exhibited by people (concept learning, skill acquisition, strategy change, etc.).

Machine learning

How to Do Things With Words

This paper surveys the field of reinforcement learning from a computer-science perspective. It is written to be accessible to researchers familiar with machine learning. Both the historical basis of the field and a broad selection of current work are summarized. Reinforcement learning is the problem faced by an agent that learns behavior through trial-and-error interactions with a dynamic environment. The work described here has a resemblance to work in psychology, but differs considerably in the details and in the use of the word "reinforcement." The paper discusses central issues of reinforcement learning, including trading off exploration and exploitation, establishing the foundations of the field via Markov decision theory, learning from delayed reinforcement, constructing empirical models to accelerate learning, making use of generalization and hierarchy, and coping with hidden state. It concludes with a survey of some implemented systems and an assessment of the practical utility of current methods for reinforcement learning.

/pdf/reinforcement-learning-a-survey-38lmyylg2y.pdf

Reinforcement learning: a survey

This paper surveys the field of reinforcement learning from a computer-science perspective. It is written to be accessible to researchers familiar with machine learning. Both the historical basis of the field and a broad selection of current work are summarized. Reinforcement learning is the problem faced by an agent that learns behavior through trial-and-error interactions with a dynamic environment. The work described here has a resemblance to work in psychology, but differs considerably in the details and in the use of the word ``reinforcement.'' The paper discusses central issues of reinforcement learning, including trading off exploration and exploitation, establishing the foundations of the field via Markov decision theory, learning from delayed reinforcement, constructing empirical models to accelerate learning, making use of generalization and hierarchy, and coping with hidden state. It concludes with a survey of some implemented systems and an assessment of the practical utility of current methods for reinforcement learning.

Reinforcement Learning: A Survey

Robot shaping: developing autonomous agents through learning

From the Publisher:
foreword by Lashon Booker

"[This] book gives a clear and comprehensive exposition of [the authors] extensive experience in integrating reinforcement learning and autonomous robotics. Their continuing contribution is to the development of a distinct engineering discipline (`Behavior engineering') through which such robots can be created. I am excited because their efforts combine some of the best theoretical ideas with a strong eye for the practical - for what will actually work."
-- Stewart W. Wilson, The Rowland Institute for Science To program an autonomous robot to act reliably in a dynamic environment is a complex task. The dynamics of the environment are unpredictable, and the robots' sensors provide noisy input. A learning autonomous robot, one that can acquire knowledge through interaction with its environment and then adapt its behavior, greatly simplifies the designer's work. A learning robot need not be given all of the details of its environment, and its sensors and actuators need not be finely tuned. 
Robot Shaping is about designing and building learning autonomous robots. The term "shaping" comes from experimental psychology, where it describes the incremental training of animals. The authors propose a new engineering discipline, "behavior engineering," to provide the methodologies and tools for creating autonomous robots. Their techniques are based on classifier systems, a reinforcement learning architecture originated by John Holland, to which they have added several new ideas, such as "mutespec," classifier system "energy," and dynamic population size. In the book they present Behavior Analysis and Training (BAT) as an example of a behavior engineering methodology.

Robot Shaping: An Experiment in Behavior Engineering

In this paper we propose an operational method to express the meaning of the messages exchanged among agents that interact in open environments. In an open environment, like for example the Internet, agents are usually designed by different constructors, so it is very important to define the meaning of a standard, widely accepted Agent Communication Language. We express the meaning of messages using the social notion of commitment. Commitments are defined operationally within an object-oriented paradigm. We give an operational specification of the commitment class that includes the concepts of conditional commitment and precommitment. Then we use commitment objects to define the meaning of some interesting speech acts, and give an example of their use in negotiation.

/pdf/operational-specification-of-a-commitment-based-agent-f824dkbx79.pdf

Operational specification of a commitment-based agent communication language

We asked "What is a Learning Classifier System" to some of the best-known researchers in the field. These are their answers.

What Is a Learning Classifier System

We propose a method for the definition of interaction protocols to be used in open multiagent systems. Starting from the assumption that language is the fundamental component of every interaction, we first propose a semantics for Agent Communication Languages based on the notion of social commitment, and then use it to define the meaning of a set of basic communicative acts. Second, we propose a verifiable and application-independent method for the definition of interaction protocols, whose main component is the specification of an interaction diagram specifying which actions may be performed by agents under given conditions. Interaction protocols fully rely on the application-independent meaning of communicative acts. We also propose a set of soundness conditions that can be used to verify whether a protocol is reasonable. Finally, our approach is exemplified by the definition of an interaction protocol for English auctions.

/pdf/defining-interaction-protocols-using-a-commitment-based-4trbrv6si4.pdf

Marco Colombetti

Papers

Robot shaping: developing autonomous agents through learning

Robot Shaping: An Experiment in Behavior Engineering

Operational specification of a commitment-based agent communication language

What Is a Learning Classifier System

Defining interaction protocols using a commitment-based agent communication language