This paper surveys the field of reinforcement learning from a computer-science perspective. It is written to be accessible to researchers familiar with machine learning. Both the historical basis of the field and a broad selection of current work are summarized. Reinforcement learning is the problem faced by an agent that learns behavior through trial-and-error interactions with a dynamic environment. The work described here has a resemblance to work in psychology, but differs considerably in the details and in the use of the word "reinforcement." The paper discusses central issues of reinforcement learning, including trading off exploration and exploitation, establishing the foundations of the field via Markov decision theory, learning from delayed reinforcement, constructing empirical models to accelerate learning, making use of generalization and hierarchy, and coping with hidden state. It concludes with a survey of some implemented systems and an assessment of the practical utility of current methods for reinforcement learning.

/pdf/reinforcement-learning-a-survey-38lmyylg2y.pdf

Reinforcement learning: a survey

This paper surveys the field of reinforcement learning from a computer-science perspective. It is written to be accessible to researchers familiar with machine learning. Both the historical basis of the field and a broad selection of current work are summarized. Reinforcement learning is the problem faced by an agent that learns behavior through trial-and-error interactions with a dynamic environment. The work described here has a resemblance to work in psychology, but differs considerably in the details and in the use of the word ``reinforcement.'' The paper discusses central issues of reinforcement learning, including trading off exploration and exploitation, establishing the foundations of the field via Markov decision theory, learning from delayed reinforcement, constructing empirical models to accelerate learning, making use of generalization and hierarchy, and coping with hidden state. It concludes with a survey of some implemented systems and an assessment of the practical utility of current methods for reinforcement learning.

Reinforcement Learning: A Survey

In this communication, we give an overview of our work on an asynchronous brain-computer interface (where the subject makes self-paced decisions on when to switch from one mental task to the next) that responds every 0.5 s. A local neural classifier tries to recognize three different mental tasks; it may also respond "unknown" for uncertain samples as the classifier has incorporated statistical rejection criteria. We report our experience with 15 subjects. We also briefly describe two brain-actuated applications we have developed: a virtual keyboard and a mobile robot (emulating a motorized wheelchair).

/pdf/asynchronous-bci-and-local-neural-classifiers-an-overview-of-2w5o5mlu5w.pdf

Asynchronous BCI and local neural classifiers: an overview of the adaptive brain interface project

A computational model of hippocampal activity during spatial cognition and navigation tasks is presented. The spatial representation in our model of the rat hippocampus is built on-line during exploration via two processing streams. An allothetic vision-based representation is built by unsupervised Hebbian learning extracting spatio-temporal properties of the environment from visual input. An idiothetic representation is learned based on internal movement-related information provided by path integration. On the level of the hippocampus, allothetic and idiothetic representations are integrated to yield a stable representation of the environment by a population of localized overlapping CA3-CA1 place fields. The hippocampal spatial representation is used as a basis for goal-oriented spatial behavior. We focus on the neural pathway connecting the hippocampus to the nucleus accumbens. Place cells drive a population of locomotor action neurons in the nucleus accumbens. Reward-based learning is applied to map place cell activity into action cell activity. The ensemble action cell activity provides navigational maps to support spatial behavior. We present experimental results obtained with a mobile Khepera robot.

/pdf/spatial-cognition-and-neuro-mimetic-navigation-a-model-of-20ytaiz1s9.pdf

Spatial cognition and neuro-mimetic navigation: a model of hippocampal place cell activity.

We motivate the need for on-line learning in brain-computer interfaces (BCI) and illustrate its benefits with the simplest method, namely fixed learning rates. However, the use of this method is supported by the risk of hampering the user to acquire suitable control of the BCI if the embedded classifier changes too rapidly. We report the results with 3 beginner subjects in a series of consecutive recordings, where the classifiers are iteratively trained with the data of a given session and tested on the next session. Interestingly, performance improved over sessions significantly for 2 of the subjects. These results show that on-line learning improves systematically the performance of the subjects. Moreover, performance with online learning is statistically similar to that obtained training the classifier off-line with the same amount of data.

/pdf/on-the-need-for-on-line-learning-in-brain-computer-5cg4fj6sly.pdf

On the need for on-line learning in brain-computer interfaces

In this paper we propose a reinforcement connectionist learning architecture that allows an autonomous robot to acquire efficient navigation strategies in a few trials. Besides rapid learning, the architecture has three further appealing features. First, the robot improves its performance incrementally as it interacts with an initially unknown environment, and it ends up learning to avoid collisions even in those situations in which its sensors cannot detect the obstacles. This is a definite advantage over nonlearning reactive robots. Second, since it learns from basic reflexes, the robot is operational from the very beginning and the learning process is safe. Third, the robot exhibits high tolerance to noisy sensory data and good generalization abilities. All these features make this learning robot's architecture very well suited to real-world applications. We report experimental results obtained with a real mobile robot in an indoor environment that demonstrate the appropriateness of our approach to real autonomous robot control.

J.del.R. Millan

Papers

Rapid, safe, and incremental learning of navigation strategies