Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning.

/pdf/reinforcement-learning-an-introduction-rzxgej9p17.pdf

Reinforcement Learning: An Introduction

Co-Planar Stereotaxic Atlas of the Human Brain—3-Dimensional Proportional System: An Approach to Cerebral Imaging, J. Talairach, P. Tournoux. Georg Thieme Verlag, New York (1988), 122 pp., 130 figs. DM 268

I have developed "tennis elbow" from lugging this book around the past four weeks, but it is worth the pain, the effort, and the aspirin. It is also worth the (relatively speaking) bargain price. Including appendixes, this book contains 894 pages of text. The entire panorama of the neural sciences is surveyed and examined, and it is comprehensive in its scope, from genomes to social behaviors. The editors explicitly state that the book is designed as "an introductory text for students of biology, behavior, and medicine," but it is hard to imagine any audience, interested in any fragment of neuroscience at any level of sophistication, that would not enjoy this book. The editors have done a masterful job of weaving together the biologic, the behavioral, and the clinical sciences into a single tapestry in which everyone from the molecular biologist to the practicing psychiatrist can find and appreciate his or

Principles of Neural Science

The hypothesis that dopamine is important for reward has been proposed in a number of forms, each of which has been challenged. Normally, rewarding stimuli such as food, water, lateral hypothalamic brain stimulation and several drugs of abuse become ineffective as rewards in animals given performance-sparing doses of dopamine antagonists. Dopamine release in the nucleus accumbens has been linked to the efficacy of these unconditioned rewards, but dopamine release in a broader range of structures is implicated in the 'stamping-in' of memory that attaches motivational importance to otherwise neutral environmental stimuli.

Dopamine, learning and motivation

http://www.columbia.edu/itc/gsas/g9600/2003/JavitchReadings/Schultz(Sulzer).pdf

Getting Formal with Dopamine and Reward

Defense Advanced Research Projects Agency (AFOSR 90-0083); Office of Naval Research (N00014-92-J-l309); Consejo Nacional de Ciencia y Tecnologia (63l462)

Navite: A Neural Network System For Sensory-Based Robot Navigation

In order to approach human hand performance levels, artificial anthropomorphic hands/fingers have increasingly incorporated human biomechanical features. However, the performance of finger reaching movements to visual targets involving the complex kinematics of multi-jointed, anthropomorphic actuators is a difficult problem. This is because the relationship between sensory and motor coordinates is highly nonlinear, and also often includes mechanical coupling of the two last joints. Recently, we developed a cortical model that learns the inverse kinematics of a simulated anthropomorphic finger. Here, we expand this previous work by assessing if this cortical model is able to learn the inverse kinematics for an actual anthropomorphic humanoid finger having its two last joints coupled and controlled by pneumatic muscles. The findings revealed that single 3D reaching movements, as well as more complex patterns of motion of the humanoid finger, were accurately and robustly performed by this cortical model while producing kinematics comparable to those of humans. This work contributes to the development of a bioinspired controller providing adaptive, robust and flexible control of dexterous robotic and prosthetic hands.

/pdf/cortex-inspired-model-for-inverse-kinematics-computation-for-zellq8bryg.pdf

Cortex inspired model for inverse kinematics computation for a humanoid robotic finger

This study investigates if the electrocortical amplitude modulations relative to the mean gait cycle are different across walking conditions (i.e., level-ground (LW), ramp ascent (RA), and stair ascent (SA)). Non-invasive electroencephalography (EEG) signals were recorded and a systematic EEG processing method was implemented to reduce artifacts. Source localization using independent component analysis and k-means clustering revealed the involvement of four clusters in the brain (Left and Right Occipital Lobe, Posterior Parietal Cortex (PPC), and Sensorimotor Area) during the walking tasks. We found that electrocortical amplitude modulations varied across different walking conditions. Specifically, our results showed that the modulations in the PPC shifted to higher frequency bands when the subjects walked in RA and SA conditions. Moreover, we found low γ modulations in the sensorimotor area in LW walking and the modulations in this cluster shifted to lower frequency bands in RA and SA walking. These results are a promising step toward the development of a non-invasive Neural-machine Interface (NMI) for locomotion mode recognition.

Electrocortical amplitude modulations of human level-ground, slope, and stair walking

In this study, we demonstrate the feasibility of predicting hand forces from brain activity recorded with scalp electroencephalography (EEG). Ten able-bodied subjects participated in two tasks: an isometric force production task and a grasp-and-lift task using unconstrained and constrained grasps. We found that EEG electrodes spanning central areas of the scalp were highly correlated to force rate trajectories. Moreover, EEG grand averages in central sites resembled force rate trajectories as opposed to force trajectories. The grasp-and-lift task resulted in higher decoding accuracies than the isometric force production task: across nine subjects, median accuracies for the isometric force production task were r=0.35 whereas median accuracies for unconstrained grasping were r=0.51 and for constrained grasping were r=0.50. Such results could lead to an understanding of the neural representation behind the control of hand forces and could be implemented in the neural control of closed-loop hand-based neuroprostheses.

Predicting hand forces from scalp electroencephalography during isometric force production and object grasping

In Part 1 of this report, we outlined a framework for creating an intelligent agent based upon modeling the large-scale functionality of the human brain. Building on those results, we begin Part 2 by specifying the behavioral requirements of a large-scale neurocognitive architecture. The core of our long-term approach remains focused on creating a network of neuromorphic regions that provide the mechanisms needed to meet these requirements. However, for the short term of the next few years, it is likely that optimal results will be obtained by using a hybrid design that also includes symbolic methods from AI/cognitive science and control processes from the field of artificial life. We accordingly propose a three-tiered architecture that integrates these different methods, and describe an ongoing computational study of a prototype “miniRoboscout” based on this architecture. We also examine the implications of some non-standard computational methods for developing a neurocognitive agent. This examination included computational experiments assessing the effectiveness of genetic programming as a design tool for recurrent neural networks for sequence processing, and experiments measuring the speed-up obtained for adaptive neural networks when they are executed on a graphical processing unit (GPU) rather than a conventional CPU. We conclude that the implementation of a large-scale neurocognitive architecture is feasible, and outline a roadmap for achieving this goal.

Jose L. Contreras-Vidal

Papers

Navite: A Neural Network System For Sensory-Based Robot Navigation

Cortex inspired model for inverse kinematics computation for a humanoid robotic finger

Electrocortical amplitude modulations of human level-ground, slope, and stair walking

Predicting hand forces from scalp electroencephalography during isometric force production and object grasping

Development of a Large-Scale Integrated Neurocognitive Architecture Part 2: Design and Architecture