Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning.

/pdf/reinforcement-learning-an-introduction-rzxgej9p17.pdf

Reinforcement Learning: An Introduction

Publisher Summary This chapter provides an account of different neural network architectures for pattern recognition. A neural network consists of several simple processing elements called neurons. Each neuron is connected to some other neurons and possibly to the input nodes. Neural networks provide a simple computing paradigm to perform complex recognition tasks in real time. The chapter categorizes neural networks into three types: single-layer networks, multilayer feedforward networks, and feedback networks. It discusses the gradient descent and the relaxation method as the two underlying mathematical themes for deriving learning algorithms. A lot of research activity is centered on learning algorithms because of their fundamental importance in neural networks. The chapter discusses two important directions of research to improve learning algorithms: the dynamic node generation, which is used by the cascade correlation algorithm; and designing learning algorithms where the choice of parameters is not an issue. It closes with the discussion of performance and implementation issues.

Neural Networks for Pattern Recognition

Long-term potentiation of synaptic transmission in the hippocampus is the primary experimental model for investigating the synaptic basis of learning and memory in vertebrates. The best understood form of long-term potentiation is induced by the activation of the N-methyl-D-aspartate receptor complex. This subtype of glutamate receptor endows long-term potentiation with Hebbian characteristics, and allows electrical events at the postsynaptic membrane to be transduced into chemical signals which, in turn, are thought to activate both pre- and postsynaptic mechanisms to generate a persistent increase in synaptic strength.

A synaptic model of memory: long-term potentiation in the hippocampus

A baseline or control state is fundamental to the understanding of most complex systems. Defining a baseline state in the human brain, arguably our most complex system, poses a particular challenge. Many suspect that left unconstrained, its activity will vary unpredictably. Despite this prediction we identify a baseline state of the normal adult human brain in terms of the brain oxygen extraction fraction or OEF. The OEF is defined as the ratio of oxygen used by the brain to oxygen delivered by flowing blood and is remarkably uniform in the awake but resting state (e.g., lying quietly with eyes closed). Local deviations in the OEF represent the physiological basis of signals of changes in neuronal activity obtained with functional MRI during a wide variety of human behaviors. We used quantitative metabolic and circulatory measurements from positron-emission tomography to obtain the OEF regionally throughout the brain. Areas of activation were conspicuous by their absence. All significant deviations from the mean hemisphere OEF were increases, signifying deactivations, and resided almost exclusively in the visual system. Defining the baseline state of an area in this manner attaches meaning to a group of areas that consistently exhibit decreases from this baseline, during a wide variety of goal-directed behaviors monitored with positron-emission tomography and functional MRI. These decreases suggest the existence of an organized, baseline default mode of brain function that is suspended during specific goal-directed behaviors.

/pdf/a-default-mode-of-brain-function-zwenkbmjw7.pdf

A default mode of brain function.

https://onlinelibrary.wiley.com/doi/pdf/10.1207/s15516709cog1402_1

Finding Structure in Time

Paramagnetic deoxyhemoglobin in venous blood is a naturally occurring contrast agent for magnetic resonance imaging (MRI). By accentuating the effects of this agent through the use of gradient-echo techniques in high fields, we demonstrate in vivo images of brain microvasculature with image contrast reflecting the blood oxygen level. This blood oxygenation level-dependent (BOLD) contrast follows blood oxygen changes induced by anesthetics, by insulin-induced hypoglycemia, and by inhaled gas mixtures that alter metabolic demand or blood flow. The results suggest that BOLD contrast can be used to provide in vivo real-time maps of blood oxygenation in the brain under normal physiological conditions. BOLD contrast adds an additional feature to magnetic resonance imaging and complements other techniques that are attempting to provide positron emission tomography-like measurements related to regional neural activity.

https://www.pnas.org/content/pnas/87/24/9868.full.pdf

Brain magnetic resonance imaging with contrast dependent on blood oxygenation

Highly-interconnected networks of nonlinear analog neurons are shown to be extremely effective in computing. The networks can rapidly provide a collectively-computed solution (a digital output) to a problem on the basis of analog input information. The problems to be solved must be formulated in terms of desired optima, often subject to constraints. The general principles involved in constructing networks to solve specific problems are discussed. Results of computer simulations of a network designed to solve a difficult but well-defined optimization problem-the Traveling-Salesman Problem-are presented and used to illustrate the computational power of the networks. Good solutions to this problem are collectively computed within an elapsed time of only a few neural time constants. The effectiveness of the computation involves both the nonlinear analog response of the neurons and the large connectivity among them. Dedicated networks of biological or microelectronic neurons could provide the computational capabilities described for a wide class of problems having combinatorial complexity. The power and speed naturally displayed by such collective networks may contribute to the effectiveness of biological information processing.

Neural computation of decisions in optimization problems

We report that visual stimulation produces an easily detectable (5-20%) transient increase in the intensity of water proton magnetic resonance signals in human primary visual cortex in gradient echo images at 4-T magnetic-field strength. The observed changes predominantly occur in areas containing gray matter and can be used to produce high-spatial-resolution functional brain maps in humans. Reducing the image-acquisition echo time from 40 msec to 8 msec reduces the amplitude of the fractional signal change, suggesting that it is produced by a change in apparent transverse relaxation time T*2. The amplitude, sign, and echo-time dependence of these intrinsic signal changes are consistent with the idea that neural activation increases regional cerebral blood flow and concomitantly increases venous-blood oxygenation.

https://www.pnas.org/content/pnas/89/13/5951.full.pdf

Intrinsic signal changes accompanying sensory stimulation: functional brain mapping with magnetic resonance imaging

We describe how several optimization problems can be rapidly solved by highly interconnected networks of simple analog processors. Analog-to-digital (A/D) conversion was considered as a simple optimization problem, and an A/D converter of novel architecture was designed. A/D conversion is a simple example of a more general class of signal-decision problems which we show could also be solved by appropriately constructed networks. Circuits to solve these problems were designed using general principles which result from an understanding of the basic collective computational properties of a specific class of analog-processor networks. We also show that a network which solves linear programming problems can be understood from the same concepts.

Simple 'neural' optimization networks: An A/D converter, signal decision circuit, and a linear programming circuit

A new conceptual framework and a minimization principle together provide an understanding of computation in model neural circuits. The circuits consist of nonlinear graded-response model neurons organized into networks with effectively symmetric synaptic connections. The neurons represent an approximation to biological neurons in which a simplified set of important computational properties is retained. Complex circuits solving problems similar to those essential in biology can be analyzed and understood without the need to follow the circuit dynamics in detail. Implementation of the model with electronic devices will provide a class of electronic circuits of novel form and function.

https://science.sciencemag.org/content/sci/233/4764/625.full.pdf

David W. Tank

Papers

Brain magnetic resonance imaging with contrast dependent on blood oxygenation

Neural computation of decisions in optimization problems

Intrinsic signal changes accompanying sensory stimulation: functional brain mapping with magnetic resonance imaging

Simple 'neural' optimization networks: An A/D converter, signal decision circuit, and a linear programming circuit

Computing with neural circuits: a model