Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning.

/pdf/reinforcement-learning-an-introduction-rzxgej9p17.pdf

Reinforcement Learning: An Introduction

Noise — random disturbances of signals — poses a fundamental problem for information processing and affects all aspects of nervous-system function. However, the nature, amount and impact of noise in the nervous system have only recently been addressed in a quantitative manner. Experimental and computational methods have shown that multiple noise sources contribute to cellular and behavioural trial-to-trial variability. We review the sources of noise in the nervous system, from the molecular to the behavioural level, and show how noise contributes to trial-to-trial variability. We highlight how noise affects neuronal networks and the principles the nervous system applies to counter detrimental effects of noise, and briefly discuss noise's potential benefits.

/pdf/noise-in-the-nervous-system-22k10mncs2.pdf

Noise in the nervous system.

Uncertainty is critical in the measure of information and in assessing the accuracy of predictions. It is determined by probability P, being maximal at P = 0.5 and decreasing at higher and lower probabilities. Using distinct stimuli to indicate the probability of reward, we found that the phasic activation of dopamine neurons varied monotonically across the full range of probabilities, supporting past claims that this response codes the discrepancy between predicted and actual reward. In contrast, a previously unobserved response covaried with uncertainty and consisted of a gradual increase in activity until the potential time of reward. The coding of uncertainty suggests a possible role for dopamine signals in attention-based learning and risk-taking behavior.

/pdf/discrete-coding-of-reward-probability-and-uncertainty-by-2vdhyeux9s.pdf

Discrete coding of reward probability and uncertainty by dopamine neurons.

Dopamine in Motivational Control: Rewarding, Aversive, and Alerting

The journal of neuroscience : the official journal of the Society for Neuroscience.

We examine the dynamics of a neural code in the context of stimuli whose statistical properties are themselves evolving dynamically. Adaptation to these statistics occurs over a wide range of timescales-from tens of milliseconds to minutes. Rapid components of adaptation serve to optimize the information that action potentials carry about rapid stimulus variations within the local statistical ensemble, while changes in the rate and statistics of action-potential firing encode information about the ensemble itself, thus resolving potential ambiguities. The speed with which information is optimized and ambiguities are resolved approaches the physical limit imposed by statistical sampling and noise.

/pdf/efficiency-and-ambiguity-in-an-adaptive-neural-code-5949gghdsv.pdf

Efficiency and ambiguity in an adaptive neural code.

Neural systems adapt to changes in stimulus statistics. The authors find that neocortical pyramidal neurons adapt with a time scale that depends on the time scale of changes in stimulus statistics, and that for individual neurons the firing is a fractional derivative of slowly varying stimulus parameters. Neural systems adapt to changes in stimulus statistics. However, it is not known how stimuli with complex temporal dynamics drive the dynamics of adaptation and the resulting firing rate. For single neurons, it has often been assumed that adaptation has a single time scale. We found that single rat neocortical pyramidal neurons adapt with a time scale that depends on the time scale of changes in stimulus statistics. This multiple time scale adaptation is consistent with fractional order differentiation, such that the neuron's firing rate is a fractional derivative of slowly varying stimulus parameters. Biophysically, even though neuronal fractional differentiation effectively yields adaptation with many time scales, we found that its implementation required only a few properly balanced known adaptive mechanisms. Fractional differentiation provides single neurons with a fundamental and general computation that can contribute to efficient information processing, stimulus anticipation and frequency-independent phase shifts of oscillatory neuronal firing.

http://adaptingneurons.com/bnl/Science_files/lundstrom_natneurosci08.pdf

Fractional differentiation by neocortical pyramidal neurons.

Neuronal responses to ongoing stimulation in many systems change over time, or “adapt.” Despite the ubiquity of adaptation, its effects on the stimulus information carried by neurons are often unknown. Here we examine how adaptation affects sensory coding in barrel cortex. We used spike-triggered covariance analysis of single-neuron responses to continuous, rapidly varying vibrissa motion stimuli, recorded in anesthetized rats. Changes in stimulus statistics induced spike rate adaptation over hundreds of milliseconds. Vibrissa motion encoding changed with adaptation as follows. In every neuron that showed rate adaptation, the input–output tuning function scaled with the changes in stimulus distribution, allowing the neurons to maintain the quantity of information conveyed about stimulus features. A single neuron that did not show rate adaptation also lacked input–output rescaling and did not maintain information across changes in stimulus statistics. Therefore, in barrel cortex, rate adaptation occurs on a slow timescale relative to the features driving spikes and is associated with gain rescaling matched to the stimulus distribution. Our results suggest that adaptation enhances tactile representations in primary somatosensory cortex, where they could directly influence perceptual decisions.

https://www.princeton.edu/~wbialek/rome/refs/maravall+al_07.pdf

Shifts in coding properties and maintenance of information transmission during adaptation in barrel cortex.

https://faculty.washington.edu/jriffell/wordpress/wp-content/uploads/2015/08/van-Breugal-et-al-2015-Current-Biology.pdf

Adrienne L. Fairhall

Papers

Efficiency and ambiguity in an adaptive neural code.

Fractional differentiation by neocortical pyramidal neurons.

Shifts in coding properties and maintenance of information transmission during adaptation in barrel cortex.

Mosquitoes use vision to associate odor plumes with thermal targets

Timescales of inference in visual adaptation.