Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning.

/pdf/reinforcement-learning-an-introduction-rzxgej9p17.pdf

Reinforcement Learning: An Introduction

Objective: Parkinson’s disease (PD) is a progressive neurological disorder characterised by a large number of motor and non-motor features that can impact on function to a variable degree. This review describes the clinical characteristics of PD with emphasis on those features that differentiate the disease from other parkinsonian disorders. Methods: A MedLine search was performed to identify studies that assess the clinical characteristics of PD. Search terms included “Parkinson’s disease”, “diagnosis” and “signs and symptoms”. Results: Because there is no definitive test for the diagnosis of PD, the disease must be diagnosed based on clinical criteria. Rest tremor, bradykinesia, rigidity and loss of postural reflexes are generally considered the cardinal signs of PD. The presence and specific presentation of these features are used to differentiate PD from related parkinsonian disorders. Other clinical features include secondary motor symptoms (eg, hypomimia, dysarthria, dysphagia, sialorrhoea, micrographia, shuffling gait, festination, freezing, dystonia, glabellar reflexes), non-motor symptoms (eg, autonomic dysfunction, cognitive/neurobehavioral abnormalities, sleep disorders and sensory abnormalities such as anosmia, paresthesias and pain). Absence of rest tremor, early occurrence of gait difficulty, postural instability, dementia, hallucinations, and the presence of dysautonomia, ophthalmoparesis, ataxia and other atypical features, coupled with poor or no response to levodopa, suggest diagnoses other than PD. Conclusions: A thorough understanding of the broad spectrum of clinical manifestations of PD is essential to the proper diagnosis of the disease. Genetic mutations or variants, neuroimaging abnormalities and other tests are potential biomarkers that may improve diagnosis and allow the identification of persons at risk.

https://jnnp.bmj.com/content/jnnp/79/4/368.full.pdf

Parkinson’s disease: clinical features and diagnosis

http://www.columbia.edu/itc/gsas/g9600/2003/JavitchReadings/Schultz(Sulzer).pdf

Getting Formal with Dopamine and Reward

Introduction
Debate continues over the precise causal contribution made by mesolimbic dopamine systems to reward. There are three competing explanatory categories: ‘liking’, learning, and ‘wanting’. Does dopamine mostly mediate the hedonic impact of reward (‘liking’)? Does it instead mediate learned predictions of future reward, prediction error teaching signals and stamp in associative links (learning)? Or does dopamine motivate the pursuit of rewards by attributing incentive salience to reward-related stimuli (‘wanting’)? Each hypothesis is evaluated here, and it is suggested that the incentive salience or ‘wanting’ hypothesis of dopamine function may be consistent with more evidence than either learning or ‘liking’. In brief, recent evidence indicates that dopamine is neither necessary nor sufficient to mediate changes in hedonic ‘liking’ for sensory pleasures. Other recent evidence indicates that dopamine is not needed for new learning, and not sufficient to directly mediate learning by causing teaching or prediction signals. By contrast, growing evidence indicates that dopamine does contribute causally to incentive salience. Dopamine appears necessary for normal ‘wanting’, and dopamine activation can be sufficient to enhance cue-triggered incentive salience. Drugs of abuse that promote dopamine signals short circuit and sensitize dynamic mesolimbic mechanisms that evolved to attribute incentive salience to rewards. Such drugs interact with incentive salience integrations of Pavlovian associative information with physiological state signals. That interaction sets the stage to cause compulsive ‘wanting’ in addiction, but also provides opportunities for experiments to disentangle ‘wanting’, ‘liking’, and learning hypotheses. Results from studies that exploited those opportunities are described here.

/pdf/the-debate-over-dopamine-s-role-in-reward-the-case-for-2f5liljkhc.pdf

The debate over dopamine’s role in reward: the case for incentive salience

Many organisms, especially humans, are characterized by their capacity for intentional, goal-directed actions. However, similar behaviours often proceed automatically, as habitual responses to antecedent stimuli. How are goal-directed actions transformed into habitual responses? Recent work combining modern behavioural assays and neurobiological analysis of the basal ganglia has begun to yield insights into the neural basis of habit formation.

The role of the basal ganglia in habit formation

Positive reinforcement helps to control the acquisition of learned behaviours. Here we report a cellular mechanism in the brain that may underlie the behavioural effects of positive reinforcement. We used intracranial self-stimulation (ICSS) as a model of reinforcement learning, in which each rat learns to press a lever that applies reinforcing electrical stimulation to its own substantia nigra. The outputs from neurons of the substantia nigra terminate on neurons in the striatum in close proximity to inputs from the cerebral cortex on the same striatal neurons. We measured the effect of substantia nigra stimulation on these inputs from the cortex to striatal neurons and also on how quickly the rats learned to press the lever. We found that stimulation of the substantia nigra (with the optimal parameters for lever-pressing behaviour) induced potentiation of synapses between the cortex and the striatum, which required activation of dopamine receptors. The degree of potentiation within ten minutes of the ICSS trains was correlated with the time taken by the rats to learn ICSS behaviour. We propose that stimulation of the substantia nigra when the lever is pressed induces a similar potentiation of cortical inputs to the striatum, positively reinforcing the learning of the behaviour by the rats.

A cellular mechanism of reward-related learning

Firing modes of midbrain dopamine cells in the freely moving rat.

Behavioral conditioning of cue-reward pairing results in a shift of midbrain dopamine (DA) cell activity from responding to the reward to responding to the predictive cue. However, the precise time course and mechanism underlying this shift remain unclear. Here, we report a combined single-unit recording and temporal difference (TD) modeling approach to this question. The data from recordings in conscious rats showed that DA cells retain responses to predicted reward after responses to conditioned cues have developed, at least early in training. This contrasts with previous TD models that predict a gradual stepwise shift in latency with responses to rewards lost before responses develop to the conditioned cue. By exploring the TD parameter space, we demonstrate that the persistent reward responses of DA cells during conditioning are only accurately replicated by a TD model with long-lasting eligibility traces (nonzero values for the parameter λ) and low learning rate (α). These physiological constraints for TD parameters suggest that eligibility traces and low per-trial rates of plastic modification may be essential features of neural circuits for reward learning in the brain. Such properties enable rapid but stable initiation of learning when the number of stimulus-reward pairings is limited, conferring significant adaptive advantages in real-world environments.

https://www.jneurosci.org/content/jneuro/25/26/6235.full.pdf

Dopamine Cells Respond to Predicted Events during Classical Conditioning: Evidence for Eligibility Traces in the Reward-Learning Network

https://www.princeton.edu/~yael/LearningCourse/Targilim/WickensReynoldsHyland2003.pdf

Neural mechanisms of reward-related motor learning.

Motor thalamus (Mthal) is implicated in the control of movement because it is strategically located between motor areas of the cerebral cortex and motor-related subcortical structures, such as the cerebellum and basal ganglia (BG). The role of BG and cerebellum in motor control has been extensively studied but how Mthal processes inputs from these two networks is unclear. Specifically, there is considerable debate about the role of BG inputs on Mthal activity. This review summarises anatomical and physiological knowledge of the Mthal and its afferents and reviews current theories of Mthal function by discussing the impact of cortical, BG and cerebellar inputs on Mthal activity. One view is that Mthal activity in BG and cerebellar-receiving territories is primarily “driven” by glutamatergic inputs from the cortex or cerebellum, respectively, whereas BG inputs are modulatory and do not strongly determine Mthal activity. This theory is steeped in the assumption that the Mthal processes information in the same way as sensory thalamus, through interactions of modulatory inputs with a single driver input. Another view, from BG models, is that BG exert primary control on the BG-receiving Mthal so it effectively relays information from BG to cortex. We propose a new “super-integrator” theory where each Mthal territory processes multiple driver or driver-like inputs (cortex and BG, cortex and cerebellum), which are the result of considerable integrative processing. Thus, BG and cerebellar Mthal territories assimilate motivational and proprioceptive motor information previously integrated in cortico-BG and cortico-cerebellar networks, respectively, to develop sophisticated motor signals that are transmitted in parallel pathways to cortical areas for optimal generation of motor programmes. Finally, we briefly review the pathophysiological changes that occur in the BG in parkinsonism and generate testable hypotheses about how these may affect processing of inputs in the Mthal.

/pdf/motor-thalamus-integration-of-cortical-cerebellar-and-basal-1x3lcd5xdp.pdf

Brian I. Hyland

Papers

A cellular mechanism of reward-related learning

Firing modes of midbrain dopamine cells in the freely moving rat.

Dopamine Cells Respond to Predicted Events during Classical Conditioning: Evidence for Eligibility Traces in the Reward-Learning Network

Neural mechanisms of reward-related motor learning.

Motor thalamus integration of cortical, cerebellar and basal ganglia information: implications for normal and parkinsonian conditions.