Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning.

/pdf/reinforcement-learning-an-introduction-rzxgej9p17.pdf

Reinforcement Learning: An Introduction

The capacity to predict future events permits a creature to detect, model, and manipulate the causal structure of its interactions with its environment. Behavioral experiments suggest that learning is driven by changes in the expectations about future salient events such as rewards and punishments. Physiological work has recently complemented these studies by identifying dopaminergic neurons in the primate whose fluctuating output apparently signals changes or errors in the predictions of future salient and rewarding events. Taken together, these findings can be understood through quantitative theories of adaptive optimizing control.

/pdf/a-neural-substrate-of-prediction-and-reward-2tj1zqgoih.pdf

A Neural Substrate of Prediction and Reward

The neural basis of drug craving: An incentive-sensitization theory of addiction

Well-being : the foundations of hedonic psychology

A model of the neuropsychology of anxiety is proposed. The model is based in the first instance upon an analysis of the behavioural effects of the antianxiety drugs (benzodiazepines, barbiturates, and alcohol) in animals. From such psychopharmacologi-cal experiments the concept of a “behavioural inhibition system” (BIS) has been developed. This system responds to novel stimuli or to those associated with punishment or nonreward by inhibiting ongoing behaviour and increasing arousal and attention to the environment. It is activity in the BIS that constitutes anxiety and that is reduced by antianxiety drugs. The effects of the antianxiety drugs in the brain also suggest hypotheses concerning the neural substrate of anxiety. Although the benzodiazepines and barbiturates facilitate the effects of γ-aminobutyrate, this is insufficient to explain their highly specific behavioural effects. Because of similarities between the behavioural effects of certain lesions and those of the antianxiety drugs, it is proposed that these drugs reduce anxiety by impairing the functioning of a widespread neural system including the septo-hippocampal system (SHS), the Papez circuit, the prefrontal cortex, and ascending monoaminergic and cholinergic pathways which innervate these forebrain structures. Analysis of the functions of this system (based on anatomical, physiological, and behavioural data) suggests that it acts as a comparator: it compares predicted to actual sensory events and activates the outputs of the BIS when there is a mismatch or when the predicted event is aversive. Suggestions are made as to the functions of particular pathways within this overall brain system. The resulting theory is applied to the symptoms and treatment of anxiety in man, its relations to depression, and the personality of individuals who are susceptible to anxiety or depression.

https://www.cambridge.org/core/services/aop-cambridge-core/content/view/S0140525X00013066

Précis of The Neuropsychology of Anxiety: An Enquiry into the Functions of the Septo-Hippocampal System..

The hypothesis that dopamine is important for reward has been proposed in a number of forms, each of which has been challenged. Normally, rewarding stimuli such as food, water, lateral hypothalamic brain stimulation and several drugs of abuse become ineffective as rewards in animals given performance-sparing doses of dopamine antagonists. Dopamine release in the nucleus accumbens has been linked to the efficacy of these unconditioned rewards, but dopamine release in a broader range of structures is implicated in the 'stamping-in' of memory that attaches motivational importance to otherwise neutral environmental stimuli.

Dopamine, learning and motivation

The theory is advanced that the common denominator of a wide range of addictive substances is their ability to cause psychomotor activation. This view is related to the theory that all positive reinforcers activate a common biological mechanism associated with approach behaviors and that this mechanism has as one of its components dopaminergic fibers that project up the medial forebrain bundle from the midbrain to limbic and cortical regions. Evidence is reviewed that links both the reinforcing and locomotor-stimulating effects of both the psychomotor stimulants and the opiates to this brain mechanism. It is suggested that nicotine, caffeine, barbiturates, alcohol, benzodiazepines, cannabis, and phencyclidine----each ofwhich also has psychomotor stimulant actions--may activate the docaminergic fibers or their output circuitry. The role of physical dependence in addiction is suggested to vary from drug to drug and to be of secondary importance in the understanding of compulsive drug self-administration. Attempts at a general theory of addiction are attempts to isr late--from a variety of irrelevant actionsmthose drug actions that are responsible for habitual, compulsive, nonmedical drug self-administration. The common assumption of addiction theorists is that general principles of addiction can be learned from the study of one drug and that these principles will have heuristic value for the study of other drugs. Thus far, attempts at a general theory of addiction have failed to isolate common actions that can account for addiction across the range of major drug classes. A major stumbling block has been the psychomotor stimulants--amph etamine and cocainemwhich do not readily fit models traditionally based on depressant drug classes. The present article offers a new attempt at a general theory of addiction. It differs from earlier theories (e.g., Collier, 1968; Himmelsbach, 1943; Jaffe & Sharpless, 1968; Jellinek, 1960; Kalant, 1977; Lindsmith, 1947; Solomon & Corbit, 1974) in that it is based on the common denominator of the psychomotor stimulants---amphetamine, cocaine, and related drugs---rather than on the common denominator of the socalled depressant drugs~opiates, barbiturates, alcohol, and others. We take up two topics before presenting the new theory. First, we briefly discuss the heuristic value of a biological approach and suggest that the biologist's distinction between homology and analogy offers a useful insight. Next we discuss the shortcomings of earlier theories--variants of dependence theory. Then we outline the new theory and review the relevant evidence for its three major assertions: (a) that all addictive drugs have psychomotor stimulant actions, (b) that the stimulant actions of these different drugs have a shared biological mechanism, and (c) that the biological mechanism of these stimulant

A psychomotor stimulant theory of addiction

While the evidence is strong that dopamine plays some fundamental and special role in the rewarding effects of brain stimulation, psychomotor stimulants, opiates, and food, the exact nature of that role is not clear. One thing is clear: Dopamine is not the only reward transmitter, and dopaminergic neurons are not the final common path for all rewards. Dopamine antagonists and lesions of the dopamine systems appear to spare the rewarding effects of nucleus accumbens and frontal cortex brain stimulation (Simon et al 1979) and certainly spare the rewarding effects of apomorphine (Roberts & Vickers 1988). It is clear that reward circuitry is multisynaptic, and since dopamine cells do not send axons to each other or receive axons from each other, dopamine can at best serve as but a single link in this circuitry. If dopamine is not a final common path for all rewards, could it be an intermediate common path for most rewards? Some workers have argued against such a view, but at present they must do so on incomplete evidence. For example, Phillips (1984) has argued that there must be multiple reward systems, functionally independent and organized in parallel with one another. His primary evidence, however, is the fact that brain stimulation is rewarding at different levels of the nervous system. As we have seen in the case of midline mesencephalic stimulation, the location of the electrode tip in relation to the dopamine cells and fibers tells us little about the role of dopamine in brain stimulation reward. It seems clear that the ventral tegmental dopamine system plays a critical role in midline mesencephalic reward, despite the distance from the electrode tip to the dopamine cells where morphine causes its dopamine-dependent facilitory effects or to the dopamine terminals where low-dose neuroleptics presumably cause theirs. Until pharmacological challenge has been extended to the cases discussed by Phillips, we can only speculate as to the role of dopamine in each of those cases. In the cases where pharmacological challenge has been examined, only nucleus accumbens and frontal cortex have been found to have dopamine-independent reward sites. It is not consistent with the dopamine hypothesis that dopamine-independent reward sites should exist in these areas, since any reward signals carried to nucleus accumbens or frontal cortex by dopamine fibers would-unless we are to believe that reward "happens" at these sites-have to be carried to the next stage of the circuit by nondopaminergic fibers (there are no dopaminergic cell bodies in any of the dopamine terminal areas).(ABSTRACT TRUNCATED AT 400 WORDS)

Brain dopamine and reward.

Neurobiology of addiction

To the degree that drugs and food activate common reward circuitry in the brain, drugs offer powerful tools for understanding the neural circuitry that mediates food-motivated habits and how this circuitry may be hijacked to cause appetitive behaviors to go awry.

Roy A. Wise

Papers

Dopamine, learning and motivation

A psychomotor stimulant theory of addiction

Brain dopamine and reward.

Neurobiology of addiction

How can drug addiction help us understand obesity