Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning.

/pdf/reinforcement-learning-an-introduction-rzxgej9p17.pdf

Reinforcement Learning: An Introduction

Despite the concern that has been expressed about potential method biases, and the pervasiveness of research settings with the potential to produce them, there is disagreement about whether they really are a problem for researchers in the behavioral sciences. Therefore, the purpose of this review is to explore the current state of knowledge about method biases. First, we explore the meaning of the terms “method” and “method bias” and then we examine whether method biases influence all measures equally. Next, we review the evidence of the effects that method biases have on individual measures and on the covariation between different constructs. Following this, we evaluate the procedural and statistical remedies that have been used to control method biases and provide recommendations for minimizing method bias.

Sources of Method Bias in Social Science Research and Recommendations on How to Control It

http://www.brainresearch.us/cognitive.pdf

The cognitive control of emotion.

Historically, the locus coeruleus-norepinephrine (LC-NE) system has been implicated in arousal, but recent findings suggest that this system plays a more complex and specific role in the control of behavior than investigators previously thought. We review neurophysiological and modeling studies in monkey that support a new theory of LC-NE function. LC neurons exhibit two modes of activity, phasic and tonic. Phasic LC activation is driven by the outcome of task-related decision processes and is proposed to facilitate ensuing behaviors and to help optimize task performance (exploitation). When utility in the task wanes, LC neurons exhibit a tonic activity mode, associated with disengagement from the current task and a search for alternative behaviors (exploration). Monkey LC receives prominent, direct inputs from the anterior cingulate (ACC) and orbitofrontal cortices (OFC), both of which are thought to monitor task-related utility. We propose that these frontal areas produce the above patterns of LC activity to optimize utility on both short and long timescales.

/pdf/an-integrative-theory-of-locus-coeruleus-norepinephrine-2ljsj0k79l.pdf

An Integrative Theory of Locus Coeruleus-Norepinephrine Function: Adaptive Gain and Optimal Performance.

Drug addiction is increasingly viewed as the endpoint of a series of transitions from initial drug use--when a drug is voluntarily taken because it has reinforcing, often hedonic, effects--through loss of control over this behavior, such that it becomes habitual and ultimately compulsive. Here we discuss evidence that these transitions depend on interactions between pavlovian and instrumental learning processes. We hypothesize that the change from voluntary drug use to more habitual and compulsive drug use represents a transition at the neural level from prefrontal cortical to striatal control over drug seeking and drug taking behavior as well as a progression from ventral to more dorsal domains of the striatum, involving its dopaminergic innervation. These neural transitions may themselves depend on the neuroplasticity in both cortical and striatal structures that is induced by chronic self-administration of drugs.

/pdf/neural-systems-of-reinforcement-for-drug-addiction-from-3s2f74bu84.pdf

Neural systems of reinforcement for drug addiction: from actions to habits to compulsion

Instrumental conditioning studies how animals and humans choose actions appropriate to the affective structure of an environment. According to recent reinforcement learning models, two distinct components are involved: a "critic," which learns to predict future reward, and an "actor," which maintains information about the rewarding outcomes of actions to enable better ones to be chosen more frequently. We scanned human participants with functional magnetic resonance imaging while they engaged in instrumental conditioning. Our results suggest partly dissociable contributions of the ventral and dorsal striatum, with the former corresponding to the critic and the latter corresponding to the actor.

/pdf/dissociable-roles-of-ventral-and-dorsal-striatum-in-12nht10ikt.pdf

Dissociable roles of ventral and dorsal striatum in instrumental conditioning

https://pure.mpg.de/pubman/item/item_2614420_3/component/file_2623299/RD_Neural_2002.pdf

Neural responses during anticipation of a primary taste reward.

The precise role of orbitofrontal cortex (OFC) in affective processing is still debated. One view suggests OFC represents stimulus reward value and supports learning and relearning of stimulus-reward associations. An alternate view implicates OFC in behavioral control after rewarding or punishing feedback. To discriminate between these possibilities, we used event-related functional magnetic resonance imaging in subjects performing a reversal task in which, on each trial, selection of the correct stimulus led to a 70% probability of receiving a monetary reward and a 30% probability of obtaining a monetary punishment. The incorrect stimulus had the reverse contingency. In one condition (choice), subjects had to choose which stimulus to select and switch their response to the other stimulus once contingencies had changed. In another condition (imperative), subjects had simply to track the currently rewarded stimulus. In some regions of OFC and medial prefrontal cortex, activity was related to valence of outcome, whereas in adjacent areas activity was associated with behavioral choice, signaling maintenance of the current response strategy on a subsequent trial. Caudolateral OFC-anterior insula was activated by punishing feedback preceding a switch in stimulus in both the choice and imperative conditions, indicating a possible role for this region in signaling a change in reward contingencies. These results suggest functional heterogeneity within the OFC, with a role for this region in representing stimulus-reward values, signaling changes in reinforcement contingencies and in behavioral control.

https://www.jneurosci.org/content/jneuro/23/21/7931.full.pdf

Dissociating valence of outcome from behavioral control in human orbital and ventral prefrontal cortices.

https://pure.mpg.de/pubman/item/item_720877_4/component/file_720876/ruff_concurrent.pdf

Ralf Deichmann

Papers

Dissociable roles of ventral and dorsal striatum in instrumental conditioning

Neural responses during anticipation of a primary taste reward.

Dissociating valence of outcome from behavioral control in human orbital and ventral prefrontal cortices.

Concurrent TMS-fMRI and psychophysics reveal frontal influences on human retinotopic visual cortex

Optimal EPI parameters for reduction of susceptibility-induced BOLD sensitivity losses: A whole-brain analysis at 3 T and 1.5 T