Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning.

/pdf/reinforcement-learning-an-introduction-rzxgej9p17.pdf

Reinforcement Learning: An Introduction

Despite the concern that has been expressed about potential method biases, and the pervasiveness of research settings with the potential to produce them, there is disagreement about whether they really are a problem for researchers in the behavioral sciences. Therefore, the purpose of this review is to explore the current state of knowledge about method biases. First, we explore the meaning of the terms “method” and “method bias” and then we examine whether method biases influence all measures equally. Next, we review the evidence of the effects that method biases have on individual measures and on the covariation between different constructs. Following this, we evaluate the procedural and statistical remedies that have been used to control method biases and provide recommendations for minimizing method bias.

Sources of Method Bias in Social Science Research and Recommendations on How to Control It

http://www.brainresearch.us/cognitive.pdf

The cognitive control of emotion.

Historically, the locus coeruleus-norepinephrine (LC-NE) system has been implicated in arousal, but recent findings suggest that this system plays a more complex and specific role in the control of behavior than investigators previously thought. We review neurophysiological and modeling studies in monkey that support a new theory of LC-NE function. LC neurons exhibit two modes of activity, phasic and tonic. Phasic LC activation is driven by the outcome of task-related decision processes and is proposed to facilitate ensuing behaviors and to help optimize task performance (exploitation). When utility in the task wanes, LC neurons exhibit a tonic activity mode, associated with disengagement from the current task and a search for alternative behaviors (exploration). Monkey LC receives prominent, direct inputs from the anterior cingulate (ACC) and orbitofrontal cortices (OFC), both of which are thought to monitor task-related utility. We propose that these frontal areas produce the above patterns of LC activity to optimize utility on both short and long timescales.

/pdf/an-integrative-theory-of-locus-coeruleus-norepinephrine-2ljsj0k79l.pdf

An Integrative Theory of Locus Coeruleus-Norepinephrine Function: Adaptive Gain and Optimal Performance.

Drug addiction is increasingly viewed as the endpoint of a series of transitions from initial drug use--when a drug is voluntarily taken because it has reinforcing, often hedonic, effects--through loss of control over this behavior, such that it becomes habitual and ultimately compulsive. Here we discuss evidence that these transitions depend on interactions between pavlovian and instrumental learning processes. We hypothesize that the change from voluntary drug use to more habitual and compulsive drug use represents a transition at the neural level from prefrontal cortical to striatal control over drug seeking and drug taking behavior as well as a progression from ventral to more dorsal domains of the striatum, involving its dopaminergic innervation. These neural transitions may themselves depend on the neuroplasticity in both cortical and striatal structures that is induced by chronic self-administration of drugs.

/pdf/neural-systems-of-reinforcement-for-drug-addiction-from-3s2f74bu84.pdf

Neural systems of reinforcement for drug addiction: from actions to habits to compulsion

Fast T2-mapping with snapshot flash imaging.

Functional magnetic resonance imaging in intact plants : Quantitative observation of flow in plant vessels

Perfluorochemicals (PFCs) are not only of interest as artificial oxygen carriers but also as 19F-NMR (19F nuclear magnetic resonance) contrast agents for the determination of partial oxygen pressuresin vivo. Since the longitudinal relaxation rate R1 increases linearly with the partial oxygen pressure (pO2), pO2 values can be determined in regions of PFC accumulation. After the i.v. injection of a PFC emulsion, pO2 measurements are possible in large vessels until most of the PFC particles are removed from the circulation by macrophages. Then intracellular pO2 measurements can be performed in regions with high macrophage concentration, i.e. in the liver and spleen, and in pathological tissues with high macrophage activity, e.g. in tumors and in regions of inflammation. In many studies with PFC emulsions, negative apparent pO2 values were reported under hypoxic conditions (F1O2 < 0.21) [1-7]. Purpose of this work was to measure extracellular pO2 values in a depot of neat PFC in the abdominal cavity and in the skeletal muscle under hyperoxic, normoxic and hypoxic conditions, and further to determine whether neat PFC also yields negative apparent pO2 values under hypoxic conditions.

Measurement of Oxygen Tensions in the Abdominal Cavity and in the Skeletal Muscle Using 19F-MRI of Neat PFC Droplets

Diffusion tensor imaging (DTI) can provide vital insights into brain connectivity, and may become an important tool for the diagnosis and treatment of neurological disease However, DTI's intrinsic low signal-to-noise ratio (SNR) and vulnerability to ghosting artifacts can result in poor image quality with low spatial resolution, which limits its clinical applications In this study, a new double-shot EPI sequence (half-FOV EPI) with high spatial resolution was developed This method enables DT measurements to be obtained with high isotropic spatial resolution and whole-brain coverage To avoid ghosting artifacts, the data are combined in image space rather than in k-space

3D DT-MRI using a reduced-FOV approach and saturation pulses.

A number of techniques that recently have been used for fast NMR-imaging are based on a hybrid sequence of echo planar imaging (EPI) and FLASH imaging: after each NMR excitation several k-space lines are measured. The complete k-space is covered by performance of several excitations. It has been observed that there is usually an optimal hybrid sequence that maximizes the signal-to-noise ratio. In this work, a method is presented that allows a determination of the optimal sequence as a function of the relaxation times T1 and T2*.

Ralf Deichmann

Papers

Fast T2-mapping with snapshot flash imaging.

Functional magnetic resonance imaging in intact plants : Quantitative observation of flow in plant vessels

Measurement of Oxygen Tensions in the Abdominal Cavity and in the Skeletal Muscle Using 19F-MRI of Neat PFC Droplets

3D DT-MRI using a reduced-FOV approach and saturation pulses.

Calculation of signal intensities in hybrid sequences for fast NMR imaging