Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning.

/pdf/reinforcement-learning-an-introduction-rzxgej9p17.pdf

Reinforcement Learning: An Introduction

抗原变异可使得多种致病微生物易于逃避宿主免疫应答。表达在感染红细胞表面的恶性疟原虫红细胞表面蛋白1（PfPMP1）与感染红细胞、内皮细胞、树突状细胞以及胎盘的单个或多个受体作用，在黏附及免疫逃避中起关键的作用。每个单倍体基因组var基因家族编码约60种成员，通过启动转录不同的var基因变异体为抗原变异提供了分子基础。

疟原虫var基因转换速率变化导致抗原变异[英]／Paul H, Robert P, Christodoulou Z, et al//Proc Natl Acad Sci U S A

Recent developments in the quantitative analysis of complex networks, based largely on graph theory, have been rapidly translated to studies of brain network organization. The brain's structural and functional systems have features of complex networks--such as small-world topology, highly connected hubs and modularity--both at the whole-brain scale of human neuroimaging and at a cellular scale in non-human animals. In this article, we review studies investigating complex brain networks in diverse experimental modalities (including structural and functional MRI, diffusion tensor imaging, magnetoencephalography and electroencephalography in humans) and provide an accessible introduction to the basic principles of graph theory. We also highlight some of the technical challenges and key questions to be addressed by future developments in this rapidly moving field.

Complex brain networks: graph theoretical analysis of structural and functional systems

http://brainimaging.waisman.wisc.edu/~chung/neuro.processing/literature/rubinov.2010.NI.network.pdf

Complex network measures of brain connectivity: uses and interpretations.

I have developed "tennis elbow" from lugging this book around the past four weeks, but it is worth the pain, the effort, and the aspirin. It is also worth the (relatively speaking) bargain price. Including appendixes, this book contains 894 pages of text. The entire panorama of the neural sciences is surveyed and examined, and it is comprehensive in its scope, from genomes to social behaviors. The editors explicitly state that the book is designed as "an introductory text for students of biology, behavior, and medicine," but it is hard to imagine any audience, interested in any fragment of neuroscience at any level of sophistication, that would not enjoy this book. The editors have done a masterful job of weaving together the biologic, the behavioral, and the clinical sciences into a single tapestry in which everyone from the molecular biologist to the practicing psychiatrist can find and appreciate his or

Principles of Neural Science

Background: Although several brain regions and electrophysiological patterns have been related to sequence learning, less attention has been paid to the role that different neuromodulators play. Aims: Here we sought to investigate the role of substance P (SP) in sequence learning in an operant conditioning preparation, supported by a reinforcement learning model. Methods: Two experiments were performed to test the effects of an NK1 receptor (at which SP primarily acts) antagonist on learning and performing action sequences. In experiment 1, rats were trained to perform an action sequence until stable performance was achieved, and then, in phase 2, they were switched to perform the reverse sequence. In experiment 2, rats were trained to perform an action sequence, and in phase 2, they continued to do the same sequence. In both experiments in the first 3 days of phase 2, rats were injected with an NK1 receptor antagonist (L-733,060, i.p.) or with vehicle. Additionally, we developed a reinforcement learning model which allowed the in silico replication of our experimental tasks. Results: We found that administering an NK1 receptor antagonist weakened the stable retention of a well-learned sequence, allowing the faster acquisition of a new sequence, without impairing the continued performance of a crystallized sequence. Using our reinforcement learning model, we suggest that SP could be acting through the state value learning rate, modulating the effects of the reward prediction error. Conclusions: Our results suggest that SP could be involved in the consolidation of a sequence representation through a modulatory effect on the reward prediction error.

https://doi.org/10.1177/02698811231161582

The NK1 antagonist L-733,060 facilitates sequence learning

Under a wide range of conditions, stimuli composed of two superimposed grating components with unequal rotation velocities (differing in sign and/or magnitude) gave a striking percept of a single, coherent, nonrigidly deforming plaid surface. Conversely, component angular velocities of the same sign and magnitude yielded rigidly rotating plaids. Rigidity and motion coherence were shown to be independent percepts, and coherent plaids were categorised unambiguously as rigid or nonrigid. Coherence and motion transparency were found to depend upon the relative orientation of components, and polar plots showed two lobes of high coherence for narrow intercomponent angles. There was a slight tendency for plaids with unequal component rotations to appear less coherent, but this was nonsignificant, once the effect of intercomponent angle was taken into account. Changes in the relative spatial frequency of components, relative contrast of components, and repeated presentation produced equivalent effects on coherence for rigid and nonrigid types of plaid motion. Manipulation of the terminators in the display by making the aperture diameters for the two component gratings unequal reduced coherence and increased transparency. The effect was the same for rigid and nonrigid plaids. Coherence in rigid and nonrigid plaids thus depends primarily on local processes and there is no strong interaction between rigidity and coherence.

Coherence and motion transparency in rigid and nonrigid plaids

Recently, Bogacz and Gurney [1] proposed that the cortico-basal-ganglia system implements asymptotically optimal decision making between several alternatives, based on sensory evidence, through a statistical algorithm known as the multi-hypothesis sequential probability ratio test (MSPRT). The original programme of work focused on architectural features of the system and made simplifying assumptions as to its physiology. Here, we extend that work to include more biologically-realistic properties and pathways, and explain their impact on the MSPRT performance. One assumption in [1] was that ‘noise’ on sensory signals was Gaussian distributed, which poses the problem of interpreting the signals in the negative tail of the distribution in terms of neural events. We addressed this issue in previous work [2] by deducing a new MSPRT that proposed these signals are Inverse Gaussian distributed (wholly positive and skewed). Using realistic parameters we showed this MSPRT requires about a tenth of the samples of its Gaussian predecessor to reach a decision with the same proportion of errors. A further simplification in [1] was that the processing elements were dynamics-free and the inter-element delays were zero. In reality neural membranes display non-trivial dynamics and significant inter-neuronal/synaptic processing delays are present, both having an effect on the performance. Hence we developed a system of first order differential equations with the time constants for membranes at rest available in the literature. We found that such a system had decision sample sizes roughly twice that of the non-dynamic MSPRT. However, active membranes have reduced time constants [3]. Using values one half of those at rest gave a 30% reduction in performance, and values of one fifth of those at rest, only a 4% decrease. Furthermore, following a closer examination of the anatomy, we added new pathways to give the architecture cortex→basal-ganglia→thalamus↔cortex (arrows indicating information flow), which completes the mapping of a ‘Bayesian structure’ onto it and represents the full decision/inference circuit. Closing such loop recursively feeds back the posterior inference from sampling step t as prior ‘knowledge’ for further steps t + Δt. By doing this we also constructed another recursive statistical test (also a MSPRT) that differs from the non-recursive (feedback-less) one proposed in [1]. To test the effect on performance of including these pathways, we incorporated them into a non-dynamic MSPRT with Inverse Gaussian inputs from [2], including the main delays. The recursion increased the number of samples to reach a decision by around 30% compared with its original non-recursive counterpart. This is nevertheless a small sacrifice considering that in return the system becomes more general and ultimately able to bias further decisions by adjusting its prior knowledge. Finally, in figure ​figure11 we show the evolution of several representative elements of the loop during a decision. These results suggest that the closed loop, delayed MSPRT might hold the key to explain the time course and magnitude of the firing rates observed, in macaque electrophysiological single unit recordings, during a decision task [4].



Figure 1

Time course of firing-rate-like signals on selected stages, during a single decision trial up to a threshold, using the closed loop, delayed MSPRT (1 sample every 15 ms). Compare with figure 7 in [4].

/pdf/sequential-tests-and-biologically-grounded-multi-alternative-2wsv1olewz.pdf

Kevin Gurney

Papers

The NK1 antagonist L-733,060 facilitates sequence learning

Coherence and motion transparency in rigid and nonrigid plaids

Sequential tests and biologically grounded multi-alternative decision making

Real and artificial neurons

Blocking NK1 receptors disrupts the sequential and temporal organization of chain grooming in rats.