Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning.

/pdf/reinforcement-learning-an-introduction-rzxgej9p17.pdf

Reinforcement Learning: An Introduction

抗原变异可使得多种致病微生物易于逃避宿主免疫应答。表达在感染红细胞表面的恶性疟原虫红细胞表面蛋白1（PfPMP1）与感染红细胞、内皮细胞、树突状细胞以及胎盘的单个或多个受体作用，在黏附及免疫逃避中起关键的作用。每个单倍体基因组var基因家族编码约60种成员，通过启动转录不同的var基因变异体为抗原变异提供了分子基础。

疟原虫var基因转换速率变化导致抗原变异[英]／Paul H, Robert P, Christodoulou Z, et al//Proc Natl Acad Sci U S A

Recent developments in the quantitative analysis of complex networks, based largely on graph theory, have been rapidly translated to studies of brain network organization. The brain's structural and functional systems have features of complex networks--such as small-world topology, highly connected hubs and modularity--both at the whole-brain scale of human neuroimaging and at a cellular scale in non-human animals. In this article, we review studies investigating complex brain networks in diverse experimental modalities (including structural and functional MRI, diffusion tensor imaging, magnetoencephalography and electroencephalography in humans) and provide an accessible introduction to the basic principles of graph theory. We also highlight some of the technical challenges and key questions to be addressed by future developments in this rapidly moving field.

Complex brain networks: graph theoretical analysis of structural and functional systems

http://brainimaging.waisman.wisc.edu/~chung/neuro.processing/literature/rubinov.2010.NI.network.pdf

Complex network measures of brain connectivity: uses and interpretations.

I have developed "tennis elbow" from lugging this book around the past four weeks, but it is worth the pain, the effort, and the aspirin. It is also worth the (relatively speaking) bargain price. Including appendixes, this book contains 894 pages of text. The entire panorama of the neural sciences is surveyed and examined, and it is comprehensive in its scope, from genomes to social behaviors. The editors explicitly state that the book is designed as "an introductory text for students of biology, behavior, and medicine," but it is hard to imagine any audience, interested in any fragment of neuroscience at any level of sophistication, that would not enjoy this book. The editors have done a masterful job of weaving together the biologic, the behavioral, and the clinical sciences into a single tapestry in which everyone from the molecular biologist to the practicing psychiatrist can find and appreciate his or

Principles of Neural Science

The striatum is the man input structure of the basal ganglia and consists principally of medium spiny neurons (MSNs). The remaining neurons comprise several species of interneuron, including the GABAergic fast spiking interneuron (FSIs). Both neuron species are highly interconnected (including a network of gap junctions between the FSIs) and both are modulated by dopamine. Understanding this complex microcircuit is therefore very challenging. Previous computational hypotheses have suggested that the inhibitory collaterals between MSNs lead to a strong competitive dynamic [1]. In contrast, Koos and Tepper [2] suggest that feed-forward inhibition from the FSIs is the dominant force in the control of MSNs. We have developed a detailed, biologically constrained model of the striatal microcircuit aimed at resolving these issues and discovering the computations performed in this critical brain area.

/pdf/does-striatum-support-competitive-dynamics-a-test-of-this-1pls6gyrbs.pdf

Does striatum support competitive dynamics? A test of this hypothesis using a biologically realistic model of the striatal microcircuit

Computational models in neuroscience: From membranes to robots

Loss of dopamine cells in Parkinson's Disease and its ani-mal models leads to profound motor deficits. An intactdopamine system also seems to be critical for many formsof learning [1]. Much work on understanding these rolesof dopamine has focused on the striatum, the main inputnucleus of the basal ganglia, as the striatum is the mainlocus of dopamine's action in the verterbrate brain [2].Damage to the striatum itself also impairs both motoraction and learning [3]. Thus, the twin problems of under-standing the computational roles of dopamine and thestriatum are inseparably intertwined.Understanding dopamine's effects on the complex striatalmicrocircuit ideally requires large-scale models that repli-cate the neuron types, numbers, and connectivity at one-to-one scale. To build at such scales, we require individualneuron models that are simple enough to be computa-tionally tractable, but sufficiently complex to capture keymembrane properties that contribute to the characteristicbehavior of a neuron species. Our neuron model of choiceis the recent canonical spiking model of Izhikevich [4].However, it has not yet been extended to account for theaction of neuromodulators.We extend the striatal medium spiny (MSN) and fast-spik-ing (FS) interneuron models of [5] to account fordopaminergic modulation of intrinsic ion channels andsynaptic inputs. We use data from a recent 189 compart-ment model of the MSN [3] to tune our simple model ofthat neuron under both current injection and spikinginput regimes with varying activation of dopamine D1-and D2-type receptors. The reduced models capture theinput-output relationships for both current injection andspiking input with remarkable accuracy. We derive a fullset of stability properties for the original and dopaminemodulated forms of the MSN model. We use these toestablish that the dopamine models do not change thestability properties and hence the models predict that theMSN is not bistable in either baseline or dopamine-satu-rated conditions. Our extensions to the simple model ofthe FS interneuron are consistent with the existing data,but tuning the new parameters is made difficult by thelack of quantitative results from experimental work. Ourwork thus establishes reduced yet accurate dopamine-modulated forms of MSN and FS interneuron models,suitable for use in large-scale models of the striatum.Moreover, these also provide a tractable framework forfurther study of dopamine's effects on individual neuroncomputation.

/pdf/reduced-models-of-striatal-neurons-dopamine-modulation-and-3hzzcqxzl4.pdf

Kevin Gurney

Papers

Does striatum support competitive dynamics? A test of this hypothesis using a biologically realistic model of the striatal microcircuit

Computational models in neuroscience: From membranes to robots

Reduced models of striatal neurons: dopamine modulation and dynamics