Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning.

/pdf/reinforcement-learning-an-introduction-rzxgej9p17.pdf

Reinforcement Learning: An Introduction

抗原变异可使得多种致病微生物易于逃避宿主免疫应答。表达在感染红细胞表面的恶性疟原虫红细胞表面蛋白1（PfPMP1）与感染红细胞、内皮细胞、树突状细胞以及胎盘的单个或多个受体作用，在黏附及免疫逃避中起关键的作用。每个单倍体基因组var基因家族编码约60种成员，通过启动转录不同的var基因变异体为抗原变异提供了分子基础。

疟原虫var基因转换速率变化导致抗原变异[英]／Paul H, Robert P, Christodoulou Z, et al//Proc Natl Acad Sci U S A

Recent developments in the quantitative analysis of complex networks, based largely on graph theory, have been rapidly translated to studies of brain network organization. The brain's structural and functional systems have features of complex networks--such as small-world topology, highly connected hubs and modularity--both at the whole-brain scale of human neuroimaging and at a cellular scale in non-human animals. In this article, we review studies investigating complex brain networks in diverse experimental modalities (including structural and functional MRI, diffusion tensor imaging, magnetoencephalography and electroencephalography in humans) and provide an accessible introduction to the basic principles of graph theory. We also highlight some of the technical challenges and key questions to be addressed by future developments in this rapidly moving field.

Complex brain networks: graph theoretical analysis of structural and functional systems

http://brainimaging.waisman.wisc.edu/~chung/neuro.processing/literature/rubinov.2010.NI.network.pdf

Complex network measures of brain connectivity: uses and interpretations.

I have developed "tennis elbow" from lugging this book around the past four weeks, but it is worth the pain, the effort, and the aspirin. It is also worth the (relatively speaking) bargain price. Including appendixes, this book contains 894 pages of text. The entire panorama of the neural sciences is surveyed and examined, and it is comprehensive in its scope, from genomes to social behaviors. The editors explicitly state that the book is designed as "an introductory text for students of biology, behavior, and medicine," but it is hard to imagine any audience, interested in any fragment of neuroscience at any level of sophistication, that would not enjoy this book. The editors have done a masterful job of weaving together the biologic, the behavioral, and the clinical sciences into a single tapestry in which everyone from the molecular biologist to the practicing psychiatrist can find and appreciate his or

Principles of Neural Science

The striatum is the primary input nucleus for the basal ganglia, and receives glutamatergic afferents from the cortex. Under the hypothesis that basal ganglia perform action selection, these cortical afferents encode potential `action requests'. Previous studies have suggested the striatum may utilise a mutually inhibitory network of medium spiny neurons (MSNs) to filter these requests so that only those of high salience are selected. However, the mechanisms enabling the striatum to perform clean, rapid switching between distinct actions that form part of a learned action sequence are still poorly understood. Substance P (SP) and enkephalin are neuropeptides co-released with GABA in MSNs preferentially expressing D1 or D2 dopamine receptors respectively. SP has a facilitatory effect on subsequent glutamatergic inputs to target MSNs, while enkephalin has an inhibitory effect. Blocking the action of SP in the striatum is also known to affect behavioural transitions. We constructed phenomenological models of the effects of SP and enkephalin, and integrated these into a hybrid model of basal ganglia comprising a spiking striatal microcircuit and rate--coded populations representing other major structures. We demonstrated that diffuse neuropeptide connectivity enhanced the selection of unordered action requests, and that for true action sequences, where action semantics define a fixed structure, a patterning of the SP connectivity reflecting this ordering enhanced selection of actions presented in the correct sequential order and suppressed incorrect ordering. We also showed that selective pruning of SP connections allowed context--sensitive inhibition of specific undesirable requests that otherwise interfered with selection of an action group. Our model suggests that the interaction of SP and enkephalin enhances the contrast between selection and rejection of action requests, and that patterned SP connectivity in the striatum allows the `chunking' of actions and improves selection of sequences. Efficient execution of action sequences may therefore result from a combination of ordered cortical inputs and patterned neuropeptide connectivity within striatum.

/pdf/striatal-neuropeptides-enhance-selection-and-rejection-of-3fk5prx5z4.pdf

Striatal Neuropeptides Enhance Selection and Rejection of Sequential Actions.

We present an application of a massively parallel processor array VLSI circuit to the implementation of neural networks in complex architectural arrangements. The work was motivated by existing biologically plausible models of a set of sub-cortical nuclei -the basal ganglia. The model includes 5 layers, each consisting of 16384 leaky integrator neurons, with inter-layer synaptic weights forming various one-to-one and diffuse connectivity patterns. The architecture of the SIMD processor array allows all the neurons per layer to be updated simultaneously. The performance of the processor array chip in simulating the model is compared with the original model being executed on a computer workstation. It is demonstrated that in this application the chip outperforms the workstation by five orders of magnitude in terms of computational performance and seven orders of magnitude in terms of energy efficiency, providing a high-speed, low-power, compact hardware platform for possible embedded robotic applications.

/pdf/implementation-of-multi-layer-leaky-integrator-networks-on-a-31v11aw6lh.pdf

Implementation of multi-layer leaky integrator networks on a cellular processor array

There is a growing requirement in computational neuroscience for tools that permit collaborative model building, model sharing, combining existing models into a larger system (multi-scale model integration), and are able to simulate models using a variety of simulation engines and hardware platforms. Layered XML model specification formats solve many of these problems, however they are difficult to write and visualise without tools. Here we describe a new graphical software tool, SpineCreator, which facilitates the creation and visualisation of layered models of point spiking neurons or rate coded neurons without requiring the need for programming. We demonstrate the tool through the reproduction and visualisation of published models and show simulation results using code generation interfaced directly into SpineCreator. As a unique application for the graphical creation of neural networks, SpineCreator represents an important step forward for neuronal modelling.

/pdf/spinecreator-a-graphical-user-interface-for-the-creation-of-3n85cbue9i.pdf

SpineCreator: a Graphical User Interface for the Creation of Layered Neural Models

Spiking neural network simulations incorporating variable transmission delays require synaptic events to be scheduled prior to delivery. Conventional methods have memory requirements that scale with the total number of synapses in a network. We introduce novel scheduling algorithms for both discrete and continuous event delivery, where the memory requirement scales instead with the number of neurons. Superior algorithmic performance is demonstrated using large-scale, benchmarking network simulations.

Kevin Gurney

Papers

Striatal Neuropeptides Enhance Selection and Rejection of Sequential Actions.

Implementation of multi-layer leaky integrator networks on a cellular processor array

SpineCreator: a Graphical User Interface for the Creation of Layered Neural Models

Spiking neural network simulation: memory-optimal synaptic event scheduling

A model for the spatial integration and differentiation of velocity signals.