Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning.

/pdf/reinforcement-learning-an-introduction-rzxgej9p17.pdf

Reinforcement Learning: An Introduction

抗原变异可使得多种致病微生物易于逃避宿主免疫应答。表达在感染红细胞表面的恶性疟原虫红细胞表面蛋白1（PfPMP1）与感染红细胞、内皮细胞、树突状细胞以及胎盘的单个或多个受体作用，在黏附及免疫逃避中起关键的作用。每个单倍体基因组var基因家族编码约60种成员，通过启动转录不同的var基因变异体为抗原变异提供了分子基础。

疟原虫var基因转换速率变化导致抗原变异[英]／Paul H, Robert P, Christodoulou Z, et al//Proc Natl Acad Sci U S A

Recent developments in the quantitative analysis of complex networks, based largely on graph theory, have been rapidly translated to studies of brain network organization. The brain's structural and functional systems have features of complex networks--such as small-world topology, highly connected hubs and modularity--both at the whole-brain scale of human neuroimaging and at a cellular scale in non-human animals. In this article, we review studies investigating complex brain networks in diverse experimental modalities (including structural and functional MRI, diffusion tensor imaging, magnetoencephalography and electroencephalography in humans) and provide an accessible introduction to the basic principles of graph theory. We also highlight some of the technical challenges and key questions to be addressed by future developments in this rapidly moving field.

Complex brain networks: graph theoretical analysis of structural and functional systems

http://brainimaging.waisman.wisc.edu/~chung/neuro.processing/literature/rubinov.2010.NI.network.pdf

Complex network measures of brain connectivity: uses and interpretations.

I have developed "tennis elbow" from lugging this book around the past four weeks, but it is worth the pain, the effort, and the aspirin. It is also worth the (relatively speaking) bargain price. Including appendixes, this book contains 894 pages of text. The entire panorama of the neural sciences is surveyed and examined, and it is comprehensive in its scope, from genomes to social behaviors. The editors explicitly state that the book is designed as "an introductory text for students of biology, behavior, and medicine," but it is hard to imagine any audience, interested in any fragment of neuroscience at any level of sophistication, that would not enjoy this book. The editors have done a masterful job of weaving together the biologic, the behavioral, and the clinical sciences into a single tapestry in which everyone from the molecular biologist to the practicing psychiatrist can find and appreciate his or

Principles of Neural Science

Background Cognitive functions are found to generally decline in manifest HD. However, data published by our group (Beste et al , J Neurosci 2008; 28 :11695–702) challenged this view by showing that some cognitive domains related to attentional processes showed an elevated level of performance. While we could only speculate about the possible mechanisms in this study, we now provide neurocomputational insights in the possible neurobiological mechanisms that may mediate the counterintutive pattern of results. Methods/techniques We simulated basal ganglia (striatal) networks in computational models and systematically varied the degree on medium spiny neuron degeneration as well as the strength of NMDA receptor currents. The results of these simulations well support the empirically observed increase in pre-attentive processing. The simulations suggest that under some special constellations concerning the degree of NMDA/AMPA receptor ratio as well as the degree of medium spiny neuron degeneration, paradoxical elevated levels of cognitive processes are evident. Conclusions In HD there are cognitive subdomains, which do not undergo a decline but an improvement in functioning. We provide a neurobiological basis for further research efforts in understanding these processes.

https://jnnp.bmj.com/content/jnnp/83/Suppl_1/A37.3.full.pdf

J07 Increased cognitive functioning in manifest HD—empirical evidence and computational modelling

Associative memories: the Hopfield net

Abstract The problem of selecting one action from a set of different possible actions, simply referred to as the problem of action selection, is a ubiquitous challenge in the animal world. For vertebrates, the basal ganglia (BG) are widely thought to implement the core computation to solve this problem, as its anatomy and physiology are well suited to this end. However, the BG still display physiological features whose role in achieving efficient action selection remains unclear. In particular, it is known that the two types of dopaminergic receptors (D1 and D2) present in the BG give rise to mechanistically different responses. The overall effect will be a difference in sensitivity to dopamine, which may have ramifications for action selection. However, which receptor type leads to a stronger response is unclear due to the complexity of the intracellular mechanisms involved. In this study, we use an existing, high-level computational model of the BG, which assumes that dopamine contributes to action selection by enabling a switch between different selection regimes, to predict which of D1 or D2 has the greater sensitivity. Thus, we ask, Assuming dopamine enables a switch between action selection regimes in the BG, what functional sensitivity values would result in improved action selection computation? To do this, we quantitatively assessed the model's capacity to perform action selection as we parametrically manipulated the sensitivity weights of D1 and D2. We show that differential (rather than equal) D1 and D2 sensitivity to dopaminergic input improves the switch between selection regimes during the action selection computation in our model. Specifically, greater D2 sensitivity compared to D1 led to these improvements.

Differential Dopamine Receptor-Dependent Sensitivity Improves the Switch Between Hard and Soft Selection in a Model of the Basal Ganglia

Visual discrimination of direction changes based upon two types of angular motion.

The use of networks of logical processing elements to effect pattern matching and pattern recognition has been the subject of much research since the birth of digital microelectronic systems. In 1959, Bledsoe and Browning (59) proposed the N-Tuple method of pattern recognition. The technique offered a relatively low-cost approximation to Bayesian classification and formed the basis for many pattern recognition studies in the sixties (see for instance, Steck 62, Ullmann 69). Later, Aleksander and colleagues exploited the potential of RAM technology to develop hardware realisations of the the N-Tuple method which allowed real time learning and classification (the WIS ARD system, Aleksander 84).

Kevin Gurney

Papers

J07 Increased cognitive functioning in manifest HD—empirical evidence and computational modelling

Associative memories: the Hopfield net

Differential Dopamine Receptor-Dependent Sensitivity Improves the Switch Between Hard and Soft Selection in a Model of the Basal Ganglia

Visual discrimination of direction changes based upon two types of angular motion.

VLSI implementation of a digital neural network with reward-penalty learning