Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning.

/pdf/reinforcement-learning-an-introduction-rzxgej9p17.pdf

Reinforcement Learning: An Introduction

抗原变异可使得多种致病微生物易于逃避宿主免疫应答。表达在感染红细胞表面的恶性疟原虫红细胞表面蛋白1（PfPMP1）与感染红细胞、内皮细胞、树突状细胞以及胎盘的单个或多个受体作用，在黏附及免疫逃避中起关键的作用。每个单倍体基因组var基因家族编码约60种成员，通过启动转录不同的var基因变异体为抗原变异提供了分子基础。

疟原虫var基因转换速率变化导致抗原变异[英]／Paul H, Robert P, Christodoulou Z, et al//Proc Natl Acad Sci U S A

Inspired by empirical studies of networked systems such as the Internet, social networks, and biological networks, researchers have in recent years developed a variety of techniques and models to help us understand or predict the behavior of these systems. Here we review developments in this field, including such concepts as the small-world effect, degree distributions, clustering, network correlations, random graph models, models of network growth and preferential attachment, and dynamical processes taking place on networks.

/pdf/the-structure-and-function-of-complex-networks-34deompc9l.pdf

The Structure and Function of Complex Networks

Probability Distributions.- Linear Models for Regression.- Linear Models for Classification.- Neural Networks.- Kernel Methods.- Sparse Kernel Machines.- Graphical Models.- Mixture Models and EM.- Approximate Inference.- Sampling Methods.- Continuous Latent Variables.- Sequential Data.- Combining Models.

Pattern Recognition and Machine Learning

Recent developments in the quantitative analysis of complex networks, based largely on graph theory, have been rapidly translated to studies of brain network organization. The brain's structural and functional systems have features of complex networks--such as small-world topology, highly connected hubs and modularity--both at the whole-brain scale of human neuroimaging and at a cellular scale in non-human animals. In this article, we review studies investigating complex brain networks in diverse experimental modalities (including structural and functional MRI, diffusion tensor imaging, magnetoencephalography and electroencephalography in humans) and provide an accessible introduction to the basic principles of graph theory. We also highlight some of the technical challenges and key questions to be addressed by future developments in this rapidly moving field.

Complex brain networks: graph theoretical analysis of structural and functional systems

: The purpose of this study is to assess sleep and wake parameters in veterans of the first Gulf War who have fatigue and other symptoms compared with veterans who do not have fatigue utilizing novel assessment techniques including high density EEG and temperature. This research study is in the data collection and data processing phase. The most significant findings to date in this study during the research period is that high density EEG marked broad band reduction in neural activity circumscribed in the frontal cortex in NREM sleep. Slow wave sleep is often thought to play a role in the recovery and restorative aspects of sleep, and is one bandwidth affected. Additionally, previously noted temperature curves, which are well-tied with sleep/wake and feelings of fatigue/alertness show different projections in veterans endorsing fatigue than those who do not.

/pdf/homeostatic-and-circadian-abnormalities-in-sleep-and-arousal-hhuc004el7.pdf

Homeostatic and Circadian Abnormalities in Sleep and Arousal in Gulf War Syndrome

If we take the subjective character of consciousness seriously, consciousness becomes a matter of "being" rather than "doing" Because "doing" can be dissociated from "being", functional criteria alone are insufficient to decide whether a system possesses the necessary requirements for being a physical substrate of consciousness The dissociation between "being" and "doing" is most salient in artificial general intelligence, which may soon replicate any human capacity: computers can perform complex functions (in the limit resembling human behavior) in the absence of consciousness Complex behavior becomes meaningless if it is not performed by a conscious being

What we are is more than what we do.

Awake, Behaving Primates Recording Chronically From the Same Neurons in

Can intense learning induce neuronal fatigue in well rested subjects? To answer this question, we recorded high density EEG during extended practice in a visuo-motor adaptation. We found that prolonged learning leads to a progressive slowing of electroencephalographic (EEG) activity over the cortical areas engaged by the task itself. Extended daytime training resulted in task-specific and local sEEG power increase in the low frequency range over areas that were engaged during training. This was accompanied by increased performance errors in the test that shared the task characteristics. Finally, an afternoon nap but not an equivalent period of quiet wake renormalized the local EEG changes induced by intense training and restored both performance and the ability to learn.

The Neural Effects of Extended Practice and the Benefits of a Nap

Although the EEG slow wave of sleep is typically considered to be a hallmark of Non Rapid Eye Movement (NREM) sleep, recent work in mice has shown that slow waves can also occur in REM sleep. Here we investigated the presence and cortical distribution of low-frequency (1-4 Hz) oscillations in human REM sleep by analyzing high-density EEG sleep recordings obtained in 28 healthy subjects. We identified two clusters of low-frequency oscillations with distinctive properties: 1) a fronto-central cluster characterized by ~2.5-3.0 Hz, relatively large, notched delta waves (so-called "sawtooth waves") that tended to occur in bursts, were associated with increased gamma activity and rapid eye movements, and upon source modeling, displayed an occipito-temporal and a fronto-central component; and 2) a medial occipital cluster characterized by more isolated, slower (<2 Hz) and smaller waves that were not associated with rapid eye movements, displayed a negative correlation with gamma activity and were also found in NREM sleep. Thus, low-frequency oscillations are an integral part of REM sleep in humans, and the two identified subtypes (sawtooth and medial-occipital slow waves) may reflect distinct generation mechanisms and functional roles. Sawtooth waves, which are exclusive to REM sleep, share many characteristics with ponto-geniculo-occipital (PGO) waves described in animals and may represent the human equivalent or a closely related event while medio-occipital slow waves appear similar to NREM sleep slow waves.

/pdf/regional-low-frequency-oscillations-in-human-rapid-eye-1x1z4at0xl.pdf

Giulio Tononi

Papers

Homeostatic and Circadian Abnormalities in Sleep and Arousal in Gulf War Syndrome

What we are is more than what we do.

Awake, Behaving Primates Recording Chronically From the Same Neurons in

The Neural Effects of Extended Practice and the Benefits of a Nap

Regional low-frequency oscillations in human rapid-eye movement sleep