Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning.

/pdf/reinforcement-learning-an-introduction-rzxgej9p17.pdf

Reinforcement Learning: An Introduction

抗原变异可使得多种致病微生物易于逃避宿主免疫应答。表达在感染红细胞表面的恶性疟原虫红细胞表面蛋白1（PfPMP1）与感染红细胞、内皮细胞、树突状细胞以及胎盘的单个或多个受体作用，在黏附及免疫逃避中起关键的作用。每个单倍体基因组var基因家族编码约60种成员，通过启动转录不同的var基因变异体为抗原变异提供了分子基础。

疟原虫var基因转换速率变化导致抗原变异[英]／Paul H, Robert P, Christodoulou Z, et al//Proc Natl Acad Sci U S A

Inspired by empirical studies of networked systems such as the Internet, social networks, and biological networks, researchers have in recent years developed a variety of techniques and models to help us understand or predict the behavior of these systems. Here we review developments in this field, including such concepts as the small-world effect, degree distributions, clustering, network correlations, random graph models, models of network growth and preferential attachment, and dynamical processes taking place on networks.

/pdf/the-structure-and-function-of-complex-networks-34deompc9l.pdf

The Structure and Function of Complex Networks

Probability Distributions.- Linear Models for Regression.- Linear Models for Classification.- Neural Networks.- Kernel Methods.- Sparse Kernel Machines.- Graphical Models.- Mixture Models and EM.- Approximate Inference.- Sampling Methods.- Continuous Latent Variables.- Sequential Data.- Combining Models.

Pattern Recognition and Machine Learning

Recent developments in the quantitative analysis of complex networks, based largely on graph theory, have been rapidly translated to studies of brain network organization. The brain's structural and functional systems have features of complex networks--such as small-world topology, highly connected hubs and modularity--both at the whole-brain scale of human neuroimaging and at a cellular scale in non-human animals. In this article, we review studies investigating complex brain networks in diverse experimental modalities (including structural and functional MRI, diffusion tensor imaging, magnetoencephalography and electroencephalography in humans) and provide an accessible introduction to the basic principles of graph theory. We also highlight some of the technical challenges and key questions to be addressed by future developments in this rapidly moving field.

Complex brain networks: graph theoretical analysis of structural and functional systems

The present disclosure pertains to a system configured to adjust an intensity of sensory stimulation delivered to a subject during a sleep session based on sleep spindles in the subject during the sleep session. The system is configured to adjust the intensity of the stimulation based on a sleep spindle frequency and/or a sleep spindle density. The system is configured to determine a recent spindle density and/or a recent spindle frequency for a recent period of time during the sleep session based on detected sleep spindles, and to determine a previous spindle density and/or a previous spindle frequency for a previous period of time during the sleep session based on the detected sleep spindles. The system controls the intensity of sensory stimulation provided to the subject based on a comparison of the previous spindle density to the recent spindle density and/or the previous spindle frequency to the recent spindle frequency.

System for adjusting the intensity of sensory stimulation during sleep based on sleep spindles

Unanesthetized Rats System Are Present in Anesthetized but not in Evoked Oscillations in the Thalamo-Cortical Auditory

Causal reductionism is the widespread assumption that there is no room for additional causes once we have accounted for all elementary mechanisms within a system. Due to its intuitive appeal, causal reductionism is prevalent in neuroscience: once all neurons have been caused to fire or not to fire, it seems that causally there is nothing left to be accounted for. Here, we argue that these reductionist intuitions are based on an implicit, unexamined notion of causation that conflates causation with prediction. By means of a simple model organism, we demonstrate that causal reductionism cannot provide a complete and coherent account of ‘what caused what’. To that end, we outline an explicit, operational approach to analyzing causal structures. In this Perspective, Tononi and colleagues argue that while knowledge of elementary mechanisms is enough to predict everything about the dynamics of a system, only the analysis of causal structures can provide a coherent account of ‘what caused what’.

Causal reductionism and causal structures.

Taskin a Choice-Reaction Time Reaching Temporal Evolution of Oscillatory Activity Predicts

Loss of consciousness (LOC) is a hallmark of many epileptic seizures and carries risks of serious injury and sudden death. While cortical sleep-like activities accompany LOC during focal impaired awareness (FIA) seizures, the mechanisms of LOC during focal to bilateral tonic-clonic (FBTC) seizures remain unclear. Quantifying differences in markers of cortical activation and ictal recruitment between FIA and FBTC seizures may also help to understand their different consequences for clinical outcomes and to optimize neuromodulation therapies. We quantified clinical signs of LOC and intracranial EEG (iEEG) activity during 129 FIA and 50 FBTC from 41 patients. We characterized iEEG changes both in the seizure onset zone (SOZ) and in areas remote from SOZ with a total of 3386 electrodes distributed across brain areas. First, we compared the dynamics of iEEG sleep-like activities: slow-wave activity (SWA; 1-4 Hz) and beta/delta ratio (B/D; a validated marker of cortical activation) during FIA vs. FBTC. Second, we quantified differences between FBTC and FIA for a marker validated to detect ictal cross-frequency coupling: phase-locked high-gamma (PLHG; high gamma phased locked to low frequencies) and a marker of ictal recruitment: the epileptogenicity index (i.e. the number of channels crossing an energy ratio threshold for high vs. low frequency power). Third, we assessed changes in iEEG activity preceding and accompanying behavioral generalization onset and their correlation with electromyogram (EMG) channels. In addition, we analyzed human cortical multi-unit activity recorded with Utah arrays during three FBTC. Compared to FIA, FBTC seizures were characterized by deeper LOC and by stronger increases in SWA in parieto-occipital cortex. FBTC also displayed more widespread increases in cortical activation (B/D), ictal cross-frequency coupling (PLHG) and ictal recruitment (epileptogenicity index). Even before generalization, FBTC displayed deeper LOC; this early LOC was accompanied by a paradoxical increase in B/D in fronto-parietal cortex. Behavioral generalization coincided with complete loss of responsiveness and a subsequent increase in high-gamma in the whole brain, which was especially synchronous in deep sources and could not be explained by EMG. Similarly, multi-unit activity analysis of FBTC revealed sustained increases in cortical firing rates during and after generalization onset in areas remote from the SOZ. Unlike during FIA, LOC during FBTC is characterized by a paradoxical increase in cortical activation and neuronal firing. These findings suggest differences in the mechanisms of ictal LOC between FIA and FBTC and may account for the more negative prognostic consequences of FBTC.

/pdf/distinct-signatures-of-loss-of-consciousness-during-focal-378vyc7uiu.pdf

Giulio Tononi

Papers

System for adjusting the intensity of sensory stimulation during sleep based on sleep spindles

Unanesthetized Rats System Are Present in Anesthetized but not in Evoked Oscillations in the Thalamo-Cortical Auditory

Causal reductionism and causal structures.

Taskin a Choice-Reaction Time Reaching Temporal Evolution of Oscillatory Activity Predicts

Distinct signatures of loss of consciousness during Focal Impaired Awareness (FIA) versus Focal to Bilateral Tonic Clonic (FBTC) seizures