Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning.

/pdf/reinforcement-learning-an-introduction-rzxgej9p17.pdf

Reinforcement Learning: An Introduction

抗原变异可使得多种致病微生物易于逃避宿主免疫应答。表达在感染红细胞表面的恶性疟原虫红细胞表面蛋白1（PfPMP1）与感染红细胞、内皮细胞、树突状细胞以及胎盘的单个或多个受体作用，在黏附及免疫逃避中起关键的作用。每个单倍体基因组var基因家族编码约60种成员，通过启动转录不同的var基因变异体为抗原变异提供了分子基础。

疟原虫var基因转换速率变化导致抗原变异[英]／Paul H, Robert P, Christodoulou Z, et al//Proc Natl Acad Sci U S A

Inspired by empirical studies of networked systems such as the Internet, social networks, and biological networks, researchers have in recent years developed a variety of techniques and models to help us understand or predict the behavior of these systems. Here we review developments in this field, including such concepts as the small-world effect, degree distributions, clustering, network correlations, random graph models, models of network growth and preferential attachment, and dynamical processes taking place on networks.

/pdf/the-structure-and-function-of-complex-networks-34deompc9l.pdf

The Structure and Function of Complex Networks

Probability Distributions.- Linear Models for Regression.- Linear Models for Classification.- Neural Networks.- Kernel Methods.- Sparse Kernel Machines.- Graphical Models.- Mixture Models and EM.- Approximate Inference.- Sampling Methods.- Continuous Latent Variables.- Sequential Data.- Combining Models.

Pattern Recognition and Machine Learning

Recent developments in the quantitative analysis of complex networks, based largely on graph theory, have been rapidly translated to studies of brain network organization. The brain's structural and functional systems have features of complex networks--such as small-world topology, highly connected hubs and modularity--both at the whole-brain scale of human neuroimaging and at a cellular scale in non-human animals. In this article, we review studies investigating complex brain networks in diverse experimental modalities (including structural and functional MRI, diffusion tensor imaging, magnetoencephalography and electroencephalography in humans) and provide an accessible introduction to the basic principles of graph theory. We also highlight some of the technical challenges and key questions to be addressed by future developments in this rapidly moving field.

Complex brain networks: graph theoretical analysis of structural and functional systems

STUDY OBJECTIVES
1) To replicate the finding that lucid dreams are associated with physiological activation, including heightened REM density, during REM sleep. 2) To critically test whether a previously reported increase in frontolateral 40 Hz power in lucid REM sleep, used to justify the claim that lucid dreaming is a "hybrid state" mixing sleep and wakefulness, is attributable to the saccadic spike potential (SP) artifact as a corollary of heightened REM density. 3) To conduct an exploratory analysis of changes in EEG features during lucid REM sleep.


METHODS
We analyzed 14 signal-verified lucid dreams (SVLDs) and baseline REM sleep segments from the same REM periods from six participants derived from the Stanford SVLD database. Participants marked lucidity onset with standard left-right-left-right-center (LR2c) eye-movement signals in polysomnography recordings.


RESULTS
Compared to baseline REM sleep, lucid REM sleep had higher REM density (p=0.002). Bayesian analysis supported the null hypothesis of no differences in frontolateral 40 Hz power after removal of the SP artifact (BH=0.18) and ICA correction (BH=0.01). Compared to the entire REM sleep period, lucid REM sleep showed small reductions in low-frequency and beta band spectral power as well as increased signal complexity (all p<0.05), which were within the normal variance of baseline REM sleep.


CONCLUSIONS
Lucid dreams are associated with higher-than-average levels of physiological activation during REM sleep, including measures of both subcortical and cortical activation. Increases in 40 Hz power in periorbital channels reflect saccadic and microsaccadic SPs as a result of higher REM density accompanying heightened activation.

Lucid Dreaming Occurs in Activated REM Sleep, Not a Mixture of Sleep and Wakefulness.

STUDY OBJECTIVES
(1) To critically test whether a previously reported increase in frontolateral 40 Hz power in lucid REM sleep, used to justify the claim that lucid dreaming is a "hybrid state" mixing sleep and wakefulness, is attributable to the saccadic spike potential (SP) artifact as a corollary of heightened REM density. (2) To replicate the finding that lucid dreams are associated with physiological activation, including heightened eye movement density, during REM sleep. (3) To conduct an exploratory analysis of changes in EEG features during lucid REM sleep.


METHODS
We analyzed 14 signal-verified lucid dreams (SVLDs) and baseline REM sleep segments from the same REM periods from six participants derived from the Stanford SVLD database. Participants marked lucidity onset with standard left-right-left-right-center (LR2c) eye-movement signals in polysomnography recordings.


RESULTS
Compared to baseline REM sleep, lucid REM sleep had higher REM density (β = 0.85, p = 0.002). Bayesian analysis supported the null hypothesis of no differences in frontolateral 40 Hz power after removal of the SP artifact (BH = 0.18) and ICA correction (BH = 0.01). Compared to the entire REM sleep period, lucid REM sleep showed small reductions in low-frequency and beta band spectral power as well as increased signal complexity (all p < 0.05), which were within the normal variance of baseline REM sleep.


CONCLUSIONS
Lucid dreams are associated with higher-than-average levels of physiological activation during REM sleep, including measures of both subcortical and cortical activation. Increases in 40 Hz power in periorbital channels reflect saccadic and microsaccadic SPs as a result of higher REM density accompanying heightened activation.

Lucid dreaming occurs in activated rapid eye movement sleep, not a mixture of sleep and wakefulness.

The invention provides methods of identifying compounds that alters vigilance, by contacting an invertebrate with a candidate compound, evaluating a vigilance property in the contacted invertebrate, and determining if the candidate compound alters the vigilance property in the contacted invertebrate. Also provided are isolated vigilance nucleic acid molecules. Methods for diagnosing and treating vigilance disorders, for determining and altering vigilance levels, and for screening for therapeutic compounds useful for treating vigilance disorders and altering vigilance level, are also provided.

Vigilance nucleic acids and related diagnostic, screening and therapeutic methods

Sleep can be distinguished from wake by changes in brain electrical activity, typically assessed using electroencephalography (EEG). The hallmark of non-rapid-eye-movement sleep are two major EEG events: slow waves and spindles. Here we sought to identify possible signatures of sleep in brain hemodynamic activity, using simultaneous fMRI-EEG. We found that, during the transition from wake to sleep, blood-oxygen-level-dependent (BOLD) activity evolved from a mixed-frequency pattern to one dominated by two distinct oscillations: a low-frequency (~0.05Hz) oscillation prominent in light sleep and a high-frequency (~0.17Hz) oscillation in deep sleep. The two BOLD oscillations correlated with the occurrences of spindles and slow waves, respectively. They were detectable across the whole brain, cortically and subcortically, but had different regional distributions and opposite onset patterns. These spontaneous BOLD oscillations provide fMRI signatures of basic sleep processes, which may be employed to study human sleep at spatial resolution and brain coverage not achievable using EEG. HIGHLIGHTS spontaneous BOLD oscillations differentiate sleep from wake low-frequency BOLD oscillation tracks sleep spindles high-frequency BOLD oscillation tracks sleep slow waves BOLD oscillations provide fMRI signatures of key sleep processes

https://www.biorxiv.org/content/biorxiv/early/2019/01/26/531186.full.pdf

BOLD signatures of sleep

Transcriptomic studies revealed that hundreds of mRNAs show differential expression in the brains of sleeping relative to awake rats, mice, flies, and sparrows. Although these results have offered clues regarding the molecular consequences of sleep and sleep loss, their functional significance thus far has been limited. This is probably because the previous studies pooled transcripts from all brain cells, including neurons and glia. In Bellesi et al. 2015 [1], we used the translating ribosome affinity purification technology (TRAP) and microarray analysis to obtain a genome-wide mRNA profiling of astrocytes as a function of sleep and wake. We used bacterial artificial chromosome (BAC) transgenic mice expressing eGFP tagged ribosomal protein L10a under the promoter of the Aldh1L1 gene, a highly expressed astrocytic gene. Using this approach, we could extract only the astrocytic mRNAs, and only those already committed to be translated into proteins (L10a is part of the translational machinery). Here, we report a detailed description of the protocol used in the study [1]. Array data have been submitted to NCBI GEO under accession number (GSE69079).

Giulio Tononi

Papers

Lucid Dreaming Occurs in Activated REM Sleep, Not a Mixture of Sleep and Wakefulness.

Lucid dreaming occurs in activated rapid eye movement sleep, not a mixture of sleep and wakefulness.

Vigilance nucleic acids and related diagnostic, screening and therapeutic methods

BOLD signatures of sleep

Transcriptome profiling of sleeping, waking, and sleep deprived adult heterozygous Aldh1L1–eGFP-L10a mice