Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning.

/pdf/reinforcement-learning-an-introduction-rzxgej9p17.pdf

Reinforcement Learning: An Introduction

1980 Preface * 1999 Preface * 1999 Acknowledgements * Introduction * 1 Circular Logic * 2 Phase Singularities (Screwy Results of Circular Logic) * 3 The Rules of the Ring * 4 Ring Populations * 5 Getting Off the Ring * 6 Attracting Cycles and Isochrons * 7 Measuring the Trajectories of a Circadian Clock * 8 Populations of Attractor Cycle Oscillators * 9 Excitable Kinetics and Excitable Media * 10 The Varieties of Phaseless Experience: In Which the Geometrical Orderliness of Rhythmic Organization Breaks Down in Diverse Ways * 11 The Firefly Machine 12 Energy Metabolism in Cells * 13 The Malonic Acid Reagent ('Sodium Geometrate') * 14 Electrical Rhythmicity and Excitability in Cell Membranes * 15 The Aggregation of Slime Mold Amoebae * 16 Numerical Organizing Centers * 17 Electrical Singular Filaments in the Heart Wall * 18 Pattern Formation in the Fungi * 19 Circadian Rhythms in General * 20 The Circadian Clocks of Insect Eclosion * 21 The Flower of Kalanchoe * 22 The Cell Mitotic Cycle * 23 The Female Cycle * References * Index of Names * Index of Subjects

https://www.cambridge.org/core/services/aop-cambridge-core/content/view/596B270197FE27DE5FABB598B6210622/S0025557200150892a.pdf/div-class-title-span-class-italic-the-geometry-of-biological-time-span-by-a-t-winfree-pp-544-dm68-corrected-second-printing-1990-isbn-3-540-52528-9-springer-div.pdf

The geometry of biological time , by A. T. Winfree. Pp 544. DM68. Corrected Second Printing 1990. ISBN 3-540-52528-9 (Springer)

The spatiotemporal expression of genes in an organism is determined by regulatory systems that involve a large number of genes connected through a complex network of interactions. As an intuitive understanding of the behavior of these systems is hard to obtain, computer tools for the modeling and simulation of genetic regulatory networks will be indispensable. This report reviews formalisms that have been employed in mathematical biology and bioinformatics to describe genetic regulatory systems, in particular directed graphs, Bayesian networks, ordinary and partial differential equations, stochastic equations, Boolean networks and their generalizations, qualitative differential equations, and rule-based formalisms. In addition, the report discusses how these formalisms have been used in the modeling and simulation of regulatory systems.

/pdf/modeling-and-simulation-of-genetic-regulatory-systems-a-2pgggkutcb.pdf

Modeling and simulation of genetic regulatory systems: a literature review.

http://www.columbia.edu/itc/gsas/g9600/2003/JavitchReadings/Schultz(Sulzer).pdf

Getting Formal with Dopamine and Reward

Initial-value ordinary differential equation solution via variable order Adams method (SIVA/DIVA) package is collection of subroutines for solution of nonstiff ordinary differential equations. There are versions for single-precision and double-precision arithmetic. Requires fewer evaluations of derivatives than other variable-order Adams predictor/ corrector methods. Option for direct integration of second-order equations makes integration of trajectory problems significantly more efficient. Written in FORTRAN 77.

Solving Ordinary Differential Equations

Mathematical Modeling of Gene Networks

Modeling transcriptional control in gene networks--methods, recent results, and future directions.

Operant conditioning is a form of associative learning through which an animal learns about the consequences of its behavior. Here, we report an appetitive operant conditioning procedure inAplysia that induces long-term memory. Biophysical changes that accompanied the memory were found in an identified neuron (cell B51) that is considered critical for the expression of behavior that was rewarded. Similar cellular changes in B51 were produced by contingent reinforcement of B51 with dopamine in a single-cell analog of the operant procedure. These findings allow for the detailed analysis of the cellular and molecular processes underlying operant conditioning.

/pdf/operant-reward-learning-in-aplysia-neuronal-correlates-and-1wm2161svq.pdf

Operant Reward Learning in Aplysia: Neuronal Correlates and Mechanisms

Both positive and negative feedback loops of transcriptional regulation have been proposed to be important for the generation of circadian rhythms. To test the sufficiency of the proposed mechanisms, two differential equation-based models were constructed to describe the Neurospora crassa and Drosophila melanogaster circadian oscillators. In the model of the Neurospora oscillator, FRQ suppresses frq transcription by binding to a complex of the transcriptional activators WC-1 and WC-2, thus yielding negative feedback. FRQ also activates synthesis of WC-1, which in turn activates frq transcription, yielding positive feedback. In the model of the Drosophila oscillator, PER and TIM are represented by a “lumped” variable, “PER.” PER suppresses its own transcription by binding to the transcriptional regulator dCLOCK, thus yielding negative feedback. PER also binds to dCLOCK to de-repress dclock , and dCLOCK in turn activates per transcription, yielding positive feedback. Both models displayed circadian oscillations that were robust to parameter variations and to noise and that entrained to simulated light/dark cycles. Circadian oscillations were only obtained if time delays were included to represent processes not modeled in detail (e.g., transcription and translation). In both models, oscillations were preserved when positive feedback was removed.

https://www.jneurosci.org/content/jneuro/21/17/6644.full.pdf

Modeling circadian oscillations with interlocking positive and negative feedback loops.

To examine the capability of genetic regulatory systems for complex dynamic activity, we developed simple kinetic models that incorporate known features of these systems. These include autoregulation and stimulus-dependent phosphorylation of transcription factors (TFs), dimerization of TFs, crosstalk, and feedback. The simplest model manifested multiple stable steady states, and brief perturbations could switch the model between these states. Such transitions might explain, for example, how a brief pulse of hormone or neurotransmitter could elicit a long-lasting cellular response. In slightly more complex models, oscillatory regimes were identified. The addition of competition between activating and repressing TFs provided a plausible explanation for optimal stimulus frequencies that give maximal transcription. Such optimal frequencies are suggested by recent experiments comparing training paradigms for long-term memory formation and examining changes in mRNA levels in repetitively stimulated cultured cells. In general, the computational approach illustrated here, combined with appropriate experiments, provides a conceptual framework for investigating the function of genetic regulatory systems.

https://nba.uth.tmc.edu/homepage/jbyrne/assets/pdf/publications/smolen_et_al_ajp_1998.pdf

Douglas A. Baxter

Papers

Mathematical Modeling of Gene Networks

Modeling transcriptional control in gene networks--methods, recent results, and future directions.

Operant Reward Learning in Aplysia: Neuronal Correlates and Mechanisms

Modeling circadian oscillations with interlocking positive and negative feedback loops.

Frequency selectivity, multistability, and oscillations emerge from models of genetic regulatory systems