Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning.

/pdf/reinforcement-learning-an-introduction-rzxgej9p17.pdf

Reinforcement Learning: An Introduction

Long-term potentiation of synaptic transmission in the hippocampus is the primary experimental model for investigating the synaptic basis of learning and memory in vertebrates. The best understood form of long-term potentiation is induced by the activation of the N-methyl-D-aspartate receptor complex. This subtype of glutamate receptor endows long-term potentiation with Hebbian characteristics, and allows electrical events at the postsynaptic membrane to be transduced into chemical signals which, in turn, are thought to activate both pre- and postsynaptic mechanisms to generate a persistent increase in synaptic strength.

A synaptic model of memory: long-term potentiation in the hippocampus

Schultz, Wolfram. Predictive reward signal of dopamine neurons. J. Neurophysiol. 80: 1–27, 1998. The effects of lesions, receptor blocking, electrical self-stimulation, and drugs of abuse suggest t...

/pdf/predictive-reward-signal-of-dopamine-neurons-3f94myirza.pdf

Predictive Reward Signal of Dopamine Neurons

The memory consolidation hypothesis proposed 100 years ago by Muller and Pilzecker continues to guide memory research. The hypothesis that new memories consolidate slowly over time has stimulated studies revealing the hormonal and neural influences regulating memory consolidation, as well as molecular and cellular mechanisms. This review examines the progress made over the century in understanding the time-dependent processes that create our lasting memories.

http://science.sciencemag.org/content/sci/287/5451/248.full.pdf

Memory--a century of consolidation.

LTP and LTD: an embarrassment of riches.

Repeated stimulation of hippocampal neurons can induce an immediate and prolonged increase in synaptic strength that is called long-term potentiation (LTP)—the primary cellular model of memory in the mammalian brain1. An early phase of LTP (lasting less than three hours) can be dissociated from late-phase LTP by using inhibitors of transcription and translation2–8. Because protein synthesis occurs mainly in the cell body9–12, whereas LTP is input-specific, the question arises of how the synapse specificity of late LTP is achieved without elaborate intracellular protein trafficking. We propose that LTP initiates the creation of a short-lasting protein-synthesis-independent 'synaptic tag' at the potentiated synapse which sequesters the relevant protein(s) to establish late LTP. In support of this idea, we now show that weak tetanic stimulation, which ordinarily leads only to early LTP, or repeated tetanization in the presence of protein-synthesis inhibitors, each results in protein-synthesis-dependent late LTP, provided repeated tetanization has already been applied at another input to the same population of neurons. The synaptic tag decays in less than three hours. These findings indicate that the persistence of LTP depends not only on local events during its induction, but also on the prior activity of the neuron.

Synaptic tagging and long-term potentiation

Hippocampal long-term potentiation (LTP) is thought to serve as an elementary mechanism for the establishment of certain forms of explicit memory in the mammalian brain. As is the case with behavioral memory, LTP in the CA1 region has stages: a short-term early potentiation lasting 1 to 3 hours, which is independent of protein synthesis, precedes a later, longer lasting stage (L-LTP), which requires protein synthesis. Inhibitors of cyclic adenosine monophosphate (cAMP)-dependent protein kinase (PKA) blocked L-LTP, and analogs of cAMP induced a potentiation that blocked naturally induced L-LTP. The action of the cAMP analog was blocked by inhibitors of protein synthesis. Thus, activation of PKA may be a component of the mechanism that generates L-LTP.

https://science.sciencemag.org/content/sci/260/5114/1661.full.pdf

Effects of cAMP simulate a late stage of LTP in hippocampal CA1 neurons

Anisomycin, an inhibitor of protein synthesis, blocks late phases of LTP phenomena in the hippocampal CA1 region in vitro.

Trans-synaptic activation of gene expression is linked to long-term plastic adaptations in the nervous system. To examine the molecular program induced by synaptic activity, we have employed molecular cloning techniques to identify an immediate early gene that is rapidly induced in the brain. We here report the entire nucleotide sequence of the cDNA, which encodes an open reading frame of 396 amino acids. Within the hippocampus, constitutive expression was low. Basal levels of expression in the cortex were high but can be markedly reduced by blockade of N-methyl-D-aspartate receptors. By contrast, synaptic activity induced by convulsive seizures increased mRNA levels in neurons of the cortex and hippocampus. High-frequency stimulation of the perforant path resulted in long-term potentiation and a spatially confined dramatic increase in the level of mRNA in the granule cells of the ipsilateral dentate gyrus. Transcripts were localized to the soma and to the dendrites of the granule cells. The dendritic localization of the transcripts offers the potential for local synthesis of the protein at activated postsynaptic sites and may underlie synapse-specific modifications during long-term plastic events.

https://www.pnas.org/content/pnas/92/12/5734.full.pdf

Somatodendritic expression of an immediate early gene is regulated by synaptic activity

https://www.cell.com/trends/neurosciences/pdf/S0166-2236(97)01189-2.pdf

Uwe Frey

Papers

Synaptic tagging and long-term potentiation

Effects of cAMP simulate a late stage of LTP in hippocampal CA1 neurons

Anisomycin, an inhibitor of protein synthesis, blocks late phases of LTP phenomena in the hippocampal CA1 region in vitro.

Somatodendritic expression of an immediate early gene is regulated by synaptic activity

Synaptic tagging: implications for late maintenance of hippocampal long-term potentiation