Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning.

/pdf/reinforcement-learning-an-introduction-rzxgej9p17.pdf

Reinforcement Learning: An Introduction

Objective
To present the International Parkinson and Movement Disorder Society (MDS) Clinical Diagnostic Criteria for Parkinson9s disease.
Background
Although several diagnostic criteria for Parkinson9s disease have been proposed, none have been officially adopted by an official Parkinson society. Moreover, the commonest-used criteria, the UK brain bank, were created more than 25 years ago. In recognition of the lack of standard criteria, the MDS initiated a task force to design new diagnostic criteria for clinical Parkinson9s disease.
Methods/Results
The MDS-PD Criteria are intended for use in clinical research, but may also be used to guide clinical diagnosis. The benchmark is expert clinical diagnosis; the criteria aim to systematize the diagnostic process, to make it reproducible across centers and applicable by clinicians with less expertise. Although motor abnormalities remain central, there is increasing recognition of non-motor manifestations; these are incorporated into both the current criteria and particularly into separate criteria for prodromal PD. Similar to previous criteria, the MDS-PD Criteria retain motor parkinsonism as the core disease feature, defined as bradykinesia plus rest tremor and/or rigidity. Explicit instructions for defining these cardinal features are included. After documentation of parkinsonism, determination of PD as the cause of parkinsonism relies upon three categories of diagnostic features; absolute exclusion criteria (which rule out PD), red flags (which must be counterbalanced by additional supportive criteria to allow diagnosis of PD), and supportive criteria (positive features that increase confidence of PD diagnosis). Two levels of certainty are delineated: Clinically-established PD (maximizing specificity at the expense of reduced sensitivity), and Probable PD (which balances sensitivity and specificity).
Conclusion
The MDS criteria retain elements proven valuable in previous criteria and omit aspects that are no longer justified, thereby encapsulating diagnosis according to current knowledge. As understanding of PD expands, criteria will need continuous revision to accommodate these advances. Disclosure: Dr. Postuma has received personal compensation for activities with Roche Diagnostics Corporation and Biotie Therapies. Dr. Berg has received research support from Michael J. Fox Foundation, the Bundesministerium fur Bildung und Forschung (BMBF), the German Parkinson Association and Novartis GmbH.

MDS Clinical Diagnostic Criteria for Parkinson's Disease (S19.001)

The basal ganglia are a chain of subcortical nuclei that facilitate action selection. Two striatal projection systems—so-called direct and indirect pathways—form the functional backbone of the basal ganglia circuit. Twenty years ago, investigators proposed that the striatum's ability to use dopamine (DA) rise and fall to control action selection was due to the segregation of D1 and D2 DA receptors in direct- and indirect-pathway spiny projection neurons. Although this hypothesis sparked a debate, the evidence that has accumulated since then clearly supports this model. Recent advances in the means of marking neural circuits with optical or molecular reporters have revealed a clear-cut dichotomy between these two cell types at the molecular, anatomical, and physiological levels. The contrast provided by these studies has provided new insights into how the striatum responds to fluctuations in DA signaling and how diseases that alter this signaling change striatal function.

/pdf/modulation-of-striatal-projection-systems-by-dopamine-4w32c0e7u5.pdf

Modulation of Striatal Projection Systems by Dopamine

Dopamine (DA) is important to a wide range of striatal functions, including associative learning and habit formation. At synapses between cortical pyramidal neurons and principal striatal medium spiny neurons (MSNs), postsynaptic D1 and D2 DA receptors are postulated to be necessary for the induction of long-term potentiation and depression, respectively. Because these receptors are restricted to two distinct MSN populations, this postulate demands that synaptic plasticity be unidirectional in each cell type. Using brain slices from DA receptor transgenic mice, we show that this is not the case. Rather, DA plays complementary roles in these two types of MSN to ensure that synaptic plasticity is bidirectional and Hebbian. In models of Parkinson's disease, this system is thrown out of balance, leading to unidirectional changes in plasticity that could underlie network pathology and symptoms.

/pdf/dichotomous-dopaminergic-control-of-striatal-synaptic-3jj1m9b0nq.pdf

Dichotomous Dopaminergic Control of Striatal Synaptic Plasticity

D1 and D2 dopamine-receptor modulation of striatal glutamatergic signaling in striatal medium spiny neurons

Dopamine-mediated regulation of corticostriatal synaptic plasticity

The basal ganglia are subcortical nuclei controlling voluntary actions and have been implicated in Parkinson's disease (PD). The prevailing model of basal ganglia function states that two circuits, the direct and indirect pathways, originate from distinct populations of striatal medium spiny neurons (MSNs) and project to different output structures. These circuits are believed to have opposite effects on movement. Specifically, the activity of direct pathway MSNs is postulated to promote movement, whereas the activation of indirect pathway MSNs is hypothesized to inhibit it. Recent findings have revealed that this model might not fully account for the concurrent activation of both pathways during movement. Accordingly, we propose a model in which intrastriatal connections are critical and the two pathways are structurally and functionally intertwined. Thus, all MSNs might either facilitate or inhibit movement depending on the form of synaptic plasticity expressed at a certain moment. In PD, alterations of dopamine-dependent synaptic plasticity could alter this coordinated activity.

Direct and indirect pathways of basal ganglia: a critical reappraisal

In the management of neurological diseases, the identification and quantification of axonal damage could allow for the improvement of diagnostic accuracy and prognostic assessment. Neurofilament light chain (NfL) is a neuronal cytoplasmic protein highly expressed in large calibre myelinated axons. Its levels increase in cerebrospinal fluid (CSF) and blood proportionally to the degree of axonal damage in a variety of neurological disorders, including inflammatory, neurodegenerative, traumatic and cerebrovascular diseases. New immunoassays able to detect biomarkers at ultralow levels have allowed for the measurement of NfL in blood, thus making it possible to easily and repeatedly measure NfL for monitoring diseases’ courses. Evidence that both CSF and blood NfL may serve as diagnostic, prognostic and monitoring biomarkers in neurological diseases is progressively increasing, and NfL is one of the most promising biomarkers to be used in clinical and research setting in the next future. Here we review the most important results on CSF and blood NfL and we discuss its potential applications and future directions.

/pdf/neurofilament-light-chain-as-a-biomarker-in-neurological-2458x917za.pdf

Neurofilament light chain as a biomarker in neurological disorders

Summary Levodopa is the most effective drug for the treatment of Parkinson's disease However, the long-term use of this dopamine precursor is complicated by highly disabling fluctuations and dyskinesias Although preclinical and clinical findings suggest pulsatile stimulation of striatal postsynaptic receptors as a key mechanism underlying levodopa-induced dyskinesias, their pathogenesis is still unclear In recent years, evidence from animal models of Parkinson's disease has provided important information to understand the effect of specific receptor and post-receptor molecular mechanisms underlying the development of dyskinetic movements Recent preclinical and clinical data from promising lines of research focus on the differential role of presynaptic versus postsynaptic mechanisms, dopamine receptor subtypes, ionotropic and metabotropic glutamate receptors, and non-dopaminergic neurotransmitter systems in the pathophysiology of levodopa-induced dyskinesias

https://www.thelancet.com/pdfs/journals/laneur/PIIS1474-4422(10)70218-0.pdf

Levodopa-induced dyskinesias in patients with Parkinson's disease: filling the bench-to-bedside gap

Summary Parkinson's disease is classically characterised as a motor neurodegenerative disorder. Motor symptoms in the disorder are secondary to an altered dopamine–acetylcholine balance due to reduced striatal dopaminergic tone and subsequent cholinergic overactivity. In the past, anticholinergic drugs were given to improve motor aspects of the disease. There is now an increasing interest in the cognitive and non-motor symptoms of Parkinson's disease and in cholinesterase-inhibitor therapy for dementia associated with Parkinson's disease. In this Personal View, we reconsider the dopamine–acetylcholine balance theory and look at recent clinical findings and the possible cooperative role of dopamine and acetylcholine in the induction and maintenance of the long-lasting changes of striatal and cortical synaptic plasticity. We also discuss a convergent versus parallel model to explain cognitive dysfunctions in Parkinson's disease according to dopamine–acetylcholine dependent alterations in synaptic plasticity.

https://www.thelancet.com/pdfs/journals/laneur/PIIS1474-4422(06)70600-7.pdf

Massimiliano Di Filippo

Papers

Dopamine-mediated regulation of corticostriatal synaptic plasticity

Direct and indirect pathways of basal ganglia: a critical reappraisal

Neurofilament light chain as a biomarker in neurological disorders

Levodopa-induced dyskinesias in patients with Parkinson's disease: filling the bench-to-bedside gap

A convergent model for cognitive dysfunctions in Parkinson's disease: the critical dopamine–acetylcholine synaptic balance