The Basal Ganglia System as an Engine for Exploration
01 Jan 2018-pp 59-96
TL;DR: This chapter argues that describing the two BG pathways as having mutually opponent actions has limitations and argues that the BG indirect pathway also plays a role in exploration, which is used to simulate various processes of the basal ganglia.
Abstract: One of the earliest attempts at building a theory of the basal ganglia (BG) is based on the clinical findings that lesions to the direct and indirect pathways of the BG produce quite opposite motor manifestations (Albin et al., in Trends Neurosci 12(10):366–375, 1989). While lesions of the direct pathway (DP), affecting particularly the projections from the striatum to GPi, are associated with hypokinetic disorders (distinguished by a paucity of movement), lesions of the indirect pathway (IP) produce hyperkinetic disorders, such as chorea and tremor. In this chapter, we argue that describing the two BG pathways as having mutually opponent actions has limitations. We argue that the BG indirect pathway also plays a role in exploration. We should evidence from various motor learning and decision-making tasks that exploration is a necessary process in various behavioral processes. Importantly, we use the exploration mechanism explained here to simulate various processes of the basal ganglia which we discuss in the following chapters.
Citations
More filters
TL;DR: The role of the GPe is highlighted as a major control hub of the basal ganglia, and the so-called 'prototypical' GPe neurons are shown to be the principal subpopulation influencing action selection.
36 citations
TL;DR: The model predicts that optimizing 5HT levels along with DA medications might be essential for improving the patients' reward-punishment learning deficits, and describes how DA and 5HT mediate activity in these classes of neurons (D1R, D2R-, D1R-D2R- MSNs).
Abstract: There is significant evidence that in addition to reward-punishment based decision making, the Basal Ganglia (BG) contributes to risk-based decision making as well Despite this evidence, little is known about the computational principles and neural correlates of risk computation in this subcortical system We have previously proposed a reinforcement learning based model of the BG that simulates the interactions between dopamine (DA) and serotonin (5HT) in a diverse set of experimental effects including reward, punishment and risk based decision making Starting with the idea that the activity of DA represents reward prediction error, the model posits that serotoninergic activity in the striatum controls risk-prediction error Our prior model of the BG was an abstract model that did not incorporate anatomical and cellular-level data In this work, we expand the earlier model into a detailed network model of the BG and demonstrate the joint contributions of DA-5HT in risk and reward-punishment sensitivity At the core of the proposed network model is the following insight regarding cellular correlates of value and risk computation Just as DA D1 receptor (D1R) expressing medium spiny neurons (MSNs) of the striatum were thought to be neural substrates for value computation, we propose that DA D1R and D2R co-expressing MSNs, reported to occupy a significant proportion of the striatum and are implicated in disorders like schizophrenia and drug addiction, are capable of computing risk Ours is the first-of-its-kind model that accounts for the significant computational possibilities of these co-expressing D1R-D2R MSNs, and describes how DA-5HT mediated activity in these classes of neurons (D1R-, D2R-, D1R-D2R- MSNs) contribute to the BG dynamics We also apply the model to capture the behaviour of PD patients in a probabilistic learning paradigm The study observes that optimizing 5HT levels along with DA medication could be essential to improving the patients' learning
33 citations
TL;DR: Computational modeling has the potential to become an invaluable tool to predict the onset of behavioral changes during disease progression and a significant decrease in sensitivity to punishment and risk was crucial for explaining behavioral changes observed in PD-ON ICD patients.
Abstract: Impulsivity, i.e. irresistibility in the execution of actions, may be prominent in Parkinson's disease (PD) patients who are treated with dopamine precursors or dopamine receptor agonists. In this study, we combine clinical investigations with computational modeling to explore whether impulsivity in PD patients on medication may arise as a result of abnormalities in risk, reward and punishment learning. In order to empirically assess learning outcomes involving risk, reward and punishment, four subject groups were examined: healthy controls, ON medication PD patients with impulse control disorder (PD-ON ICD) or without ICD (PD-ON non-ICD), and OFF medication PD patients (PD-OFF). A neural network model of the Basal Ganglia (BG) that has the capacity to predict the dysfunction of both the dopaminergic (DA) and the serotonergic (5HT) neuromodulator systems was developed and used to facilitate the interpretation of experimental results. In the model, the BG action selection dynamics were mimicked using a utility function based decision making framework, with DA controlling reward prediction and 5HT controlling punishment and risk predictions. The striatal model included three pools of Medium Spiny Neurons (MSNs), with D1 receptor (R) alone, D2R alone and co-expressing D1R-D2R. Empirical studies showed that reward optimality was increased in PD-ON ICD patients while punishment optimality was increased in PD-OFF patients. Empirical studies also revealed that PD-ON ICD subjects had lower reaction times (RT) compared to that of the PD-ON non-ICD patients. Computational modeling suggested that PD-OFF patients have higher punishment sensitivity, while healthy controls showed comparatively higher risk sensitivity. A significant decrease in sensitivity to punishment and risk was crucial for explaining behavioral changes observed in PD-ON ICD patients. Our results highlight the power of computational modelling for identifying neuronal circuitry implicated in learning, and its impairment in PD. The results presented here not only show that computational modelling can be used as a valuable tool for understanding and interpreting clinical data, but they also show that computational modeling has the potential to become an invaluable tool to predict the onset of behavioral changes during disease progression.
21 citations
TL;DR: Activity in the right dorsolateral PFC specifically increased when participants encountered tempting false shortcuts that led along suboptimal paths that needed to be differentiated from real shortcuts, providing insight to the temporal evolution of response to encountering detours and shortcuts.
Abstract: Central to the concept of the "cognitive map" is that it confers behavioral flexibility, allowing animals to take efficient detours, exploit shortcuts, and avoid alluring, but unhelpful, paths. The neural underpinnings of such naturalistic and flexible behavior remain unclear. In two neuroimaging experiments, we tested human participants on their ability to navigate to a set of goal locations in a virtual desert island riven by lava, which occasionally spread to block selected paths (necessitating detours) or receded to open new paths (affording real shortcuts or false shortcuts to be avoided). Detours activated a network of frontal regions compared with shortcuts. Activity in the right dorsolateral PFC specifically increased when participants encountered tempting false shortcuts that led along suboptimal paths that needed to be differentiated from real shortcuts. We also report modulation in event-related fields and theta power in these situations, providing insight to the temporal evolution of response to encountering detours and shortcuts. These results help inform current models as to how the brain supports navigation and planning in dynamic environments.
19 citations
TL;DR: A computational model of the cognitive and motor cortico-basal ganglia loops that explains the effects of sensory and cognitive processes on FOG is presented, providing a plausible framework for understanding the influence of cognition on automatic motor actions in controls and Parkinson's Disease.
Abstract: Experimental data show that perceptual cues can either exacerbate or ameliorate freezing of gait (FOG) in Parkinson’s Disease (PD). For example, simple visual stimuli like stripes on the floor can alleviate freezing whereas complex stimuli like narrow doorways can trigger it. We present a computational model of the cognitive and motor cortico-basal ganglia loops that explains the effects of sensory and cognitive processes on FOG. The model simulates strong causative factors of FOG including decision conflict (a disagreement of various sensory stimuli in their association with a response) and cognitive load (complexity of coupling a stimulus with downstream mechanisms that control gait execution). Specifically, the model simulates gait of PD patients (freezers and non-freezers) as they navigate a series of doorways while simultaneously responding to several Stroop word cues in a virtual reality setup. The model is based on an actor-critic architecture of Reinforcement Learning involving Utility-based decision making, where Utility is a weighted sum of Value and Risk functions. The model accounts for the following experimental data: (a) the increased foot-step latency seen in relation to high conflict cues, (b) the high number of motor arrests seen in PD freezers when faced with a complex cue compared to the simple cue, and (c) the effect of dopamine medication on these motor arrests. The freezing behavior arises as a result of addition of task parameters (doorways and cues) and not due to inherent differences in the subject group. The model predicts a differential role of risk sensitivity in PD freezers and non-freezers in the cognitive and motor loops. Additionally this first-of-its-kind model provides a plausible framework for understanding the influence of cognition on automatic motor actions in controls and Parkinson’s Disease.
17 citations
References
More filters
TL;DR: There is a deep and useful connection between statistical mechanics and multivariate or combinatorial optimization (finding the minimum of a given function depending on many parameters), and a detailed analogy with annealing in solids provides a framework for optimization of very large and complex systems.
Abstract: There is a deep and useful connection between statistical mechanics (the behavior of systems with many degrees of freedom in thermal equilibrium at a finite temperature) and multivariate or combinatorial optimization (finding the minimum of a given function depending on many parameters). A detailed analogy with annealing in solids provides a framework for optimization of the properties of very large and complex systems. This connection to statistical mechanics exposes new information and provides an unfamiliar perspective on traditional optimization problems and methods.
41,772 citations
Book•
01 Jan 1988TL;DR: This book provides a clear and simple account of the key ideas and algorithms of reinforcement learning, which ranges from the history of the field's intellectual foundations to the most recent developments and applications.
Abstract: Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning.
37,989 citations
TL;DR: Findings in this work indicate that dopaminergic neurons in the primate whose fluctuating output apparently signals changes or errors in the predictions of future salient and rewarding events can be understood through quantitative theories of adaptive optimizing control.
Abstract: The capacity to predict future events permits a creature to detect, model, and manipulate the causal structure of its interactions with its environment. Behavioral experiments suggest that learning is driven by changes in the expectations about future salient events such as rewards and punishments. Physiological work has recently complemented these studies by identifying dopaminergic neurons in the primate whose fluctuating output apparently signals changes or errors in the predictions of future salient and rewarding events. Taken together, these findings can be understood through quantitative theories of adaptive optimizing control.
8,163 citations
TL;DR: A model in which specific types of basal ganglia disorders are associated with changes in the function of subpopulations of striatal projection neurons is proposed, which suggests that the activity of sub Populations of Striatal projections neurons is differentially regulated by striatal afferents and that different striatal projections may mediate different aspects of motor control.
5,094 citations
TL;DR: Recent evidence indicating that a parallel functional architecture may also be characteristic of the organization within each individual circuit is discussed, which represents a significant departure from earlier concepts of basal ganglia organization.
4,011 citations