Dopamine in Motivational Control: Rewarding, Aversive, and Alerting

Motor control is the study of how organisms make accurate goal-directed movements. Here we consider two problems that the motor system must solve in order to achieve such control. The first problem is that sensory feedback is noisy and delayed, which can make movements inaccurate and unstable. The second problem is that the relationship between a motor command and the movement it produces is variable, as the body and the environment can both change. A solution is to build adaptive internal models of the body and the world. The predictions of these internal models, called forward models because they transform motor commands into sensory consequences, can be used to both produce a lifetime of calibrated movements, and to improve the ability of the sensory system to estimate the state of the body and the world around it. Forward models are only useful if they produce unbiased predictions. Evidence shows that forward models remain calibrated through motor adaptation: learning driven by sensory prediction errors.

/pdf/error-correction-sensory-prediction-and-adaptation-in-motor-23crjyjldy.pdf

Error Correction, Sensory Prediction, and Adaptation in Motor Control

Scientists in many different fields have been attracted to the study of habits because of the power habits have over behavior and because they invoke a dichotomy between the conscious, voluntary control over behavior, considered the essence of higher-order deliberative behavioral control, and lower-order behavioral control that is scarcely available to consciousness. A broad spectrum of behavioral routines and rituals can become habitual and stereotyped through learning. Others have a strong innate basis. Repetitive behaviors can also appear as cardinal symptoms in a broad range of neurological and neuropsychiatric illness and in addictive states. This review suggests that many of these behaviors could emerge as a result of experience-dependent plasticity in basal ganglia–based circuits that can influence not only overt behaviors but also cognitive activity. Culturally based rituals may reflect privileged interactions between the basal ganglia and cortically based circuits that influence social, emotional...

Habits, Rituals, and the Evaluative Brain

Theories of instrumental learning are centred on understanding how success and failure are used to improve future decisions. These theories highlight a central role for reward prediction errors in updating the values associated with available actions. In animals, substantial evidence indicates that the neurotransmitter dopamine might have a key function in this type of learning, through its ability to modulate cortico-striatal synaptic efficacy. However, no direct evidence links dopamine, striatal activity and behavioural choice in humans. Here we show that, during instrumental learning, the magnitude of reward prediction error expressed in the striatum is modulated by the administration of drugs enhancing (3,4-dihydroxy-L-phenylalanine; L-DOPA) or reducing (haloperidol) dopaminergic function. Accordingly, subjects treated with L-DOPA have a greater propensity to choose the most rewarding action relative to subjects treated with haloperidol. Furthermore, incorporating the magnitude of the prediction errors into a standard action-value learning algorithm accurately reproduced subjects' behavioural choices under the different drug conditions. We conclude that dopamine-dependent modulation of striatal activity can account for how the human brain uses reward prediction errors to improve future decisions.

/pdf/dopamine-dependent-prediction-errors-underpin-reward-seeking-3u1jotu7ad.pdf

Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans

The functions of rewards are based primarily on their effects on behavior and are less directly governed by the physics and chemistry of input events as in sensory systems. Therefore, the investigation of neural mechanisms underlying reward functions requires behavioral theories that can conceptualize the different effects of rewards on behavior. The scientific investigation of behavioral processes by animal learning theory and economic utility theory has produced a theoretical framework that can help to elucidate the neural correlates for reward functions in learning, goal-directed approach behavior, and decision making under uncertainty. Individual neurons can be studied in the reward systems of the brain, including dopamine neurons, orbitofrontal cortex, and striatum. The neural activity can be related to basic theoretical terms of reward and uncertainty, such as contiguity, contingency, prediction error, magnitude, probability, expected value, and variance.

/pdf/behavioral-theories-and-the-neurophysiology-of-reward-4veo3p2mub.pdf

Behavioral Theories and the Neurophysiology of Reward

https://www.cell.com/neuron/pdf/S0896-6273(03)00869-9.pdf

Dopamine Neurons Can Represent Context-Dependent Prediction Error

Reward is a primary goal of behavior and is crucial for survival of animals. To explore the mechanisms underlying such reward-oriented behavior, we devised a memory-guided saccade task in which only one fixed direction out of four was rewarded, which was called the one-direction-rewarded task (1DR). As the rewarded direction was changed in four blocks, saccades in a given direction were rewarded in one block (constituting reward-oriented behavior), but non-rewarded in the other blocks (non-reward-oriented behavior). As a control, an all-directions-rewarded task (ADR) was used. Using these tasks, we found that the parameters of saccades changed depending on whether or not the saccade was followed by reward. (1) The mean saccadic peak velocity was higher and the mean saccade latency was shorter in the rewarded condition than in the non-rewarded condition. (2) The mean saccade amplitude showed no difference in two out of three monkeys. (3) The variations of saccadic velocity, latency and amplitude were smaller in the rewarded condition. (4) Within a block of 1DR, the saccade velocity remained high in the rewarded condition, but decreased gradually in the non-rewarded condition; it decreased only slightly in ADR. The saccade latency showed the opposite pattern of change, but less clearly. (5) The saccades in the non-rewarded condition tended to have slower velocities and longer latencies in the trials shortly after a rewarded trial. (6) The ratio of error trials was much higher in the non-rewarded condition than the rewarded condition. (7) The errors, which were due to premature or incorrect saccades, showed unique spatiotemporal patterns that would reflect the competition between the cognitive and motivational processes. These results provide important constraints to the neuronal mechanism underlying reward-oriented behavior because it must satisfy these rules.

http://prism.bham.ac.uk/%7Ejackscpz/jc/TakKawIto2002.pdf

Modulation of saccadic eye movements by predicted reward outcome.

The effect of genetic polymorphisms for glutathione S-transferase ( GST) M1, GSTT1, GSTP1-1 ( GSTP1), cytochrome P450 2E1 ( CYP2E1) and aldehyde dehydrogenase 2 ( ALDH2) on the risk of hepatocellular carcinoma (HCC) was observed in 78 Japanese patients with HCC and 138 non-cancer hospital controls. We found a positive association between cumulative amounts of alcohol consumption (≧600,000 ml in a lifetime) and the risk of HCC (OR=4.52, 95% CI 2.39–8.55). However, cigarette smoking was not significantly related to the risk of HCC (OR=1.23, 95% CI 0.57–2.68). The allelic frequencies of GSTM1, GSTT1, GSTP1, CYP2E1and ALDH2of HCC patients were not significantly different from those of controls when odds ratios were only adjusted for age and gender except for any 2 alleles of ALDH2in drinkers (OR=2.53, 95% CI 1.21–5.31). However, the frequency of any C2alleles of CYP2E1 and any 2 alleles of ALDH2were significantly higher than those of controls (OR=5.77, 95% CI 1.24–27.39, OR=9.77, 95% CI 1.63–58.60) when covariates including viremia were selected by using stepwise logistic regression analysis. We conclude that habitual alcohol drinking is likely to lead to an increased risk of HCC, and any C2alleles of CYP2E1as well as any two alleles of ALDH2were also associated with an increased risk of HCC.

Genetic polymorphisms of tobacco- and alcohol-related metabolizing enzymes and the risk of hepatocellular carcinoma

The polymorphic arylamine N-acetyltransferases (NAT1 and NAT2) have been implicated in increased susceptibility to certain malignancies. We analyzed genetic polymorphisms in both the NAT1 and NAT2 genes among 140 gastric adenocarcinoma patients, 103 colorectal adenocarcinoma patients and 122 healthy controls from Japan. The frequency of the specific genotype NAT1*10 allele, which contains a variant polyadenylation signal, was higher among all gastric adenocarcinoma cases, but this increase did not reach statistical significance. After grouping according to tumor differentiation of gastric adenocarcinoma patients, NAT1 polymorphism was a risk factor among the well-differentiated type of tumors (OR = 3.03, 95% CI 1. 08-8.46). Stratifying by smoking status, we found that the OR for heavy smokers with the NAT1*10 allele was 2.97 (95% CI 1.23-7.14). When the combined risk of NAT1*10 allele from smoking and tumor differentiation was calculated, we found that the risk of the NAT1*10 allele with heavy smoking was increased among the well - differentiated type of gastric adenocarcinoma (OR = 4.24, 95% CI 0. 87-20.6). The NAT1*10 genotype was not a significant risk factor in colorectal adenocarcinoma. No statistically significant differences were observed in the frequency of NAT2 rapid acetylation genotype in gastric (91.4%) or colorectal (95.2%) adenocarcinoma patients when compared with the control population (94.3%). Our results suggest the NAT1*10 allele may be an important genetic determinant of the well-differentiated type of gastric adenocarcinoma, which may be induced by smoking.

Inherited polymorphism in the N-acetyltransferase 1 (NAT1) and 2 (NAT2) genes and susceptibility to gastric and colorectal adenocarcinoma.

Changes in the reward context are associated with changes in neuronal activity in the basal ganglia as well as changes in motor outputs. A typical example is found in the caudate (CD) projection neurons and saccade parameters. It raised the possibility that the changes in CD neuronal activity contribute to the changes in saccade parameters. To examine this possibility, we calculated the correlation coefficients (CORs) of the firing rates of each neuron with saccade parameters (peak saccade velocity and latency) on a trial-by-trial basis. We then calculated the mean CORs separately for two CD populations: reward-enhanced type neurons (RENs) that showed enhanced activity and reward-depressed type neurons (RDNs) that showed depressed activity when reward was expected. The activity of RENs was positively correlated with the saccadic peak velocity and negatively correlated with the saccade latency. The activity of RDNs was not significantly correlated with the saccade parameters. We further analyzed the CORs for RENs, a major type of CD neurons. First, we examined the time courses of the CORs using a moving time window (duration: 200 ms). The positive correlation with the saccade velocity and the negative correlation with the saccade latency were present not only in the peri-saccadic period but also during the pre- and postcue periods. Second, we asked whether the CORs with the saccade parameters were direction-selective. A majority of RENs were more active before contralateral saccades (contralateral-preferring neurons) and their activity was correlated more strongly with contralateral saccades than with ipsilateral saccades. A minority of RENs, ipsilateral-preferring neurons, showed no such preference. These results are consistent with the hypothesis that CD neuronal activity exerts facilitatory effects on contralateral saccades and that the effects start well before saccade execution. Furthermore, a multiple regression analysis indicated that changes in activity of some, but not all, CD neurons could be explained by changes in saccade parameters; a major determinant was reward context (presence or absence of reward). These results suggest that, while a majority of CD neurons receive reward-related signals, only some of them can make a significant contribution to change saccadic outputs based on expected reward.

/pdf/correlation-of-primate-caudate-neural-activity-and-saccade-1qjta8juir.pdf

Hideaki Itoh

Papers

Dopamine Neurons Can Represent Context-Dependent Prediction Error

Modulation of saccadic eye movements by predicted reward outcome.

Genetic polymorphisms of tobacco- and alcohol-related metabolizing enzymes and the risk of hepatocellular carcinoma

Inherited polymorphism in the N-acetyltransferase 1 (NAT1) and 2 (NAT2) genes and susceptibility to gastric and colorectal adenocarcinoma.

Correlation of primate caudate neural activity and saccade parameters in reward-oriented behavior.