The capacity to predict future events permits a creature to detect, model, and manipulate the causal structure of its interactions with its environment. Behavioral experiments suggest that learning is driven by changes in the expectations about future salient events such as rewards and punishments. Physiological work has recently complemented these studies by identifying dopaminergic neurons in the primate whose fluctuating output apparently signals changes or errors in the predictions of future salient and rewarding events. Taken together, these findings can be understood through quantitative theories of adaptive optimizing control.

/pdf/a-neural-substrate-of-prediction-and-reward-2tj1zqgoih.pdf

A Neural Substrate of Prediction and Reward

A free-energy principle has been proposed recently that accounts for action, perception and learning. This Review looks at some key brain theories in the biological (for example, neural Darwinism) and physical (for example, information theory and optimal control theory) sciences from the free-energy perspective. Crucially, one key theme runs through each of these theories — optimization. Furthermore, if we look closely at what is optimized, the same quantity keeps emerging, namely value (expected reward, expected utility) or its complement, surprise (prediction error, expected cost). This is the quantity that is optimized under the free-energy principle, which suggests that several global brain theories might be unified within a free-energy framework.

/pdf/the-free-energy-principle-a-unified-brain-theory-3vz1twg7nv.pdf

The free-energy principle: a unified brain theory?

The authors present a unified account of 2 neural systems concerned with the development and expression of adaptive behaviors: a mesencephalic dopamine system for reinforcement learning and a “generic” error-processing system associated with the anterior cingulate cortex The existence of the error-processing system has been inferred from the error-related negativity (ERN), a component of the event-related brain potential elicited when human participants commit errors in reaction-time tasks The authors propose that the ERN is generated when a negative reinforcement learning signal is conveyed to the anterior cingulate cortex via the mesencephalic dopamine system and that this signal is used by the anterior cingulate cortex to modify performance on the task at hand They provide support for this proposal using both computational modeling and psychophysiological experimentation

/pdf/the-neural-basis-of-human-error-processing-reinforcement-246y7vy4ka.pdf

The neural basis of human error processing: Reinforcement learning, dopamine, and the error-related negativity.

Several formal models of excitatory classical conditioning are reviewed. It is suggested that a central problem for all of them is the explanation of cases in which learning does not occur in spite of the fact that the conditioned stimulus is a signal for the reinforcer. We propose a new model that deals with this problem by specifying that certain procedures cause a conditioned stimulus (CS) to lose effectiveness; in particular, we argue that a CS will lose associability when its consequences are accurately predicted. In contrast to other current models, the effectiveness of the reinforcer remains constant throughout conditioning. The second part of the article presents a reformulation of the nature of the learning produced by inhibitory-conditioning procedures and a discussion of the way in which such learning can be accommodated within the model outlined for excitatory learning.

http://smash.psych.nyu.edu/courses/spring16/learnmem/papers/PearceHall1980.pdf

A model for Pavlovian learning: Variations in the effectiveness of conditioned but not of unconditioned stimuli.

An evolved module for fear elicitation and fear learning with 4 characteristics is proposed. (a) The fear module is preferentially activated in aversive contexts by stimuli that are fear relevant in an evolutionary perspective. (b) Its activation to such stimuli is automatic. (c) It is relatively impenetrable to cognitive control. (d) It originates in a dedicated neural circuitry, centered on the amygdala. Evidence supporting these propositions is reviewed from conditioning studies, both in humans and in monkeys; illusory correlation studies; studies using unreportable stimuli; and studies from animal neuroscience. The fear module is assumed to mediate an emotional level of fear learning that is relatively independent and dissociable from cognitive learning of stimulus relationships.

Fears, phobias and preparedness: Toward an evolved module of fear and fear learning

Classical conditioning II: Current research and theory.

Classical conditioning : a symposium

Comments on "Learned helplessness: Theory and evidence." by Maier and Seligman.

Studies on the operant conditioning of central nervous system activity have produced results interpreted as demonstrating that responses, certain properties of responses, or response-produced stimuli can function as discriminative stimuli. It is assumed that the feedback stimulus in biofeedback makes the subject aware of the internal response and that by becoming aware of the response, the subject can acquire voluntary control over it. In this context, awareness is operationally defined as the ability to use the response as a discriminative stimulus. Since direct evidence for the assumed relationship between control and discrimination is lacking, an attempt was made to test the hypothesis that discrimination of a response automatically leads to control over that response. The discriminative stimuli were the presence and absence of occipital alpha electroencephalograph (EEG) activity. Data from two experiments are reported. The first study, employing naive subjects, was designed to answer the following questions: (a) Since pilot data indicated that subjects seemed to match their responses to the more probable type of trial, would increases in the probability of a correct response result when the probabilities of alpha and nonalpha trials were held near .50? (b) If correct responding does increase, would performance of these subjects in an alpha feedback task be enhanced relative to that of subjects not previously given discrimination training? and (c) If subjects could not learn the discrimination task, would feedback training enhance their performance in a subsequent discrimination task? Results from this study indicate that holding the probabilities of alpha and nonalpha discrimination trials near .50 results in an absence of learning curves, but leaves open the possibility that sophisticated subjects are capable of discriminating alpha and nonalpha activity. The second study deals with two questions: (a) Can sophisticated subjects learn to discriminate occipital alpha activity from nonalpha activity? and (b) Does the procedure of providing subjects with salient stimuli, contingent on the presence and absence of alpha activity, establish stimulus control of the presence and absence of alpha activity? Results indicate that it is not possible to conclude that subjects can learn to discriminate alpha and nonalpha activity. However, learning to increase percent-time nonalpha or decrease percent-time alpha with respect to baseline levels by means of EEG-contingent stimulation provides subjects with the ability to suppress percent-time alpha in the absence of feedback. Information gained in both studies through subject interviews indicates that subjects most often acquired their control of alpha activity during feedback by a specific strategy and then used the strategy during the stimulus-control tests.

Abraham H. Black

Papers

Classical conditioning II: Current research and theory.

Classical conditioning : a symposium

Comments on "Learned helplessness: Theory and evidence." by Maier and Seligman.

Operant conditioning and discrimination of alpha: some methodological limitations inherent in response-discrimination experiments.