Predictability, surprise, attention, and conditioning
01 Dec 1967-
TL;DR: The role of attention in Pavlovian conditioning, and use of auditory and visual stimuli to condition rats is discussed in this article, where the authors discuss the use of both visual and auditory stimuli.
Abstract: Role of attention in Pavlovian conditioning, and use of auditory and visual stimuli to condition rats
Citations
More filters
Book•
01 Jan 1988TL;DR: This book provides a clear and simple account of the key ideas and algorithms of reinforcement learning, which ranges from the history of the field's intellectual foundations to the most recent developments and applications.
Abstract: Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning.
37,989Â citations
TL;DR: Mechanisms that govern the processing of emotional information, particularly those involved in fear reduction, are proposed and applications to therapeutic practice and to the broader study of psychopathology are discussed.
Abstract: In this article we propose mechanisms that govern the processing of emotional information, particularly those involved in fear reduction. Emotions are viewed as represented by information structures in memory, and anxiety is thought to occur when an information structure that serves as program to escape or avoid danger is activated. Emotional processing is denned as the modification of memory structures that underlie emotions. It is argued that some form of exposure to feared situations is common to many psychotherapies for anxiety, and that confrontation with feared objects or situations is an effective treatment. Physiological activation and habituation within and across exposure sessions are cited as indicators of emotional processing, and variables that influence activation and habituation of fear responses are examined. These variables and the indicators are analyzed to yield an account of what information must be integrated for emotional processing of a fear structure. The elements of such a structure are viewed as cognitive representations of the stimulus characteristic of the fear situation, the individual's responses in it, and aspects of its meaning for the individual. Treatment failures are interpreted with respect to the interference of cognitive defenses, autonomic arousal, mood state, and erroneous ideation with reformation of targeted fear structures. Applications of the concepts advanced here to therapeutic practice and to the broader study of psychopathology are discussed.
4,667Â citations
Cites background from "Predictability, surprise, attention..."
...The argument that the "signal value" of a stimulus is an important predictor of conditioning (e.g., Grings, 1976; Kamin, 1969) has indicated the need for an "informational model" of learning (Reiss, 1980; see also Levey & Martin, 1983)....
[...]
TL;DR: A new model is proposed that deals with the explanation of cases in which learning does not occur in spite of the fact that the conditioned stimulus is a signal for the reinforcer by specifying that certain procedures cause a conditioned stimulus to lose effectiveness.
Abstract: Several formal models of excitatory classical conditioning are reviewed. It is suggested that a central problem for all of them is the explanation of cases in which learning does not occur in spite of the fact that the conditioned stimulus is a signal for the reinforcer. We propose a new model that deals with this problem by specifying that certain procedures cause a conditioned stimulus (CS) to lose effectiveness; in particular, we argue that a CS will lose associability when its consequences are accurately predicted. In contrast to other current models, the effectiveness of the reinforcer remains constant throughout conditioning. The second part of the article presents a reformulation of the nature of the learning produced by inhibitory-conditioning procedures and a discussion of the way in which such learning can be accommodated within the model outlined for excitatory learning.
2,779Â citations
Cites background from "Predictability, surprise, attention..."
...A more subtle example of the same basic phenomenon is supplied by the occurrence of "blocking" (Kamin, 1969)....
[...]
TL;DR: Overshadowing and blocking are better explained by the choice of an appropriate rule for changing a, such that a decreases to stimuli that signal no change from the probability of reinforcement predicted by other stimuli.
Abstract: According to theories of selective attention, learning about a stimulus depends on attending to that stimulus; this is represented in two-stage models by saying that subjects switch in analyzers as well as learning stimulusresponse associations This assumption, however, is equally well represented in a formal model by the incorporation of a stimulus-specific learning-rate parameter, a, into the equations describing changes in the associative strength of stimuli Theories of selective attention have also assumed (a) that subjects learn to attend to and ignore relevant and irrelevant stimuli (ie, that a may increase or decrease depending on the correlation of a stimulus with reinforcement) and (b) that there is an inverse relationship between the probabilities of attending to different stimuli (ie, that an increase in a to one stimulus is accompanied by a decrease in a to others) The first assumption is used to explain the phenomena of acquired distinctiveness and dimensional transfer, the second those of overshadowing and blocking Although the first assumption is justified by the data, the second is not: Overshadowing and blocking are better explained by the choice of an appropriate rule for changing a, such that a decreases to stimuli that signal no change from the probability of reinforcement predicted by other stimuli
2,604Â citations
TL;DR: The learned helplessness hypothesis is proposed, which argues that when events are uncontrollable the organism learns that its behavior and outcomes are independent, and that this learning produces the motivational, cognitive, and emotional effects of uncontrollabi lity.
Abstract: \, SUMMARY In 1967, Overmier and Seligman found that dogs exposed to inescapable and unavoidable electric shocks in one situation later failed to learn to escape shock in a different situation where escape was possible. Shortly thereafter Seligman and Maier (1967) demonstrated that this effect was caused by the uncontrollability of the original shocks. In this article we review the effects of exposing organisms to aversive events which they cannot control, and we review the explanations which have been offered. There seem to be motivational, cognitive, and emotional effects of uncontrollability. (a) Motivation. Dogs that have been exposed to inescapable shocks do not subsequently initiate escape response in the presence of shock. We review parallel phenomena in cats, fish, rats, and man. Of particular interest is the discussion of learned helplessness in rats and man. Rats are of interest because learned helplessness has been difficult to demonstrate in rats. However, we show that inescapably shocked rats do fail to learn to escape if the escape task is reasonably difficult. With regard to man, we review a variety of studies using inescapable noise and unsolvable problems as agents which produce learned helplessness effects on both instrumental and cognitive tasks, (b) Cognition. We argue that exposure to uncontrollabl e events interferes with the organism's tendency to perceive contingent relationships between its behavior and outcomes. Here we review a variety of studies showing such a cognitive set. (c) Emotion. We review a variety of experiments which show that uncontrollable aversive events produce greater emotional disruption than do controllable aversive events. We have proposed an explanation for these effects, which we call the learned helplessness hypothesis. It argues that when events are uncontrollable the organism learns that its behavior and outcomes are independent, and that this learning produces the motivational, cognitive, and emotional effects of uncontrollabi lity. We describe the learned helplessness hypothesis and research which supports it. Finally, we describe and discuss in detail alternative hypotheses which have been offered as accounts of the learned helplessness effect. One set of hypotheses argues that organisms learn motor responses during exposure to uncontrollabl e shock that compete with the response required in the test task. Another explanation holds that uncontrollable shock is a severe stressor and depletes a neurochemical necessary for the mediation of movement. We examine the logical structure of these explanations and present a variety of evidence which bears on them directly.
2,187Â citations