Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory.
TL;DR: The account presented here suggests that memories are first stored via synaptic changes in the hippocampal system, that these changes support reinstatement of recent memories in the neocortex, that neocortical synapses change a little on each reinstatement, and that remote memory is based on accumulated neocorticals changes.
Abstract: Damage to the hippocampal system disrupts recent memory but leaves remote memory intact. The account presented here suggests that memories are first stored via synaptic changes in the hippocampal system, that these changes support reinstatement of recent memories in the neocortex, that neocortical synapses change a little on each reinstatement, and that remote memory is based on accumulated neocortical changes. Models that learn via changes to connections help explain this organization. These models discover the structure in ensembles of items if learning of each item is gradual and interleaved with learning about other items. This suggests that the neocortex learns slowly to discover the structure in ensembles of experiences. The hippocampal system permits rapid learning of new items without disrupting this structure, and reinstatement of new memories interleaves them with others to integrate them into structured neocortical memory systems.
Citations
More filters
TL;DR: This work bridges the divide between high-dimensional sensory inputs and actions, resulting in the first artificial agent that is capable of learning to excel at a diverse array of challenging tasks.
Abstract: The theory of reinforcement learning provides a normative account, deeply rooted in psychological and neuroscientific perspectives on animal behaviour, of how agents may optimize their control of an environment. To use reinforcement learning successfully in situations approaching real-world complexity, however, agents are confronted with a difficult task: they must derive efficient representations of the environment from high-dimensional sensory inputs, and use these to generalize past experience to new situations. Remarkably, humans and other animals seem to solve this problem through a harmonious combination of reinforcement learning and hierarchical sensory processing systems, the former evidenced by a wealth of neural data revealing notable parallels between the phasic signals emitted by dopaminergic neurons and temporal difference reinforcement learning algorithms. While reinforcement learning agents have achieved some successes in a variety of domains, their applicability has previously been limited to domains in which useful features can be handcrafted, or to domains with fully observed, low-dimensional state spaces. Here we use recent advances in training deep neural networks to develop a novel artificial agent, termed a deep Q-network, that can learn successful policies directly from high-dimensional sensory inputs using end-to-end reinforcement learning. We tested this agent on the challenging domain of classic Atari 2600 games. We demonstrate that the deep Q-network agent, receiving only the pixels and the game score as inputs, was able to surpass the performance of all previous algorithms and achieve a level comparable to that of a professional human games tester across a set of 49 games, using the same algorithm, network architecture and hyperparameters. This work bridges the divide between high-dimensional sensory inputs and actions, resulting in the first artificial agent that is capable of learning to excel at a diverse array of challenging tasks.
23,074 citations
TL;DR: It is proposed that cognitive control stems from the active maintenance of patterns of activity in the prefrontal cortex that represent goals and the means to achieve them, which provide bias signals to other brain structures whose net effect is to guide the flow of activity along neural pathways that establish the proper mappings between inputs, internal states, and outputs needed to perform a given task.
Abstract: ▪ Abstract The prefrontal cortex has long been suspected to play an important role in cognitive control, in the ability to orchestrate thought and action in accordance with internal goals. Its neural basis, however, has remained a mystery. Here, we propose that cognitive control stems from the active maintenance of patterns of activity in the prefrontal cortex that represent goals and the means to achieve them. They provide bias signals to other brain structures whose net effect is to guide the flow of activity along neural pathways that establish the proper mappings between inputs, internal states, and outputs needed to perform a given task. We review neurophysiological, neurobiological, neuroimaging, and computational studies that support this theory and discuss its implications as well as further issues to be addressed
10,943 citations
TL;DR: This review examines the progress made over the century in understanding the time-dependent processes that create the authors' lasting memories.
Abstract: The memory consolidation hypothesis proposed 100 years ago by Muller and Pilzecker continues to guide memory research. The hypothesis that new memories consolidate slowly over time has stimulated studies revealing the hormonal and neural influences regulating memory consolidation, as well as molecular and cellular mechanisms. This review examines the progress made over the century in understanding the time-dependent processes that create our lasting memories.
3,902 citations
TL;DR: This article reviews a diverse set of proposals for dual processing in higher cognition within largely disconnected literatures in cognitive and social psychology and suggests that while some dual-process theories are concerned with parallel competing processes involving explicit and implicit knowledge systems, others are concerns with the influence of preconscious processes that contextualize and shape deliberative reasoning and decision-making.
Abstract: This article reviews a diverse set of proposals for dual processing in higher cognition within largely disconnected literatures in cognitive and social psychology. All these theories have in common the distinction between cognitive processes that are fast, automatic, and unconscious and those that are slow, deliberative, and conscious. A number of authors have recently suggested that there may be two architecturally (and evolutionarily) distinct cognitive systems underlying these dual-process accounts. However, it emerges that (a) there are multiple kinds of implicit processes described by different theorists and (b) not all of the proposed attributes of the two kinds of processing can be sensibly mapped on to two systems as currently conceived. It is suggested that while some dual-process theories are concerned with parallel competing processes involving explicit and implicit knowledge systems, others are concerned with the influence of preconscious processes that contextualize and shape deliberative reasoning and decision-making.
3,859 citations
Cites background from "Why there are complementary learnin..."
...…to link dual-process accounts in social cognition with those in cognitive psychology was made by Smith and DeCoster (2000) who build on the distinction of two kinds of memory, one based on slow acquisition through associative learning and one linked to explicit memory (McClelland et al 1995)....
[...]
...A recent attempt to link dual-process accounts in social cognition with those in cognitive psychology was made by Smith & DeCoster (2000), who build on the distinction of two kinds of memory, one based on slow acquisition through associative learning and one linked to explicit memory (McClelland et al. 1995)....
[...]
TL;DR: Episodic memory is a neurocognitive (brain/mind) system, uniquely different from other memory systems, that enables human beings to remember past experiences as discussed by the authors, which is a true, even if as yet generally unappreciated, marvel of nature.
Abstract: ▪ Abstract Episodic memory is a neurocognitive (brain/mind) system, uniquely different from other memory systems, that enables human beings to remember past experiences. The notion of episodic memory was first proposed some 30 years ago. At that time it was defined in terms of materials and tasks. It was subsequently refined and elaborated in terms of ideas such as self, subjective time, and autonoetic consciousness. This chapter provides a brief history of the concept of episodic memory, describes how it has changed (indeed greatly changed) since its inception, considers criticisms of it, and then discusses supporting evidence provided by (a) neuropsychological studies of patterns of memory impairment caused by brain damage, and (b) functional neuroimaging studies of patterns of brain activity of normal subjects engaged in various memory tasks. I also suggest that episodic memory is a true, even if as yet generally unappreciated, marvel of nature.
3,618 citations
References
More filters
TL;DR: A model of a system having a large number of simple equivalent components, based on aspects of neurobiology but readily adapted to integrated circuits, produces a content-addressable memory which correctly yields an entire memory from any subpart of sufficient size.
Abstract: Computational properties of use of biological organisms or to the construction of computers can emerge as collective properties of systems having a large number of simple equivalent components (or neurons). The physical meaning of content-addressable memory is described by an appropriate phase space flow of the state of a system. A model of such a system is given, based on aspects of neurobiology but readily adapted to integrated circuits. The collective properties of this model produce a content-addressable memory which correctly yields an entire memory from any subpart of sufficient size. The algorithm for the time evolution of the state of the system is based on asynchronous parallel processing. Additional emergent collective properties include some capacity for generalization, familiarity recognition, categorization, error correction, and time sequence retention. The collective properties are only weakly sensitive to details of the modeling or the failure of individual devices.
16,652 citations
17 Jul 1986
15,313 citations
Book•
03 Jan 1986
TL;DR: In this paper, the problem of the generalized delta rule is discussed and the Generalized Delta Rule is applied to the simulation results of simulation results in terms of the generalized delta rule.
Abstract: This chapter contains sections titled: The Problem, The Generalized Delta Rule, Simulation Results, Some Further Generalizations, Conclusion
13,579 citations