Showing papers by "James L. McClelland published in 1987"

Parallel Distributed Processing: Explorations in the Microstructures of Cognition

Abstract: 1. Very rarely, a book is published which not only advances our knowledge of a particular topic, but fundamentally recasts our methods of investigating and thinking about large tracts of the map of learning. Linguists remember 1957 as the publication year of Noam Chomsky's Syntactic structures-a book whose ostensible subjects were the structure of English grammatical rules and the goals of grammatical description, but which can be seen with hindsight as the first shot in an intellectual revolution which ended by radically changing the texture of day-to-day research activity and discourse throughout almost all of linguistics, and in substantial parts of other cognition-related disciplines. In decades to come, perhaps 1986 will be remembered by academics as the year of publication of the pair of volumes reviewed here: they constitute the first large-scale public statement of an intellectual paradigm fully as revolutionary as the generative paradigm ever was (there have been scattered journal articles in the preceding four or five years). I would go further and suggest that, if the promises of this book can be redeemed, the contrast in linguistics and neighboring disciplines between the 1990's and the 1970's will be significantly greater than the contrast between the 1970's and the 1950's. (I need hardly add, of course, that it is one thing to fire an opening salvo, but another to achieve ultimate predominance.) The new paradigm is called Parallel Distributed Processing by the sixteen writers who contributed to this book, many of whom work either at the University of California, San Diego, or at Carnegie-Mellon University in Pittsburgh. Some other researchers (e.g. Feldman 1985) use the term 'connectionism' for the same concept. These two volumes comprise 26 chapters which, among them, (i) explain the over-all nature and aims of PDP/connectionist models, (ii) define a family of specific variants of the general paradigm, and (iii) exemplify it by describing experiments in which PDP models were used to simulate human performance in various cognitive domains. The experiments, inevitably, treat their respective domains in a simplified, schematic way by comparison with the endless complexity found in any real-life cognitive area; but simplification in this case does not mean trivialization. There are also auxiliary chapters on relevant related topics; thus Chap. 9, by M. I. JORDAN, is a tutorial on linear algebra, a branch of mathematics having special significance for the PDP paradigm. (Each chapter is attributed to a particular author or

The Case for Interactionism in Language Processing.

Abstract: : Interactive models of language processing assume that information flows both bottom-up and top-down, so that the representations formed at each level may be influenced by higher as well as lower levels. I describe a framework called the interactive activation framework that embeds this key assumption among others, including the assumption that influences from different sources are combined non-linearly. This non-linearity means that information that may be decisive under some circumstances have little or no effect under other conditions. Two attempts to rule out an interactive account in favor of models in which individual components of the language processing system act autonomously are considered in light of the interactive activation framework. In both cases, the facts are as expected from the principles of interactive activation. In general, existing facts do not rule out an interactive account, but they do not require one either. To demonstrate that more definitive tests of interaction are possible. I describe an experiment that demonstrates a new kind of influence of a higher level factor (lexical membership) a lower level of processing (phoneme identification). The experiment illustrates one reason why feedback from higher levels is computationally desirable; it allows lower levels to be tuned by contextual factors so that they can supply more accurate information to higher levels.

Mechanisms of Sentence Processing: Assigning Roles to Constituents of Sentences

Abstract: Abstract : How do we assign nouns correctly to their underlying case roles in English, and how do we select an appropriate verb frame to assign these nouns to? How do we know whether a noun phrase is a modifier of a preceding noun phrase or an argument of the verb? How do we select the correct meaning of each noun in the sentence, and how do we allow content to modulate its meaning? How do we know how to handle these new nouns and verbs? In this article we describe a simulation model that addresses these questions from a perspective quite different from the conventional perspective found in Computational Linguistics. Words are treated as patterns of activation, and knowledge about them is stored in distributed form, in the connections in a large network of simple neutron-like processing units. The model exhibits considerable facility in dealing with the problems of frame selection, role assignment, disambiguation, etc, and suggests a natural way to resolve unappealing aspects of the idea that there is a fixed set of individuated case roles. So far, our simulation model can only process one clause sentences. Possible extensions to multi-clause sentences are described.

Learning Representations by Recirculation

Abstract: We describe a new learning procedure for networks that contain groups of nonlinear units arranged in a closed loop. The aim of the learning is to discover codes that allow the activity vectors in a "visible" group to be represented by activity vectors in a "hidden" group. One way to test whether a code is an accurate representation is to try to reconstruct the visible vector from the hidden vector. The difference between the original and the reconstructed visible vectors is called the reconstruction error, and the learning procedure aims to minimize this error. The learning procedure has two passes. On the first pass, the original visible vector is passed around the loop, and on the second pass an average of the original vector and the reconstructed vector is passed around the loop. The learning procedure changes each weight by an amount proportional to the product of the "presynaptic" activity and the difference in the post-synaptic activity on the two passes. This procedure is much simpler to implement than methods like back-propagation. Simulations in simple networks show that it usually converges rapidly on a good set of codes, and analysis shows that in certain restricted cases it performs gradient descent in the squared reconstruction error.

Reflections on Cognition and Parallel Distributed Processing

Abstract: This chapter contains sections titled: What is New: The PDP Approach to the Study of Cognition, Toward a New Understanding of Human Information Processing, Acknowledgments

Information Processing in Dynamical Systems: Foundations of Harmony Theory

Abstract: This chapter contains sections titled: Section 1: Schema Theory and Self-Consistency, Section 2: Harmony Theory

Biologically Plausible Models of Place Recognition and Goal Location

Abstract: This chapter contains sections titled: Place Recognition, The Model, Location Parameters, Simulated Experiments, More about Landmarks, Place-Field Shape and Size, Place-Field Location, Scope of the Place-Field Model, Goal Location, The Distributed View-Field Model, The β-Coefficient Model, Conclusion, Appendix A, Appendix B

Parallel distributed processing and role assignment constraints

Abstract: I believe that parallel-distributed processing models (i.e., conneelionist models which make use of distributed representations) provide the mechanisms that are needed for these lasks. Argument altachments and role assignments seem to require a consideration of the relative merits of competing possibilities (Marcus, 1980; Bates and MacWhinney, 1987; MaeWhinney, 1987), as deles lexical dlsambigualion. Conuectionist models provide a very natural substrate for these kinds of competition processes (Cottrell, 1985; Wallz and Pollack, 1985).

Certain Aspects of the Anatomy and Physiology of the Cerebral Cortex

Abstract: This chapter contains sections titled: The Cerebral Cortex, The Nature of Neocortical Neurons, Acknowledgments

An Introduction to Linear Algebra in Parallel Distributed Processing

Abstract: This chapter contains sections titled: Vectors, Matrices and Linear Systems, Matrices, Nonlinear Systems

State-Dependent Factors Influencing Neural Plasticity: A Partial Account of the Critical Period

Abstract: This chapter contains sections titled: Plasticity and Learning, The Critical Period, The Ocularity State and Its Effect on Plasticity, Comparison with Experimental Data, Discussion, Beyond Visual Cortex, Summary, Acknowledgments

The PDP Perspective

Abstract: What makes people smarter than computers? These volumes by a pioneering neurocomputing group suggest that the answer lies in the massively parallel architecture of the human mind They describe a new theory of cognition called connectionism that is challenging the idea of symbolic computation that has traditionally been at the center of debate in theoretical discussions about the mind The authors' theory assumes the mind is composed of a great number of elementary units connected in a neural network Mental processes are interactions between these units which excite and inhibit each other in parallel rather than sequential operations In this context, knowledge can no longer be thought of as stored in localized structures; instead, it consists of the connections between pairs of units that are distributed throughout the network Volume 1 lays the foundations of this exciting theory of parallel distributed processing, while Volume 2 applies it to a number of specific issues in cognitive science and neuroscience, with chapters describing models of aspects of perception, memory, language, and thought

Three Short Papers on Language and Connectionism

Abstract: : This technical report contains three short articles on different aspects of language and connectionism. Together, the articles illustrate both the promise and the challenges facing the application of connectionist models to central issues in language processing. The first paper, (Reconstructive memory for sentences, by St. John and McClelland) describes a connectionist model in which background knowledge is used to aid recall and fill in missing arguments in sentences. The second, (Parallel distributed processing and role assignment constraints, by J. L. McClelland) discusses the application of connectionist models to the problem of using semantic/pragmatic constraints to processing sentences like 'John ate the cake that his mother baked in the oven' as opposed to 'John ate the cake that his mother baked in the dining room.' The third paper gives a brief overview of the model of past tense learning developed by Rumelhart and McClelland. Keywords: Cognitive psychology, Learning, Language, Connectionism, PDP, PP attachment, Case assignment, Verb tense.

The Logic of Activation Functions

Abstract: This chapter contains sections titled: Examples of Activation Rules, The Main Concepts, Illustration of These Concepts, Some Results, Conclusion, Acknowledgments