Showing papers on "Recurrent neural network published in 1988"

A simple neural network generating an interactive memory

Abstract: A model of a neural system where a group of neurons projects to another group of neurons is discussed. We assume that a trace is the simultaneous pattern of individual activities shown by a group of neurons. We assume synaptic interactions add linearly and that synaptic weights (quantitative measure of degree of coupling between two cells) can be coded in a simple but optimal way where changes in synaptic weight are proportional to the product of pre-and postsynaptic activity at a given time. Then it is shown that this simple system is capable of “memory” in the sense that it can (1) recognize a previously presented trace and (2) if two traces have been associated in the past (that is, if trace f was impressed on the first group of neurons and trace ḡ was impressed on the second group of neurons and synaptic weights coupling the two groups changed according to the above rule) presentation of f to the first group of neurons gives rise to f plus a calculable amount of noise at the second set of neurons. This kind of memory is called an “interactive memory” since distinct stored traces interact in storage. It is shown that this model can effectively perform many functions. Quantitative expressions are derived for the average signal to noise ratio for recognition and one type of association. The selectivity of the system is discussed. References to physiological data are made where appropriate. A sketch of a model of mammalian cerebral cortex which generates an interactive memory is presented and briefly discussed. We identify a trace with the activity of groups of cortical pyramidal cells. Then it is argued that certain plausible assumptions about the properties of the synapses coupling groups of pyramidal cells lead to the generation of an interactive memory.

Neural net pruning-why and how

Abstract: A continuing question in neural net research is the size of network needed to solve a particular problem. If training is started with too small a network for the problem no learning can occur. The researcher must then go through a slow process of deciding that no learning is taking place, increasing the size of the network and training again. If a network that is larger than required is used, then processing is slowed, particularly on a conventional von Neumann computer. An approach to this problem is discussed that is based on learning with a net which is larger than the minimum size network required to solve the problem and then pruning the solution network. The result is a small, efficient network that performs as well or better than the original which does not give a complete answer to the question, since the size of the initial network is still largely based on guesswork but it gives a very useful partial answer and sheds some light on the workings of a neural network in the process. >

Asynchronous VLSI neural networks using pulse-stream arithmetic

Abstract: The relationship between neural networks and VLSI is explored. An introduction to neural networks relates the Hopfield model and the Delta learning rule to S. Grossberg's (1968) description of neural dynamics. A computational style that mimics that of a biological neural network, using pulse-stream signaling and analog summation, is described. Digitally programmable weights allow learning networks to be constructed. Functional and structural forms of neural and synaptic functions are presented, along with simulation results. Finally a neural network implemented in 3- mu m CMOS is presented with preliminary measurements. >

Introduction to neural networks for intelligent control

Abstract: Neural network architecture is presented as one approach to the design and implementation of intelligent control systems. Neural networks can be considered as massively parallel distributed processing systems with the potential for ever-improving performance through dynamical learning. The nomenclature and characteristics of neural networks are outlined. Two simple examples are presented to illustrate applications to control systems: one is fault isolation mapping, and the other involves optimization of a Hopfield network that defines a clockless analog-to-digital conversion. >

Efficient Parallel Learning Algorithms for Neural Networks

Abstract: Parallelizable optimization techniques are applied to the problem of learning in feedforward neural networks. In addition to having superior convergence properties, optimization techniques such as the Polak-Ribiere method are also significantly more efficient than the Backpropagation algorithm. These results are based on experiments performed on small boolean learning problems and the noisy real-valued learning problem of hand-written character recognition.

Potts-glass models of neural networks.

Abstract: The theory of neural networks is extended to include discrete neurons with more than two discrete states. The dynamics of such systems are studied. The maximum number of storage patterns is found to be proportional to Nq(q-1), where q is the number of Potts states and N is the size of the network. The properties of the Potts neural network are compared with the Ising case, and the similarity between the Potts neural network and a diluted multineuron interacting Hopfield model is discussed.

Neural networks. II. What are they and why is everybody so interested in them now

Abstract: For part I, see ibid., vol. EX-2, no. 4, p. 10-11 (1987). The learning ability of neural networks and their ability to generalize and to abstract or generate ideals from an imperfect training set are examined. Their potential for multiprocessing is considered. A brief history of neural network research is followed by a discussion of their architectures and a presentation of several specific architectures and learning techniques. The Cauchy machine, which represents a possible solution to the local minima problem encountered with virtually every other neural network training algorithm, is described. The outlook for neural nets is briefly considered. >

A study on neural networks

Abstract: The Hopfield neural network is a mathematical model in which each neuron performs a threshold logic function. an important property of the model is that a neural network always converges to a stable state when operating in a serial mode. This property is the basis of potential applications of neural networks such as associative memory devices, computational models, etc. This article reviews some of the known properties of the model and presents some new results regarding its possible applications. the principal contributions which are developed in this article are: (1) Showing that a very large class of mappings are not feasible by neural nets, in particular mappings which contain spheres, e.g., Hamming codes. (2) Showing that the neural network model can be designed to perform a local search algorithm for the Directed Min Cut problem. (3) Exploring the term “capacity of the neural network model” and criticizing some results known in the literature. (4) Showing the limitations of the model for its use as a pattern recognizer by proving that all images with a single black point can be recognized by the network iff the network is fully connected.

Electronic implementation of associative memory based on neural network models

Abstract: An electronic embodiment of a neural network based associative memory in the form of a binary connection matrix is described. The nature of false memory errors, their effect on the information storage capacity of binary connection matrix memories, and a novel technique to eliminate such errors with the help of asymmetrical extra connections are discussed. The stability of the matrix memory system incorporating a unique local inhibition scheme is analyzed in terms of local minimization of an energy function. The memory's stability, dynamic behavior, and recall capability are investigated using a 32-"neuron" electronic neural network memory with a 1024-programmable binary connection matrix.

Learning state space trajectories in recurrent neural networks : a preliminary report.

Abstract: : We describe a procedure for finding learning state space trajectories in recurrent neural networks. Keywords: Connectionism; Learning algorithm; Trajectories following; Minimizing functionals.

Algorithm Development For Neural Networks

Abstract: This paper is concerned with the development of algorithms for solving optimiza­ tion problems with a network of artificial neurons. Although there is no notion of step-by-step sequencing in a neural network, it is possible to develop tools and techniques for interconnecting a network of neurons so that it will achieve stable states corresponding to possible solutions of a problem. An approach to implementing an important class of constraints in a network of artificial neurons is presented and illustrated by developing a solution to a resource allocation problem. 1. A neural network is a collection of highly-interconnected, simple analog process­ ing elements that mimic biological neurons. Although neural networks are greatly sim­ plified models of biological neural systems, they possess computational properties that are applicable to a variety of problems including speech and image processing, pattern recognition, cognitive information processing and combinatorial optimization.Neural computing relies upon massive parallelism and a high degree of connectivity among simple analog processors and therefore represents a radical departure from the von Neumann architecture. Unlike von Neumann machines, neural computations have no notion of step-by-step sequencing. Instead, computations result from the collective emergent behavior of a dynamical system of analog elements. While neural elements can assume continuous values between 0 and 1 as the computation progresses, they are usually designed to settle eventually into binary-valued states which collectively correspond to problem solutions. Neural networks, therefore, bear little resemblance to the analog processing that was popular in the 1950's.Much current research in neural networks is directed toward exploring their poten­ tial to produce intelligent behavior. The use of neural networks to produce intelli­ gent behavior has already been demonstrated in a variety of areas such as feature extraction and pattern classification. An equally important application of neural net­ works, pioneered by Hopfield and Tank,l>2 is the solution of optimization problems by simultaneously evaluating multiple competing hypotheses. Neural networks can rapidly produce good, though not necessarily optimal, solutions to such problems3.4.

Neural network models and their application to handwritten digit recognition

Abstract: Several neural network paradigms are discussed, and their application to the recognition of handwritten digits is considered. In particular, linear autoassociative systems, threshold logic networks, background error propagation models, Hopfield networks, and Boltzmann machines are considered. An explanation of each technique is presented and its application to digit recognition is discussed. The tradeoffs of time and space complexity versus recognition accuracy are considered. The objective is to determine the applicability of these techniques to the real-world need of the United States Postal Service (USPS) for a high-accuracy handwritten digit recognition algorithm. Recognition experiments are presented that were performed on a database of over 10000 handwritten digits that were extracted from live mail in a USPS mail-processing facility. The time required by each method and the recognition rates of the methods are discussed. >

Pattern Recognition Using A Neural Network

Abstract: In this paper we introduce a new "neural" network for pattern recognition based on a gradient system. It is not, however, attempted to model any known behavior of biological neurons. This network stores any number of non-binary patterns (as its limit points) and retrieves them by associative recall. The network does not suffer from erroneous limit points. A realization of the network is given, which have heavily interconnected computing units. Finally two network examples are discussed.

Neural networks: translation-, rotation- and scale-invariant pattern recognition

Abstract: A neural network model which is capable of recognising transformed versions of a set of learnt patterns is proposed The group of transformations includes global translations, rotations and scale transformations The neural firing thresholds are used as additional degrees of freedom

Recognition of general patterns using neural networks

Abstract: The Hopfield model of neural network stores memory in its symmetric synaptic connections and can only learn to recognize sets of nearly "orthogonal" patterns. A new algorithm is put forth to permit the recognition of general ("non-orthogonal") patterns. The algorithm specifies the construction of the new network's memory matrix Tij, which is, in general, asymmetrical and contains the Hopfield neural network (Hopfield 1982) as a special case. We find further that in addition to this new algorithm for general pattern recognition, there exists in fact a large class of Tij memory matrices which permit the recognition of non-orthogonal patterns. The general form of this class of Tij memory matrix is presented, and the projection matrix neural network (Personnaz et al. 1985) is found as a special case of this general form. This general form of memory matrix extends the library of memory matrices which allow a neural network to recognize non-orthogonal patterns. A neural network which followed this general form of memory matrix was modeled on a computer and successfully recognized a set of non-orthogonal patterns. The new network also showed a tolerance for altered and incomplete data. Through this new method, general patterns may be taught to the neural network.

Neural network learning algorithms

Abstract: The earliest network models of associative memory were based on correlations between input and output patterns of activity in linear processing units. These models have several features that make them attractive: The synaptic strengths are computed from information available locally at each synapse in a single trial; the information is distributed in a large number of connection strengths, the recall of stored information is associative, and the network can generalize to new input patterns that are similar to stored patterns. There are also severe limitations with this class of linear associative matrix models, including interference between stored items, especially between ones that are related, and inability to make decisions that are contingent on several inputs. New neural network models and neural network learning algorithms have been introduced recently that overcome some of the shortcomings of the associative matrix models of memory. These learning algorithms require many training examples to create the internal representations needed to perform a difficult task and generalize properly. They share some properties with human skill acquisition.

Analog Implementation of Shunting Neural Networks

Abstract: An extremely compact, all analog and fully parallel implementation of a class of shunting recurrent neural networks that is applicable to a wide variety of FET-based integration technologies is proposed. While the contrast enhancement, data compression, and adaptation to mean input intensity capabilities of the network are well suited for processing of sensory information or feature extraction for a content addressable memory (CAM) system, the network also admits a global Liapunov function and can thus achieve stable CAM storage itself. In addition the model can readily function as a front-end processor to an analog adaptive resonance circuit.

A continuous-time optical neural network

Abstract: A description is given of the architecture and functioning of an all-optical, continuous-time recurrent neural network. The network is a ring resonator which contains a saturable, two-beam amplifier, two volume holograms, and a linear, two-beam amplifier. The saturable amplifier permits, through the use of a spatially patterned signal beam, the realization of an optical neuron array; the two volume holograms provide global network interconnectivity; and the linear amplifier supplies sufficient cavity gain to permit resonant, convergent operation of the network. Numerical solutions of the network equations of motion indicate that, for real-valued neural state vectors, the network functions in much the same way as either J.J. Hopfield's continuous-time model (1984) or a continuous-time version of D.Z. Anderson and M.C. Erie's BSB model (1987). For complex-valued neural state vectors, the network always converges to the dominant network attractor, thereby suggesting a paradigm for solving optimization problems in which entrapment by local minima is avoided. >

A neural network architecture for cue-based motion planning

Abstract: The principles of memory organization of plans are presented, and the role of sensory cues in the timely selection and execution of plans is demonstrated. The two major components of learning a cue-based plan, developing the ability to detect cues and associating cues with the relevant responses, are described. The preliminary development of neural-network mechanisms for learning cue-based plans is presented. It is shown that hard-wired neural networks provide the input to adaptive neural networks that learn an internal representation of the relevant cues and the threshold levels associated with them. Self-organizing neural networks learn to associate cues with changes in action and to construct cue-based plans. >

Neural network computation

Abstract: Since 1982 there has been an enormous resurgence of interest in the possibility of making trainable, general purpose pattern recognition machines which are intended to mimic some of the processing abilities of human brains. A neural network 'computer' consists of a set of processing units (or artificial neurons) joined together by a set of weighted connections. Such networks are programmed by applying training patterns which fix the output states of some or all of the units, and a learning algorithm then adjusts the connections in response to the training patterns. The author provides an introduction to the field and to the current state of the art.

A self-learning threshold-controlled neural network

Abstract: A neural network structure is proposed that is controlled by device thresholds rather than multiplicative factors. This network has the feature that the learning parameter is embodied locally in the device thresholds. The network is shown to be capable of learning by example, as well as exhibiting other desirable features of the Hopfield type networks. >