Showing papers on "Artificial neural network published in 1986"

Simple 'neural' optimization networks: An A/D converter, signal decision circuit, and a linear programming circuit

Abstract: We describe how several optimization problems can be rapidly solved by highly interconnected networks of simple analog processors. Analog-to-digital (A/D) conversion was considered as a simple optimization problem, and an A/D converter of novel architecture was designed. A/D conversion is a simple example of a more general class of signal-decision problems which we show could also be solved by appropriately constructed networks. Circuits to solve these problems were designed using general principles which result from an understanding of the basic collective computational properties of a specific class of analog-processor networks. We also show that a network which solves linear programming problems can be understood from the same concepts.

Experiments on Learning by Back Propagation.

Abstract: : Rumelhart, Hinton and Williams (Rumelhart 86) describe a learning procedure for layered networks of deterministic, neuron-like units. This paper describes further research on the learning procedure. We start by describing the units, the way they are connected, the learning procedure, and the extension to iterative nets. We then give an example in which a network learns a set of filters that enable it to discriminate formant-like patterns in the presence of noise. The speed of learning is strongly dependent on the shape of the surface formed by the error measure in weight space . We give examples of the shape of the error surface for a typical task and illustrate how an acceleration method speeds up descent in weight space. The main drawback of the learning procedure is the way it scales as the size of the task and the network increases. We give some preliminary results on scaling and show how the magnitude of the optimal weight changes depends on the fan-in of the units. Additional results illustrate the effects on learning speed of the amount of interaction between the weights. A variation of the learning procedure that back-propagates desired state information rather than error gradients is developed and compared with the standard procedure. Finally, we discuss the relationship between our iterative networks and the analog networks described by Hopefield and Tank (Hopfield 85). The learning procedure can discover appropriate weights in their kind of network, as well as determine an optimal schedule for varying the nonlinearity of the units during a search.

Sequential state generation by model neural networks

Abstract: Sequential patterns of neural output activity form the basis of many biological processes, such as the cyclic pattern of outputs that control locomotion I show how such sequences can be generated by a class of model neural networks that make defined sets of transitions between selected memory states Sequence-generating networks depend upon the interplay between two sets of synaptic connections One set acts to stabilize the network in its current memory state, while the second set, whose action is delayed in time, causes the network to make specified transitions between the memories The dynamic properties of these networks are described in terms of motion along an energy surface The performance of the networks, both with intact connections and with noisy or missing connections, is illustrated by numerical examples In addition, I present a scheme for the recognition of externally generated sequences by these networks

Neural networks for computation: number representations and programming complexity.

Abstract: Methods for using neural networks for computation are considered. The success of such networks in finding good solutions to complex problems is found to be dependent on the number representation schemes used. Redundant schemes are found to offer advantages in terms of convergence. Neural networks are applied to the combinatorial optimization problem known as the Hitchcock problem and signal processing problems, such as matrix inversion and Fourier transformation. The concept of programming complexity is introduced. It is shown that for some computational problems, the programming complexity may be so great as to limit the utility of neural networks, while for others the investment of computation in programming the network is justified. Simulations of neural networks using a digital computer are presented.

Neural networks with nonlinear synapses and a static noise

Abstract: The theory of neural networks is extended to include a static noise as well as nonlinear updating of synapses by learning. The noise appears either in the form of spin-glass interactions, which are independent of the learning process, or as a random decaying of synapses. In an unsaturated network, the nonlinear learning algorithms may modify the energy surface and lead to interesting new computational capabilities. Close to saturation, they act as an additional source of a static noise. The effect of the noise on memory storage is calculated.

Networks of Formal Neurons and Memory Palimpsests

Abstract: One characteristic behaviour of the Hopfield model of neural networks, namely the catastrophic deterioration of the memory due to overloading, is interpreted in simple physical terms. A general formulation allows for an exploration of some basic issues in learning theory. Two learning schemes are constructed, which avoid the overloading deterioration and keep learning and forgetting, with a stationary capacity.

Asymmetric neural networks and the process of learning

Abstract: Studies the influence of a strong asymmetry of the synaptic strengths on the behavior of a neural network which works as an associative memory. The author finds that the asymmetry in the synaptic strengths may be crucial for the process of learning.

Learning Process in an Asymmetric Threshold Network

Abstract: Threshold functions and related operators are widely used as basic elements of adaptive and associative networks [Nakano 72, Amari 72, Hopfield 82]. There exist numerous learning rules for finding a set of weights to achieve a particular correspondence between input-output pairs. But early works in the field have shown that the number of threshold functions (or linearly separable functions) in N binary variables is small compared to the number of all possible boolean mappings in N variables, especially if N is large. This problem is one of the main limitations of most neural networks models where the state is fully specified by the environment during learning: they can only learn linearly separable functions of their inputs. Moreover, a learning procedure which requires the outside world to specify the state of every neuron during the learning session can hardly be considered as a general learning rule because in real-world conditions, only a partial information on the “ideal” network state for each task is available from the environment. It is possible to use a set of so-called “hidden units” [Hinton,Sejnowski,Ackley. 84], without direct interaction with the environment, which can compute intermediate predicates. Unfortunately, the global response depends on the output of a particular hidden unit in a highly non-linear way, moreover the nature of this dependence is influenced by the states of the other cells.

Neural Representation of Conceptual Knowledge.

Abstract: : The neural encoding of memory is a problem of great interest and importance. Trlalditional proposals have taken one of two extreme views: The one-concept, one-neuron, punctate view and the full distributed, holographic alternative. Major advances in the behavioral, biological and computational sciences have greatly increased our understanding of the question and its potential answers. There is now good reason to reject both extreme views, but a compact encoding that derives from the punctate model appears to fit well with all the facts. Much of the work espousing holographic models is reinterpreted as studying system properties of neural networks and, as such, considered to be of great importance. Some suggestions for directions of further research are discussed.

Feature discovery by competitive learning

Abstract: This paper reporis the results of our studies with an unsupervised learning paradigm which we have called “Competitive Learning.” We have examined competitive learning using both computer simulation and formal analysis and hove found that when it is applied to parallel networks of neuron-like elements, many potentially useful learning tasks can be accomplished. We were attracted to competitive learning because it seems to provide o way to discover the salient, general features which can be used to classify o set of patterns. We show how o very simply competitive mechanism con discover a set of feature detectors which capture important aspects of the set of stimulus input patterns. We 0150 show how these feature detectors con form the basis of o multilayer system that con serve to learn categorizations of stimulus sets which ore not linearly separable. We show how the use of correlated stimuli con serve IX o kind of “teaching” input to the system to allow the development of feature detectors which would not develop otherwise. Although we find the competitive learning mechanism o very interesting and powerful learning principle, we do not, of course, imagine thot it is the only learning principle. Competitive learning is cm essentially nonassociative stotisticol learning scheme. We certainly imagine that other kinds of learning mechanisms will be involved in the building of associations among patterns of activation in o more complete neural network. We offer this analysis of these competitive learning mechanisms to further our understanding of how simple adaptive networks can discover features importont in the description of the stimulus environment in which the system finds itself.

The ultrametric organization of memories in a neural network

Abstract: In the original formulation of Hopfield's memory model, the learning rule setting the interaction strengths is best suited for orthogonal words. From the point of view of categorization, this feature is not convenient unless we reinterpret these words as primordial categories. But then one has to complete the model so as to be able to store a full hierarchical tree of categories embodying subcategories and so on. We use recent results on the spin glass mean field theories to show that this completion can be done in a natural way with a minimal modification of Hebb's rule for learning. Categorization emerges naturally from an encoding stage structured in layers.

Evaluation of the use of the Hopfield neural network model as a nearest-neighbor algorithm.

Abstract: Neural network models are receiving increasing attention because of their collective computational capabilities. We evaluate the use of the Hopfield neural network model in optically determining the nearest-neighbor of a binary bipolar test vector from a set of binary bipolar reference vectors. The use of the Hopfield model is compared with that of a direct technique called direct storage nearest-neighbor that accomplishes the task of nearest-neighbor determination.

Optimization algorithms: simulated annealing and neural network processing

Abstract: On developpe et l'on compare deux algorithmes precedemment utilises pour une optimisation discrete. On montre comment ces algorithmes peuvent etre utilises pour trouver des extremes globaux de fonctions tout en evitant d'etre piege dans des extremes locaux

Artificial neural networks for computing

Abstract: Recent proposals for neural network models indicate that an array of amplifiers coupled to a lattice of wires with resistive components at the crosspoints can perform calculations using collective properties similar to those observed in biological systems. Such a network can perform both memory and processing functions. The promise of the connection matrix processor lies in its very high density, fault tolerance, and massively parallel operation. This paper describes the operation of a neural network and exploratory fabrication techniques for its implementation.

Collective Properties of Neural Networks

Abstract: A neural network is basically made of interconnected threshold automata i. (i=1,…., N) collecting the signals Sj produced by a set of upstream units j and delivering an output signal Si to a set of downstream units k. The evolution of the dynamic variables is driven by the following equations: $$ {S_i}(t) = F({V_i}(\{ {S_i}(t - {\Delta_{{ij}}})\} - {e_i}) $$ (1) where F is a sigmoid function, Vi the membrane potential of i, Δij the transmission delay between the units j and i and ei the threshold of i.

Stochastic Dynamics of Neural Networks

Abstract: The dynamics of stochastic neural networks are presented. The model is an algorithm based upon the assumption that the activities of neurons are asynchronous. It is proven that networks with fixed synaptic efficacies cannot sustain oscillating activities. Following Choi and Huberman, the dynamics of instantaneous frequencies is derived. The equations are solved for associative and for recursive networks by introducing order parameters coupled to stored patterns. It is shown that the networks relax towards one of the stored configurations in a matter of a few refractory periods, whatever the size of the network. The relaxation time diverges in recursive networks at a critical noise Bc, above which no stored pattern can be retrieved. These results have been confirmed using computer simulations.

Storing and Retrieving Information in a Layered Spin System

Abstract: We introduce a neural network model with layered architecture and binary (spin) variables. Hebbian rules are used to define unidirectional couplings between spins of adjacent layers. A fast learning algorithm produces couplings that store a large number of random patterns, and efficiently recognizes noisy patterns. Performance of this network is compared with spin-glass type models of pattern recognition.

Associative recognition and storage in a model network of physiological Neurons

Abstract: We consider a neural network model in which the single neurons are chosen to closely resemble known physiological properties. The neurons are assumed to be linked by synapses which change their strength according to Hebbian rules on a short time scale (100 ms). The dynamics of the network--the time evolution of the cell potentials and the synapses--is investigated by computer simulation. As in more abstract network models (Cooper 1973; Hopfield 1982; Kohonen 1984) it is found that the local dynamics of the cell potentials and the synaptic strengths result in global cooperative properties of the network and enable the network to process an incoming flux of information and to learn and store patterns associatively. A trained net can associate missing details of a pattern, can correct wrong details and can suppress noise in a pattern. The network can further abstract the prototype from a series of patterns with variations. A suitable coupling constant connecting the dynamics of the cell potentials with the synaptic strengths is derived by a mean field approximation. This coupling constant controls the neural sensitivity and thereby avoids both extremes of the network state, the state of permanent inactivity and the state of epileptic hyperactivity.

Associative Learning, Adaptive Pattern Recognition, And Cooperative-Competitive Decision Making By Neural Networks

Abstract: This article describes models of associative pattern learning, adaptive pattern recognition, and parallel decision-making by neural networks. It is shown that a small set of real-time non-linear neural equations within a larger set of specialized neural circuits can be used to study a wide variety of such problems. Models of energy minimization, cooperative-competitive decision making, competitive learning, adaptive resonance, interactive activation, and back propagation are discussed and compared.

A biologically constrained learning mechanism in networks of formal neurons

Abstract: A new learning mechanism is proposed for networks of formal neurons analogous to Ising spin systems; it brings such models substantially closer to biological data in three respects: first, the learning procedure is applied initially to a network with random connections (which may be similar to a spin-glass system), instead of starting from a system void of any knowledge (as in the Hopfield model); second, the resultant couplings are not symmetrical; third, patterns can be stored without changing the sign of the coupling coefficients. It is shown that the storage capacity of such networks is similar to that of the Hopfield network, and that it is not significantly affected by the restriction of keeping the couplings' signs constant throughout the learning phase. Although this approach does not claim to model the central nervous system, it provides new insight on a frontier area between statistical physics, artificial intelligence, and neurobiology.

Temporal sequences with neural networks

Abstract: A sequence generator employing a neural network having its output coupled to at least one plurality of delay elements. The delayed outputs are fed back to an input interconnection network, wherein they contribute to the next state transition through an appropriate combination of interconnections.

Neural networks, pattern recognition, and fingerprint hallucination

Abstract: Many interesting and globally ordered patterns of behavior, such as solidification, arise in statistical physics and are generally referred to as collective phenomena. The obvious analogies to parallel computation can be extended quite far, so that simple computations may be endowed with the most desirable properties of collective phenomena: robustness against circuit defects, extreme parallelism, asynchronous operation and efficient implementation in silicon. To obtain these advantages for more complicated and useful computations, the relatively simple pattern recognition task of fingerprint identification has been selected. Simulations show that an intuitively understandable neural network can generate fingerprint-like patterns within a framework which should allow control of wire length and scale invariance. The purpose of generating such patterns is to create a network whose stable states are noiseless fingerprint patterns, so that noisy fingerprint patterns used as input to the network will evoke the corresponding noiseless patterns as output. There is a developing theory for predicting the behavior of such networks and thereby reducing the amount of simulation that must be done to design them.

Associative Networks and Cell Assemblies

Abstract: Since the time of McCulloch and Pitts’ Theory (1943) there have been many attempts to model the flow of activity in neural networks. It is possible to simulate neural networks (of rather small size) on a computer, relying on quite reasonable — more or less simplified — assumptions on the dynamic behavior of single neurons. One problem is the arbitrariness of the design of the network (i.e. the connectivity matrix). Here many investigations have studied random connectivity (e.g. Anninos et al. 1970, Griffith 1971, Amari 1974, Dammasch and Wagner 1984) or connectivity that itself changes subject to certain rules (for an overview see Palm 1982).