Showing papers on "Artificial neural network published in 1988"

Cellular neural networks: theory

Abstract: A novel class of information-processing systems called cellular neural networks is proposed. Like neural networks, they are large-scale nonlinear analog circuits that process signals in real time. Like cellular automata, they consist of a massive aggregate of regularly spaced circuit clones, called cells, which communicate with each other directly only through their nearest neighbors. Each cell is made of a linear capacitor, a nonlinear voltage-controlled current source, and a few resistive linear circuit elements. Cellular neural networks share the best features of both worlds: their continuous-time feature allows real-time signal processing, and their local interconnection feature makes them particularly adapted for VLSI implementation. Cellular neural networks are uniquely suited for high-speed parallel signal processing. >

An introduction to computing with neural nets

Abstract: Artificial neural net models have been studied for many years in the hope of achieving human-like performance in the fields of speech and image recognition. These models are composed of many nonlinear computational elements operating in parallel and arranged in patterns reminiscent of biological neural nets. Computational elements or nodes are connected via weights that are typically adapted during use to improve performance. There has been a recent resurgence in the field of artificial neural nets caused by new net topologies and algorithms, analog VLSI implementation techniques, and the belief that massive parallelism is essential for high performance speech and image recognition. This paper provides an introduction to the field of artificial neural nets by reviewing six important neural net models that can be used for pattern classification. These nets are highly parallel building blocks that illustrate neural net components and design principles and can be used to construct more complex systems. In addition to describing these nets, a major emphasis is placed on exploring how some existing classification and clustering algorithms can be performed using simple neuron-like components. Single-layer nets can implement algorithms required by Gaussian maximum-likelihood classifiers and optimum minimum-error classifiers for binary patterns corrupted by noise. More generally, the decision regions required by any classification algorithm can be generated in a straightforward manner by three-layer feed-forward nets.

Cellular neural networks: applications

Abstract: The theory of a novel class of information-processing systems, called cellular neural networks, which are capable of high-speed parallel signal processing, was presented in a previous paper (see ibid., vol.35, no.10, p.1257-72, 1988). A dynamic route approach for analyzing the local dynamics of this class of neural circuits is used to steer the system trajectories into various stable equilibrium configurations which map onto binary patterns to be recognized. Some applications of cellular neural networks to such areas as image processing and pattern recognition are demonstrated, albeit with only a crude circuit. In particular, examples of cellular neural networks which can be designed to recognize the key features of Chinese characters are presented. >

Self-organization in a perceptual network

Abstract: The emergence of a feature-analyzing function from the development rules of simple, multilayered networks is explored. It is shown that even a single developing cell of a layered network exhibits a remarkable set of optimization properties that are closely related to issues in statistics, theoretical physics, adaptive signal processing, the formation of knowledge representation in artificial intelligence, and information theory. The network studied is based on the visual system. These results are used to infer an information-theoretic principle that can be applied to the network as a whole, rather than a single cell. The organizing principle proposed is that the network connections develop in such a way as to maximize the amount of information that is preserved when signals are transformed at each processing stage, subject to certain constraints. The operation of this principle is illustrated for some simple cases. >

Feature discovery by competitive learning

Abstract: This paper reporis the results of our studies with an unsupervised learning paradigm which we have called “Competitive Learning” We have examined competitive learning using both computer simulation and formal analysis and hove found that when it is applied to parallel networks of neuron-like elements, many potentially useful learning tasks can be accomplished We were attracted to competitive learning because it seems to provide o way to discover the salient, general features which can be used to classify o set of patterns We show how o very simply competitive mechanism con discover a set of feature detectors which capture important aspects of the set of stimulus input patterns We 0150 show how these feature detectors con form the basis of o multilayer system that con serve to learn categorizations of stimulus sets which ore not linearly separable We show how the use of correlated stimuli con serve IX o kind of “teaching” input to the system to allow the development of feature detectors which would not develop otherwise Although we find the competitive learning mechanism o very interesting and powerful learning principle, we do not, of course, imagine thot it is the only learning principle Competitive learning is cm essentially nonassociative stotisticol learning scheme We certainly imagine that other kinds of learning mechanisms will be involved in the building of associations among patterns of activation in o more complete neural network We offer this analysis of these competitive learning mechanisms to further our understanding of how simple adaptive networks can discover features importont in the description of the stimulus environment in which the system finds itself

Auto-association by multilayer perceptrons and singular value decomposition

Abstract: The multilayer perceptron, when working in auto-association mode, is sometimes considered as an interesting candidate to perform data compression or dimensionality reduction of the feature space in information processing applications. The present paper shows that, for auto-association, the nonlinearities of the hidden units are useless and that the optimal parameter values can be derived directly by purely linear techniques relying on singular value decomposition and low rank matrix approximation, similar in spirit to the well-known Karhunen-Loeve transform. This approach appears thus as an efficient alternative to the general error back-propagation algorithm commonly used for training multilayer perceptrons. Moreover, it also gives a clear interpretation of the role of the different parameters.

The ART of adaptive pattern recognition by a self-organizing neural network

Abstract: The adaptive resonance theory (ART) suggests a solution to the stability-plasticity dilemma facing designers of learning systems, namely how to design a learning system that will remain plastic, or adaptive, in response to significant events and yet remain stable in response to irrelevant events. ART architectures are discussed that are neural networks that self-organize stable recognition codes in real time in response to arbitrary sequences of input patterns. Within such an ART architecture, the process of adaptive pattern recognition is a special case of the more general cognitive process of hypothesis discovery, testing, search, classification, and learning. This property opens up the possibility of applying ART systems to more general problems of adaptively processing large abstract information sources and databases. The main computational properties of these ART architectures are outlined and contrasted with those of alternative learning and recognition systems. >

A multilayered neural network controller

Abstract: A multilayered neural network processor is used to control a given plant Several learning architectures are proposed for training the neural controller to provide the appropriate inputs to the plant so that a desired response is obtained A modified error-back propagation algorithm, based on propagation of the output error through the plant, is introduced The properties of the proposed architectures are studied through a simulation example >

Neural networks for nonlinear programming

Abstract: The dynamics of the modified canonical nonlinear programming circuit are studied and how to guarantee the stability of the network's solution. By considering the total cocontent function, the solution of the canonical nonlinear programming circuit is reconciled with the problem being modeled. In addition, it is shown how the circuit can be realized using a neural network, thereby extending the results of D.W. Tank and J.J. Hopefield (ibid., vol.CAS-33, p.533-41, May 1986) to the general nonlinear programming problem. >

Dynamic pattern generation in behavioral and neural systems

Abstract: In the search for principles of pattern generation in complex biological systems, an operational approach is presented that embraces both theory and experiment. The central mathematical concepts of self-organization in nonequilibrium systems (including order parameter dynamics, stability, fluctuations, and time scales) are used to show how a large number of empirically observed features of temporal patterns can be mapped onto simple low-dimensional (stochastic, nonlinear) dynamical laws that are derivable from lower levels of description. The theoretical framework provides a language and a strategy, accompanied by new observables, that may afford an understanding of dynamic patterns at several scales of analysis (including behavioral patterns, neural networks, and individual neurons) and the linkage among them.

Accelerating the convergence of the back-propagation method

Abstract: The utility of the back-propagation method in establishing suitable weights in a distributed adaptive network has been demonstrated repeatedly. Unfortunately, in many applications, the number of iterations required before convergence can be large. Modifications to the back-propagation algorithm described by Rumelhart et al. (1986) can greatly accelerate convergence. The modifications consist of three changes:1) instead of updating the network weights after each pattern is presented to the network, the network is updated only after the entire repertoire of patterns to be learned has been presented to the network, at which time the algebraic sums of all the weight changes are applied:2) instead of keeping ź, the "learning rate" (i.e., the multiplier on the step size) constant, it is varied dynamically so that the algorithm utilizes a near-optimum ź, as determined by the local optimization topography; and3) the momentum factor ź is set to zero when, as signified by a failure of a step to reduce the total error, the information inherent in prior steps is more likely to be misleading than beneficial. Only after the network takes a useful step, i.e., one that reduces the total error, does ź again assume a non-zero value. Considering the selection of weights in neural nets as a problem in classical nonlinear optimization theory, the rationale for algorithms seeking only those weights that produce the globally minimum error is reviewed and rejected.

An empirical study of learning speed in back-propagation networks

Abstract: Most connectionist or "neural network" learning systems use some form of the back-propagation algorithm. However, back-propagation learning is too slow for many applications, and it scales up poorly as tasks become larger and more complex. The factors governing learning speed are poorly understood. I have begun a systematic, empirical study of learning speed in backprop-like algorithms, measured against a variety of benchmark problems. The goal is twofold: to develop faster learning algorithms and to contribute to the development of a methodology that will be of value in future studies of this kind. This paper is a progress report describing the results obtained during the first six months of this study. To date I have looked only at a limited set of benchmark problems, but the results on these are encouraging: I have developed a new learning algorithm that is faster than standard backprop by an order of magnitude or more and that appears to scale up very well as the problem size increases. This research was sponsored in part by the National Science Foundation under Contract Number EET-8716324 and by the Defense Advanced Research Projects Agency (DOD), ARPA Order No. 4976 under Contract F33615-87C-1499 and monitored by the Avionics Laboratory, Air Force Wright Aeronautical Laboratories, Aeronautical Systems Division (AFSC), Wright-Patterson AFB, OH 45433-6543. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of these agencies or of the U.S. Government.

Neural networks and physical systems with emergent collective computational abilities

Abstract: Computational properties of use to biological organisms or to the construction of computers can emerge as collective properties of systems having a large number of simple equivalent components (or neurons). The physical meaning of content-addressable memory is described by an appropriate phase space flow of the state of a system. A model of such a system is given, based on aspects of neurobiology but readily adapted to integrated circuits. The collective properties of this model produce a content-addressable memory which correctly yields an entire memory from any subpart of sufficient size. The algorithm for the time evolution of the state of the system is based on asynchronous parallel processing. Additional emergent collective properties include some capacity for generalization, familiarity recognition, categorization, error correction, and time sequence retention. The collective properties are only weakly sensitive to details of the modeling or the failure of individual devices.

Art 2: Self-Organization Of Stable Category Recognition Codes For Analog Input Patterns

Abstract: Adaptive resonance architectures are neural networks that self-organize stable pattern recognition codes in real-time in response to arbitrary sequences of input patterns. This article introduces ART 2, a class of adaptive resonance architectures which rapidly self-organize pattern recognition categories in response to arbitrary sequences of either analog of binary input patterns. In order to cope with arbitrary sequences of analog input patterns, ART 2 architectures embody solutions to a number of design principles, such as the stability-plasticity tradeoff, the search-direct access tradeoff, and the match-reset tradeoff. In these architectures, top-down learned expectation and matching mechanisms are critical in self-stabilizing the code learning process. A parallel search scheme updates itself adaptively as the learning process unfolds, and realizes a form of real-time hypothesis discovery, testing, learning, and recognition. After learning self-stabilizes, the search process is automatically disengaged. Thereafter input patterns directly access their recognition codes without any search. Thus recognition time for familiar inputs does not increase with the complexity of the learned code. A novel input pattern can directly access a category if it shares invariant properties with the set of familiar exemplars of that category. A parameter called the attentional vigilance parameter determines how fine the categories will be. If vigilance increases (decreases) due to environmental feedback, then the system automatically searches for and learns finer (coarser) recognition categories. Gain control parameters enable the architecture to suppress noise up to a prescribed level. The architecture's global design enables it to learn effectively despite the high degree of nonlinearity of such mechanisms.

Using the ADAP Learning Algorithm to Forecast the Onset of Diabetes Mellitus

Abstract: Neural networks or connectionist models for parallel processing are not new. However, a resurgence of interest in the past half decade has occurred. In part, this is related to a better understanding of what are now referred to as hidden nodes. These algorithms are considered to be of marked value in pattern recognition problems. Because of that, we tested the ability of an early neural network model, ADAP, to forecast the onset of diabetes mellitus in a high risk population of Pima Indians. The algorithm's performance was analyzed using standard measures for clinical tests: sensitivity, specificity, and a receiver operating characteristic curve. The crossover point for sensitivity and specificity is 0.76. We are currently further examining these methods by comparing the ADAP results with those obtained from logistic regression and linear perceptron models using precisely the same training and forecasting sets. A description of the algorithm is included.

Neural Networks and Natural Intelligence

Abstract: From the Publisher: Stephen Grossberg and his colleagues at Boston University's Center for Adaptive Systems are producing some of the most exciting research in the neural network approach to making computers "think." Packed with real-time computer simulations and rigorous demonstrations of these phenomena, this book includes results on vision, speech, cognitive information processing; adaptive pattern recognition, adaptive robotics, conditioning and attention, cognitive-emotional interactions, and decision making under risk.

Probabilistic neural networks for classification, mapping, or associative memory

Abstract: It can be shown that by replacing the sigmoid activation function often used in neural networks with an exponential function, a neural network can be formed which computes nonlinear decision boundaries. This technique yields decision surfaces which approach the Bayes optimal under certain conditions. There is a continuous control of the linearity of the decision boundaries, from linear for small training sets to any degree of nonlinearity justified by larger training sets. A four-layer neural network of the type proposed can map any input pattern to any number of classifications. The input variables can be either continuous or binary. Modification of the decision boundaries based on new data can be accomplished in real time simply by defining a set of weights equal to the new training vector. The decision boundaries can be implemented using analog 'neurons', which operate entirely in parallel. The organization proposed takes into account the projected pin limitations of neural-net chips of the near future. By a change in architecture, these same components could be used as associative memories, to compute nonlinear multivariate regression surfaces, or to compute a posteriori probabilities of an event. >

Adding a conscience to competitive learning

Abstract: There are a number of neural networks that self-organize on the basis of what has come to be known as Kohonen learning. The author introduces a modification of Kohonen learning that provides rapid convergence and improved representation of the input data. In many areas of pattern recognition, statistical analysis, and control, it is essential to form a nonparametric model of a probability density function p(x). The purpose of the improvement to Kohonen learning presented is to form a better approximation of p(x). Simulation results are presented to illustrate the operation of this competitive learning algorithm. >

Training a 3-Node Neural Network is NP-Complete

Abstract: We consider a 2-layer, 3-node, n-input neural network whose nodes compute linear threshold functions of their inputs. We show that it is NP-complete to decide whether there exist weights and thresholds for the three nodes of this network so that it will produce output consistent with a given set of training examples. We extend the result to other simple networks. This result suggests that those looking for perfect training algorithms cannot escape inherent computational difficulties just by considering only simple or very regular networks. It also suggests the importance, given a training problem, of finding an appropriate network and input encoding for that problem. It is left as an open problem to extend our result to nodes with non-linear functions such as sigmoids.

The 'neural' phonetic typewriter

Abstract: The factors that make speech recognition difficult are examined, and the potential of neural computers for this purpose is discussed. A speaker-adaptive system that transcribes dictation using an unlimited vocabulary is presented that is based on a neural network processor for the recognition of phonetic units of speech. The acoustic preprocessing, vector quantization, neural network model, and shortcut learning algorithm used are described. The utilization of phonotopic maps and of postprocessing in symbolic forms are discussed. Hardware implementations and performance of the neural networks are considered. >

Statistical pattern recognition with neural networks: benchmarking studies

Abstract: Three basic types of neural-like networks (backpropagation network, Boltzmann machine, and learning vector quantization), were applied to two representative artificial statistical pattern recognition tasks, each with varying dimensionality. The performance of each network's approach to solving the tasks was evaluated and compared, both to the performance of the other two networks and to the theoretical limit. The learning vector quantization was further benchmarked against the parametric Bayes classifier and the k-nearest-neighbor classifier using natural speech data. A novel learning vector quantization classifier called LVQ2 is introduced. >

Cellular neural networks

Abstract: A novel class of information-processing systems called cellular neural networks is proposed. Like neural networks, they are large-scale nonlinear circuits which process signals in real time. Like cellular automata, they are made of a massive aggregate of regularly spaced circuit clones, called cells, which communicate with each other directly only through their nearest neighbors. Some applications in such areas as image processing are demonstrated, albeit with only a crude circuit. In particular, examples of cellular neural networks which can be designed to recognize the key features of Chinese characters are presented. Some theorems concerning the dynamic range and the steady states of cellular neural networks are proved. In view of the near-neighbor interactive property of cellular neural networks, they are much more amenable to VLSI implementation than general neural networks. >

Neural nets for adaptive filtering and adaptive pattern recognition

Abstract: The adaptive linear combiner (ALC) is described, and practical applications of the ALC in signal processing and pattern recognition are presented. Six signal processing examples are given, which are system modeling, statistical prediction, noise canceling, echo canceling, universe modeling, and channel equalization. Adaptive pattern recognition using neural nets is then discussed. The concept involves the use of an invariance net followed by a trainable classifier. It makes use of a multilayer adaptation algorithm that descrambles output and reproduces original patterns. >

The Enhanced Storage Capacity in Neural Networks with Low Activity Level

Abstract: The modified Hopfield model defined in terms of "V-variables" (V = 0; 1), which is appropriate for storage of correlated patterns, is considered. The learning algorithm is proposed to enhance significantly the storage capacity in comparison with previous estimates. At low levels of neural activity, p 1, we obtain αc(p) ~ (p|ln p|)-1 which resembles Gardner's estimate for the maximum storage capacity.