Showing papers on "Artificial neural network published in 1990"

The self-organizing map

Abstract: The self-organized map, an architecture suggested for artificial neural networks, is explained by presenting simulation experiments and practical applications. The self-organizing map has the property of effectively creating spatially organized internal representations of various features of input signals and their abstractions. One result of this is that the self-organization process can discover semantic relationships in sentences. Brain maps, semantic maps, and early work on competitive learning are reviewed. The self-organizing map algorithm (an algorithm which order responses spatially) is reviewed, focusing on best matching cell selection and adaptation of the weight vectors. Suggestions for applying the self-organizing map algorithm, demonstrations of the ordering process, and an example of hierarchical clustering of data are presented. Fine tuning the map by learning vector quantization is addressed. The use of self-organized maps in practical speech recognition and a simulation experiment on semantic mapping are discussed. >

Identification and control of dynamical systems using neural networks

Abstract: It is demonstrated that neural networks can be used effectively for the identification and control of nonlinear dynamical systems. The emphasis is on models for both identification and control. Static and dynamic backpropagation methods for the adjustment of parameters are discussed. In the models that are introduced, multilayer and recurrent networks are interconnected in novel configurations, and hence there is a real need to study them in a unified fashion. Simulation results reveal that the identification and adaptive control schemes suggested are practically feasible. Basic concepts and definitions are introduced throughout, and theoretical questions that have to be addressed are also described. >

Backpropagation through time: what it does and how to do it

Abstract: Basic backpropagation, which is a simple method now being widely used in areas like pattern recognition and fault diagnosis, is reviewed. The basic equations for backpropagation through time, and applications to areas like pattern recognition involving dynamic systems, systems identification, and control are discussed. Further extensions of this method, to deal with systems other than neural networks, systems involving simultaneous equations, or true recurrent networks, and other practical issues arising with the method are described. Pseudocode is provided to clarify the algorithms. The chain rule for ordered derivatives-the theorem which underlies backpropagation-is briefly discussed. The focus is on designing a simpler version of backpropagation which can be translated into computer code and applied directly by neutral network users. >

Neural network ensembles

Abstract: Several means for improving the performance and training of neural networks for classification are proposed Crossvalidation is used as a tool for optimizing network parameters and architecture It is shown that the remaining residual generalization error can be reduced by invoking ensembles of similar networks >

Networks for approximation and learning

Abstract: The problem of the approximation of nonlinear mapping, (especially continuous mappings) is considered. Regularization theory and a theoretical framework for approximation (based on regularization techniques) that leads to a class of three-layer networks called regularization networks are discussed. Regularization networks are mathematically related to the radial basis functions, mainly used for strict interpolation tasks. Learning as approximation and learning as hypersurface reconstruction are discussed. Two extensions of the regularization approach are presented, along with the approach's corrections to splines, regularization, Bayes formulation, and clustering. The theory of regularization networks is generalized to a formulation that includes task-dependent clustering and dimensionality reduction. Applications of regularization networks are discussed. >

Large-scale neurocognitive networks and distributed processing for attention, language, and memory

Abstract: Cognition and comportment are subserved by interconnected neural networks that allow high-level computational architectures including parallel distributed processing. Cognitive problems are not resolved by a sequential and hierarchical progression toward predetermined goals but instead by a simultaneous and interactive consideration of multiple possibilities and constraints until a satisfactory fit is achieved. The resultant texture of mental activity is characterized by almost infinite richness and flexibility. According to this model, complex behavior is mapped at the level of multifocal neural systems rather than specific anatomical sites, giving rise to brain-behavior relationships that are both localized and distributed. Each network contains anatomically addressed channels for transferring information content and chemically addressed pathways for modulating behavioral tone. This approach provides a blueprint for reexploring the neurological foundations of attention, language, memory, and frontal lobe function.

30 years of adaptive neural networks: perceptron, Madaline, and backpropagation

Abstract: Fundamental developments in feedforward artificial neural networks from the past thirty years are reviewed. The history, origination, operating characteristics, and basic theory of several supervised neural-network training algorithms (including the perceptron rule, the least-mean-square algorithm, three Madaline rules, and the backpropagation technique) are described. The concept underlying these iterative adaptation algorithms is the minimal disturbance principle, which suggests that during training it is advisable to inject new information into a network in a manner that disturbs stored information to the smallest extent possible. The two principal kinds of online rules that have developed for altering the weights of a network are examined for both single-threshold elements and multielement networks. They are error-correction rules, which alter the weights of a network to correct error in the output response to the present input pattern, and gradient rules, which alter the weights of a network during each pattern presentation by gradient descent with the objective of reducing mean-square error (averaged over all training patterns). >

Improving the learning speed of 2-layer neural networks by choosing initial values of the adaptive weights

Abstract: The authors describe how a two-layer neural network can approximate any nonlinear function by forming a union of piecewise linear segments. A method is given for picking initial weights for the network to decrease training time. The authors have used the method to initialize adaptive weights over a large number of different training problems and have achieved major improvements in learning speed in every case. The improvement is best when a large number of hidden units is used with a complicated desired response. The authors have used the method to train the truck-backer-upper and were able to decrease the training time from about two days to four hours

Simulated Annealing and Boltzmann Machines: A Stochastic Approach to Combinatorial Optimization and Neural Computing.

Abstract: SIMULATED ANNEALING. Combinatorial Optimization. Simulated Annealing. Asymptotic Convergence. Finite-Time Approximation. Simulated Annealing in Practice. Parallel Simulated Annealing Algorithms. BOLTZMANN MACHINES. Neural Computing. Boltzmann Machines. Combinatorial Optimization and Boltzmann Machines. Classification and Boltzmann Machines. Learning and Boltzmann Machines. Appendix. Bibliography. Indices.

Statistical Pattern Recognition

Abstract: This chapter introduces the subject of statistical pattern recognition (SPR). It starts by considering how features are defined and emphasizes that the nearest neighbor algorithm achieves error rates comparable with those of an ideal Bayes’ classifier. The concepts of an optimal number of features, representativeness of the training data, and the need to avoid overfitting to the training data are stressed. The chapter shows that methods such as the support vector machine and artificial neural networks are subject to these same training limitations, although each has its advantages. For neural networks, the multilayer perceptron architecture and back-propagation algorithm are described. The chapter distinguishes between supervised and unsupervised learning, demonstrating the advantages of the latter and showing how methods such as clustering and principal components analysis fit into the SPR framework. The chapter also defines the receiver operating characteristic, which allows an optimum balance between false positives and false negatives to be achieved.

Regularization algorithms for learning that are equivalent to multilayer networks.

Abstract: Learning an input-output mapping from a set of examples, of the type that many neural networks have been constructed to perform, can be regarded as synthesizing an approximation of a multidimensional function (that is, solving the problem of hypersurface reconstruction). From this point of view, this form of learning is closely related to classical approximation techniques, such as generalized splines and regularization theory. A theory is reported that shows the equivalence between regularization and a class of three-layer networks called regularization networks or hyper basis functions. These networks are not only equivalent to generalized splines but are also closely related to the classical radial basis functions used for interpolation tasks and to several pattern recognition and neural network algorithms. They also have an interesting interpretation in terms of prototypes that are synthesized and optimally combined during the learning stage.

Non-linear system identification using neural networks

Abstract: Multi-layered neural networks offer an exciting alternative for modelling complex non-linear systems. This paper investigates the identification of discrete-time nonlinear systems using neural networks with a single hidden layer. New parameter estimation algorithms are derived for the neural network model based on a prediction error formulation and the application to both simulated and real data is included to demonstrate the effectiveness of the neural network approach.

Single-layer learning revisited: a stepwise procedure for building and training a neural network

Abstract: A stepwise procedure for building and training a neural network intended to perform classification tasks, based on single layer learning rules, is presented. This procedure breaks up the classification task into subtasks of increasing complexity in order to make learning easier. The network structure is not fixed in advance: it is subject to a growth process during learning. Therefore, after training, the architecture of the network is guaranteed to be well adapted for the classification problem.

The multilayer perceptron as an approximation to a Bayes optimal discriminant function

Abstract: The multilayer perceptron, when trained as a classifier using backpropagation, is shown to approximate the Bayes optimal discriminant function. The result is demonstrated for both the two-class problem and multiple classes. It is shown that the outputs of the multilayer perceptron approximate the a posteriori probability functions of the classes being trained. The proof applies to any number of layers and any type of unit activation function, linear or nonlinear. >

Neural Computing - An Introduction

Abstract: INTRODUCTION Humans and computers The structure of the brain Learning in machines The differences Summary PATTERN RECOGNITION Introduction Pattern recognition in perspective Pattern recognition-a definition Feature vectors and feature space Discriminant functions Classification techniques Linear classifiers Statistical techniques Pattern recognition-a summary THE BASIC NEURON Introduction Modeling the single neuron Learning in simple neurons The perceptron: a vectorial perspective The perceptron learning rule: proof Limitations of perceptrons The end of the line? Summary THE MULTILAYER PERCEPTRON Introduction Altering the perceptron model The new model The new learning rule The multilayer perceptron algorithm The XOR problem revisited Visualizing network behavior Multilayer perceptrons as classifiers Generalization Fault tolerance Learning difficulties Radial basis functions Applications Summary KOHONEN SELF-ORGANIZING NETWORKS Introduction The Kohonen algorithm Weight training Neighborhoods Reducing the neighborhood Learning vector quantization (LVQ) The phonetic typewriter Summary HOPFIELD NETWORKS Introduction The Hopfield model The energy landscape The Boltzmann machine Constraint satisfaction Summary ADAPTIVE RESONANCE THEORY Introduction Adaptive resonance theory (ART) Architecture and operation ART algorithm Training the ART network Classification Conclusion Summary of ART ASSOCIATIVE MEMORY Standard computer memory Implementing associative memory Implementation in RAMs RAMs and N-tupling Willshaw's associative net The ADAM system Kanerva's sparse distributed memory Bidirectional associative memories Conclusion Summary INTO THE LOOKING GLASS Overview Hardware and software implementations Optical computing Optical computing and neural networks INDEX

Neural networks for self-learning control systems

Abstract: It is shown that a neural network can learn of its own accord to control a nonlinear dynamic system. An emulator, a multilayered neural network, learns to identify the system's dynamic characteristics. The controller, another multilayered neural network, next learns to control the emulator. The self-trained controller is then used to control the actual dynamic system. The learning process continues as the emulator and controller improve and track the physical process. An example is given to illustrate these ideas. The 'truck backer-upper', a neural network controller that steers a trailer truck while the truck is backing up to a loading dock, is demonstrated. The controller is able to guide the truck to the dock from almost any initial position. The technique explored should be applicable to a wide variety of nonlinear control problems. >

Layered neural networks with Gaussian hidden units as universal approximations

Abstract: A neural network with a single layer of hidden units of gaussian type is proved to be a universal approximator for real-valued maps defined on convex, compact sets of Rn.

A neural network model for bankruptcy prediction

Abstract: A neural network model is developed for prediction of bankruptcy, and it is tested using financial data from various companies. The same set of data is analyzed using a more traditional method of bankruptcy prediction, multivariate discriminant analysis. A comparison of the predictive abilities of both the neural network and the discriminant analysis method is presented. The results show that neural networks might be applicable to this problem

Genetic algorithms and neural networks: optimizing connections and connectivity

Abstract: Genetic algorithms are a robust adaptive optimization method based on biological principles. A population of strings representing possible problem solutions is maintained. Search proceeds by recombining strings in the population. The theoretical foundations of genetic algorithms are based on the notion that selective reproduction and recombination of binary strings changes the sampling rate of hyperplanes in the search space so as to reflect the average fitness of strings that reside in any particular hyperplane. Thus, genetic algorithms need not search along the contours of the function being optimized and tend not to become trapped in local minima. This paper is an overview of several different experiments applying genetic algorithms to neural network problems. These problems include 1. (1) optimizing the weighted connections in feed-forward neural networks using both binary and real-valued representations, and 2. (2) using a genetic algorithm to discover novel architectures in the form of connectivity patterns for neural networks that learn using error propagation. Future applications in neural network optimization in which genetic algorithm can perhaps play a significant role are also presented.

Designing Neural Networks Using Genetic Algorithms with Graph Generation System

Abstract: We present a new method of designing neural networks using the genetic algorithm. Recently there have been several reports claiming attempts to design neural networks using genetic algorithms were successful. However, these methods have a problem in scalability, i.e., the convergence characteristic degrades significantly as the size of the network increases. This is because these methods employ direct mapp ing of chromosomes into network connectivities. As an alternative approach, we propose a graph grammatical encoding that will encode graph generation grammar to the chromosome so that it generates more regular connectivity patterns with shorter chromosome length. Experimental results support that our new scheme provides magnitude of speedup in convergence of neural network design and exhibits desirable scaling property.

A simple procedure for pruning back-propagation trained neural networks

Abstract: The sensitivity of the global error (cost) function to the inclusion/exclusion of each synapse in the artificial neural network is estimated. Introduced are shadow arrays which keep track of the incremental changes to the synaptic weights during a single pass of back-propagating learning. The synapses are then ordered by decreasing sensitivity numbers so that the network can be efficiently pruned by discarding the last items of the sorted list. Unlike previous approaches, this simple procedure does not require a modification of the cost function, does not interfere with the learning process, and demands a negligible computational overhead. >