Showing papers on "Deep learning published in 1990"

Backpropagation through time: what it does and how to do it

Abstract: Basic backpropagation, which is a simple method now being widely used in areas like pattern recognition and fault diagnosis, is reviewed. The basic equations for backpropagation through time, and applications to areas like pattern recognition involving dynamic systems, systems identification, and control are discussed. Further extensions of this method, to deal with systems other than neural networks, systems involving simultaneous equations, or true recurrent networks, and other practical issues arising with the method are described. Pseudocode is provided to clarify the algorithms. The chain rule for ordered derivatives-the theorem which underlies backpropagation-is briefly discussed. The focus is on designing a simpler version of backpropagation which can be translated into computer code and applied directly by neutral network users. >

Single-layer learning revisited: a stepwise procedure for building and training a neural network

Abstract: A stepwise procedure for building and training a neural network intended to perform classification tasks, based on single layer learning rules, is presented. This procedure breaks up the classification task into subtasks of increasing complexity in order to make learning easier. The network structure is not fixed in advance: it is subject to a growth process during learning. Therefore, after training, the architecture of the network is guaranteed to be well adapted for the classification problem.

Designing Neural Networks Using Genetic Algorithms with Graph Generation System

Abstract: We present a new method of designing neural networks using the genetic algorithm. Recently there have been several reports claiming attempts to design neural networks using genetic algorithms were successful. However, these methods have a problem in scalability, i.e., the convergence characteristic degrades significantly as the size of the network increases. This is because these methods employ direct mapp ing of chromosomes into network connectivities. As an alternative approach, we propose a graph grammatical encoding that will encode graph generation grammar to the chromosome so that it generates more regular connectivity patterns with shorter chromosome length. Experimental results support that our new scheme provides magnitude of speedup in convergence of neural network design and exhibits desirable scaling property.

CMAS: an associative neural network alternative to backpropagation

Abstract: The CMAC (cerebellar model arithmetic computer) neural network, an alternative to backpropagated multilayer networks, is described. The following advantages of CMAC are discussed: local generalization, rapid algorithmic computation based on LMS (least-mean-square) training, incremental training, functional representation, output superposition, and a fast practical hardware realization. A geometrical explanation of how CMAC works is provided, and applications in robot control, pattern recognition, and signal processing are briefly described. Possible disadvantages of CMAC are that it does not have global generalization and that it can have noise due to hash coding. Care must be exercised (as with all neural networks) to assure that a low error solution will be learned. >

Entropy nets: from decision trees to neural networks

Abstract: How the mapping of decision trees into a multilayer neural network structure can be exploited for the systematic design of a class of layered neural networks, called entropy nets (which have far fewer connections), is shown. Several important issues such as the automatic tree generation, incorporation of the incremental learning, and the generalization of knowledge acquired during the tree design phase are discussed. A two-step methodology for designing entropy networks is presented. The methodology specifies the number of neurons needed in each layer, along with the desired output, thereby leading to a faster progressive training procedure that allows each layer to be trained separately. Two examples are presented to show the success of neural network design through decision-tree mapping. >

Parallel Processing in Neural Systems and Computers

Abstract: General Introduction. Development and Application of Parallel Computers. New Concepts of Information Processing in the Brain. Parallel Processing in the Visual System. Self-Organization and Learning in Neural Networks. Evaluation of Artificial Neural Networks. Hardware and Software Simulators for Neural Networks. Neural Networks for Visual Pattern Recognition. Neural Networks for Auditory Pattern Recognition. Neural Networks for Motor Control. Selected Applications for Neural Networks. Parallel Processing in Artificial Intelligence. Optical and Molecular Computing. Indices. References. Invited Papers: Neuroinformatics and Cybernetics (G. Hauske). Processing of Figure and Background Motion in the Visual System of the Fly (W. Reichardt). Internal Representations and Associative Memory (T. Kohonen). Neural Network Models for Visual Pattern Recognition (K. Fukushima). Adaptive Resonance Theory: Neural Network Architectures for Self-Organizing Pattern Recognition (G.A. Carpenter, S. Grossberg). Recognition of Patterns and Movement Patterns by a Synergetic Computer (H. Haken). Parallel Process Interfaces to Knowledge Systems (K.H. Becks, W. Burgard, A.B. Cremers, A. Hemker, A. Ultsch). Apparent Motion and Other Mysteries (J.A. Feldman). Novel Logic and Architectures for Molecular Computing (J.R. Barker). New Laser Techniques for Quasi-Molecular Storage (D. Haarer). Learning in Optical Neural Networks (D. Psaltis, D. Brady, K. Hsu).

Mathematical foundations of neurocomputing

Abstract: An attempt is made to establish a mathematical theory that shows the intrinsic mechanisms, capabilities, and limitations of information processing by various architectures of neural networks. A method of statistically analyzing one-layer neural networks is given, covering the stability of associative mapping and mapping by totally random networks. A fundamental problem of statistical neurodynamics is considered in a way that is different from the spin-glass approach. A dynamic analysis of associative memory models and a general theory of neural learning, in which the learning potential function plays a role, are given. An advanced theory of learning and self-organization is proposed, covering backpropagation and its generalizations as well as the formation of topological maps and neural representations of information. >

Dynamic recurrent neural networks

Abstract: We survey learning algorithms for recurrent neural networks with hidden units and attempt to put the various techniques into a common framework. We discuss fixpoint learning algorithms, namely recurrent backpropagation and deterministic Boltzmann Machines, and non-fixpoint algorithms, namely backpropagation through time, Elman's history cutoff nets, and Jordan's output feedback architecture. Forward propagation, an online technique that uses adjoint equations, is also discussed. In many cases, the unified presentation leads to generalizations of various sorts. Some simulations are presented, and at the end, issues of computational complexity are addressed.

Self-organizing optical neural network for unsupervised learning

Abstract: One of the features in neural computing must be the ability to adapt to a changeable environment and to recognize unknown objects. This paper deals with an adaptive optical neural network using Kohonen's self-organizing feature map algorithm for unsupervised learning. A compact optical neural network of 64 neurons using liquid crystal televisions is used for this study. To test the performance of the self-organizing neural network, experimental demonstrations and computer simulations are provided. Effects due to unsupervised learning parameters are analyzed. We show that the optical neural network is capable of performing both unsupervised learning and pattern recognition operations simultaneously, by setting two matching scores in the learning algorithm. By using a slower learning rate, the construction of the memory matrix becomes more organized topologically. Moreover, the introduction of forbidden regions in the memory space enables the neural network to learn new patterns without erasing the old ones.

Back-propagation heuristics: a study of the extended delta-bar-delta algorithm

Abstract: An investigation is presented of an extension, proposed by A.A. Minai and R.D. Williams (Proc. Int. Joint Conf. on Neural Networks, vol.1, p.676-79, Washington, DC, 1990), to an algorithm for training neural networks in real-valued, continuous approximation domains. Specifically, the most effective aspects of the proposed extension are isolated. It is found that while momentum is particularly useful for the delta-bar-delta algorithm, it cannot be used conveniently because of sensitivity considerations. It is also demonstrated that by using more subtle versions of the algorithm, the advantages of momentum can be retained without any significant drawbacks

Theory of morphological neural networks

Abstract: The theory of classical artificial neural networks has been used to solve pattern recognition problems in image processing that is different from traditional pattern recognition approaches. In standard neural network theory, the first step in performing a neural network calculation involves the linear operation of multiplying neural values by their synaptic strengths and adding the results. Thresholding usually follows the linear operation in order to provide for non-linearity of the network. This paper presents the fundamental theory for a morphological neural network which, instead of multiplication and summation, uses the non-linear operation of addition and maximum. Several basic applications which are distinctly different from pattern recognition techniques are given, including a net which performs a sieving algorithm.

Sunspot numbers forecasting using neural networks

Abstract: A recurrent connectionist network has been designed to model sunspot data. The network architecture, sunspot data, and statistical models are described, and experimental results are provided. This preliminary experimental work shows that the network can produce competitive prediction results that compare with those of traditional autoregressive models. The method is not problem specific and could be applied to other problems in dynamical system modeling, recognition, prediction, and control fields. >

Clustered neural networks

Abstract: A plurality of neural networks are coupled to an output neural network, or judge network, to form a clustered neural network. Each of the plurality of clustered networks comprises a supervised learning rule back-propagated neural network. Each of the clustered neural networks are trained to perform substantially the same mapping function before they are clustered. Following training, the clustered neural network computes its output by taking an "average" of the outputs of the individual neural networks that make up the cluster. The judge network combines the outputs of the plurality of individual neural networks to provide the output from the entire clustered network. In addition, the output of the judge network may be fed back to each of the individual neural networks and used as a training input thereto, in order to provide for continuous training. The use of the clustered network increases the speed of learning and results in better generalization. In addition, clustering multiple back-propagation networks provides for increased performance and fault tolerance when compared to a single unclustered network having substantially the same computational complexity. The present invention may be used in applications that are amenable to neural network solutions, including control and image processing applications. Clustering of the networks also permits the use of smaller networks and provides for improved performance. The clustering of multiple back-propagation networks provides for synergy that improves the properties of the clustered network over a comparably complex non-clustered network.

Neural network models in artificial intelligence

Abstract: Issues in neural network modelling neural network models for learning and relaxation production systems and expert systems knowledge representation speech recognition and syntheses: comparing algorithms for speech recognition visual perception and pattern recognition language understanding.

Complexity Theory of Neural Networks and Classification Problems

Abstract: Multilayered, feedforward neural network techniques have been proposed for a variety of classification and recognition problems ranging from speech to sonar signal processing problems. It is generally assumed that the underlying application does not need to be modeled very much and that an artificial neural network solution can be obtained instead by training from empirical data with little or no a priori information about the application. We argue that the right network architecture is fundamental for a good solution to exist and the class of network architectures forms a basis for a complexity theory of classification problems. An abstraction of this notion of complexity leads to ideas similar to Kolmogorov's minimum length description criterion, entropy and k-widths. We will present some basic results on this measure of complexity. From this point of view, artificial neural network solutions to real engineering problems may not ameliorate the difficulties of classification problems, but rather obscure and postpone them. In particular, we doubt that the design of neural networks for solving interesting nontrivial engineering problems will be any easier than other large scale engineering design problems (such as in aerodynamics and semiconductor device modeling).

Neural networks and their applications in component design data retrieval

Abstract: Neural networks have gained increased importance in the past few years. One of the basic characteristics of neural networks is the property of associative memory. In this paper we study the possibility of using the ideas of neural networks and associative memory in the manufacturing domain, with specific reference to design data retrieval in group technology. A two-layer feed-forward perceptron with backpropagation is simulated on a Vax-8550 to train example parts. The complete scheme along with the simulation results are explained and future directions indicated.

Fixed-weight networks can learn

Abstract: A theorem describing how fixed-weight recurrent neural networks can approximate adaptive-weight learning algorithms is proved. The theorem applies to most networks and learning algorithms currently in use. It is concluded from the theorem that a system which exhibits learning behavior may exhibit no synaptic weight modifications. This idea is demonstrated by transforming a backward error propagation network into a fixed-weight system

Artificial neural networks: concept learning

Abstract: Learning is one of the most important features of artificial neural networks (ANN). This volume is a representative overview of the most important ANN learning techniques. The topics covered include connectionist learning procedures, dynamic connections in neural networks, connectionist recruitment

SLONN: a simulation language for modeling of neural networks

Abstract: This paper presents a general purpose Simulation Language for modeling Of Neural Networks (SLONN) which has been implemented in our laboratory. Based on a new neuron model, SLONN can represent both spatial and temporal summation of a single neuron and synaptic plasticity. By introducing fork to describe a connection pattern between neurons and by using repetition connec tion, module type and module array to specify large networks, SLONN can be used to specify both small and large neural networks effectively. This language is distinguished by its hierarchical organiza tion, which makes it possible to catch very general aspects at higher levels as well as very specific properties at lower levels. As an example to demonstrate some features of SLONN, we have modeled the habitua tion and sensitization behaviors in Aplysia.

Generalization of the backpropagation neural network learning algorithm to permit complex weights

Abstract: The backpropagation neural network learning algorithm is generalized to include complex-valued interconnections for possible optical implementations.

Evolving a neural network

Abstract: Although artificial neural networks have been shown to be effective in the computation of solutions to difficult problems a general theory has not yet been developed to provide guidance in their design and implementation. Genetic algorithms have also been shown to be effective in evolving solutions to optimization problems which involve objective functions that are not 'nice'. The approach presented here is the utilization of genetic algorithms to evolve the number of neurons in an artificial neural network, the weights of their interconnects, and the interconnect structure itself. With this approach, no a priori assumptions about interconnect structure, weights, number of layers. or to which neurons the inputs or outputs are connected need to be made. A combined neural network evaluation and genetic algorithm evaluation program has been written in C on a Sun workstation. The method has been successfully applied to the 9*9 bit character recognition problem. >

Artificial neural networks for automatic target recognition

Abstract: This paper will review recent advances in the applications of artificial neural network technology to problems in automatic target recognition. The application of feedforward networks for segmentation feature extraction and classification of targets in Forward Looking Infrared (FLIR) and laser radar range scenes will be presented. Biologically inspired Gabor functions will be shown to be a viable alternative to heuristic image processing techniques for segmentation. The use of local transforms such as the Gabor transform fed into a feedforward network is proposed as an architecture for neural based segmentation. Techniques for classification of segmented blobs will be reviewed along with neural network procedures for determining relevant features. A brief review of previous work on comparing neural network based classifiers to conventional Bayesian and K-nearest neighbor techniques will be presented. Results from testing several alternative learning algorithms for these neural network classifiers are presented. A technique for fusing information from multiple sensors using neural networks is presented and conclusions are made. 1

Neural networks for voiced/unvoiced speech classification

Abstract: The results of designing, training, and testing a neural network for the voiced/unvoiced (V/UV) speech classification problem are described A feedforward multilayer backpropagation network was used with six input, ten internal, and two output nodes-for a binary decision The six features are common and easily computed Training was done with 72 frames from two speakers Testing was done with 479 frames from four speakers and resulted in a total of two errors (04%) Thus, a small neural network performs well on the V/UV problem >

Adjoint-Functions and Temporal Learning Algorithms in Neural Networks

Abstract: The development of learning algorithms is generally based upon the minimization of an energy function. It is a fundamental requirement to compute the gradient of this energy function with respect to the various parameters of the neural architecture, e.g., synaptic weights, neural gain, etc. In principle, this requires solving a system of nonlinear equations for each parameter of the model, which is computationally very expensive. A new methodology for neural learning of time-dependent nonlinear mappings is presented. It exploits the concept of adjoint operators to enable a fast global computation of the network's response to perturbations in all the systems parameters. The importance of the time boundary conditions of the adjoint functions is discussed. An algorithm is presented in which the adjoint sensitivity equations are solved simultaneously (i.e., forward in time) along with the nonlinear dynamics of the neural networks. This methodology makes real-time applications and hardware implementation of temporal learning feasible.