Showing papers on "Recurrent neural network published in 1990"

Backpropagation through time: what it does and how to do it

Abstract: Basic backpropagation, which is a simple method now being widely used in areas like pattern recognition and fault diagnosis, is reviewed. The basic equations for backpropagation through time, and applications to areas like pattern recognition involving dynamic systems, systems identification, and control are discussed. Further extensions of this method, to deal with systems other than neural networks, systems involving simultaneous equations, or true recurrent networks, and other practical issues arising with the method are described. Pseudocode is provided to clarify the algorithms. The chain rule for ordered derivatives-the theorem which underlies backpropagation-is briefly discussed. The focus is on designing a simpler version of backpropagation which can be translated into computer code and applied directly by neutral network users. >

Non-linear system identification using neural networks

Abstract: Multi-layered neural networks offer an exciting alternative for modelling complex non-linear systems. This paper investigates the identification of discrete-time nonlinear systems using neural networks with a single hidden layer. New parameter estimation algorithms are derived for the neural network model based on a prediction error formulation and the application to both simulated and real data is included to demonstrate the effectiveness of the neural network approach.

Designing Neural Networks Using Genetic Algorithms with Graph Generation System

Abstract: We present a new method of designing neural networks using the genetic algorithm. Recently there have been several reports claiming attempts to design neural networks using genetic algorithms were successful. However, these methods have a problem in scalability, i.e., the convergence characteristic degrades significantly as the size of the network increases. This is because these methods employ direct mapp ing of chromosomes into network connectivities. As an alternative approach, we propose a graph grammatical encoding that will encode graph generation grammar to the chromosome so that it generates more regular connectivity patterns with shorter chromosome length. Experimental results support that our new scheme provides magnitude of speedup in convergence of neural network design and exhibits desirable scaling property.

A simple procedure for pruning back-propagation trained neural networks

Abstract: The sensitivity of the global error (cost) function to the inclusion/exclusion of each synapse in the artificial neural network is estimated. Introduced are shadow arrays which keep track of the incremental changes to the synaptic weights during a single pass of back-propagating learning. The synapses are then ordered by decreasing sensitivity numbers so that the network can be efficiently pruned by discarding the last items of the sorted list. Unlike previous approaches, this simple procedure does not require a modification of the cost function, does not interfere with the learning process, and demands a negligible computational overhead. >

Neural networks for control systems

Abstract: A description is given of 11 papers from the April 1990 special issue on neural networks in control systems of IEEE Control Systems Magazine. The emphasis was on presenting as varied and current a picture as possible of the use of neural networks in control. The papers described cover: the design of associative memories using feedback neural networks; a method to use neural networks to control highly nonlinear systems; the modeling of nonlinear chemical systems using neural networks; the identification of dynamical systems; the comparison of conventional adaptive controllers and neural-network-based controllers; a method to provide adaptive control for nonlinear systems; neural networks and back-propagation; the back-propagation algorithm; the use of trained neural networks to regulate the pitch attitude of an underwater telerobot; the control of mobile robots; and the issues involved in integrating neural networks and knowledge-based systems. >

Entropy nets: from decision trees to neural networks

Abstract: How the mapping of decision trees into a multilayer neural network structure can be exploited for the systematic design of a class of layered neural networks, called entropy nets (which have far fewer connections), is shown. Several important issues such as the automatic tree generation, incorporation of the incremental learning, and the generalization of knowledge acquired during the tree design phase are discussed. A two-step methodology for designing entropy networks is presented. The methodology specifies the number of neurons needed in each layer, along with the desired output, thereby leading to a faster progressive training procedure that allows each layer to be trained separately. Two examples are presented to show the success of neural network design through decision-tree mapping. >

Stock price pattern recognition-a recurrent neural network approach

Abstract: Recurrent neural networks were applied to the recognition of stock patterns, and a method for evaluating the networks was developed. In stock trading, triangle patterns indicate an important clue to the trend of future change in stock prices, but the patterns are not clearly defined by rule-based approaches. From stock-price data for all names of corporations listed in the first section of the Tokyo Stock Exchange, an expert called chart reader extracted 16 triangles. These patterns were divided into two groups, 15 training patterns and one test pattern. Using stock data from the past three years for 16 names, 16 recognition experiments in which the groups were cyclically used were carried out. The experiments revealed that the given test triangle was accurately recognized in 15 out of 16 experiments and that the number of the mismatching patterns was 1.06 per name on the average. A method was developed for evaluating recurrent networks with context transition performances, particularly temporal transition performances. The method for the triangle sequences is applicable to reducing mismatching patterns

Parallel Processing in Neural Systems and Computers

Abstract: General Introduction. Development and Application of Parallel Computers. New Concepts of Information Processing in the Brain. Parallel Processing in the Visual System. Self-Organization and Learning in Neural Networks. Evaluation of Artificial Neural Networks. Hardware and Software Simulators for Neural Networks. Neural Networks for Visual Pattern Recognition. Neural Networks for Auditory Pattern Recognition. Neural Networks for Motor Control. Selected Applications for Neural Networks. Parallel Processing in Artificial Intelligence. Optical and Molecular Computing. Indices. References. Invited Papers: Neuroinformatics and Cybernetics (G. Hauske). Processing of Figure and Background Motion in the Visual System of the Fly (W. Reichardt). Internal Representations and Associative Memory (T. Kohonen). Neural Network Models for Visual Pattern Recognition (K. Fukushima). Adaptive Resonance Theory: Neural Network Architectures for Self-Organizing Pattern Recognition (G.A. Carpenter, S. Grossberg). Recognition of Patterns and Movement Patterns by a Synergetic Computer (H. Haken). Parallel Process Interfaces to Knowledge Systems (K.H. Becks, W. Burgard, A.B. Cremers, A. Hemker, A. Ultsch). Apparent Motion and Other Mysteries (J.A. Feldman). Novel Logic and Architectures for Molecular Computing (J.R. Barker). New Laser Techniques for Quasi-Molecular Storage (D. Haarer). Learning in Optical Neural Networks (D. Psaltis, D. Brady, K. Hsu).

Graph partitioning using annealed neural networks

Abstract: A new algorithm, mean field annealing (MFA), is applied to the graph-partitioning problem. The MFA algorithm combines characteristics of the simulated-annealing algorithm and the Hopfield neural network. MFA exhibits the rapid convergence of the neural network while preserving the solution quality afforded by simulated annealing (SA). The rate of convergence of MFA on graph bipartitioning problems is 10-100 times that of SA, with nearly equal quality of solutions. A new modification to mean-field annealing is also presented which supports partitioning graphs into three or more bins, a problem which has previously shown resistance to solution by neural networks. The temperature-behavior of MFA during graph partitioning is analyzed approximately and shown to possess a critical temperature at which most of the optimization occurs. This temperature is analogous to the gain of the neurons in a neural network and can be used to tune such networks for better performance. The value of the repulsion penalty needed to force MFA (or a neural network) to divide a graph into equal-sized pieces is also estimated. >

Associative memories via artificial neural networks

Abstract: Several design techniques that can be used for continuous-time and discrete-time neural networks to realize associative memories are presented. Associative memory is discussed, and neural network models are presented. Some stability concepts are outlined. The applicability of these techniques is demonstrated by means of specific examples that illustrate strengths and weaknesses. >

Dynamic recurrent neural networks

Abstract: We survey learning algorithms for recurrent neural networks with hidden units and attempt to put the various techniques into a common framework. We discuss fixpoint learning algorithms, namely recurrent backpropagation and deterministic Boltzmann Machines, and non-fixpoint algorithms, namely backpropagation through time, Elman's history cutoff nets, and Jordan's output feedback architecture. Forward propagation, an online technique that uses adjoint equations, is also discussed. In many cases, the unified presentation leads to generalizations of various sorts. Some simulations are presented, and at the end, issues of computational complexity are addressed.

Recursive neural networks for associative memory

Abstract: Principles, Problems and Approaches The Deterministic Approach The Statistical Approach Thermodynamic Extension Higher Order Networks Network Design Bibliography Index

Self-organizing optical neural network for unsupervised learning

Abstract: One of the features in neural computing must be the ability to adapt to a changeable environment and to recognize unknown objects. This paper deals with an adaptive optical neural network using Kohonen's self-organizing feature map algorithm for unsupervised learning. A compact optical neural network of 64 neurons using liquid crystal televisions is used for this study. To test the performance of the self-organizing neural network, experimental demonstrations and computer simulations are provided. Effects due to unsupervised learning parameters are analyzed. We show that the optical neural network is capable of performing both unsupervised learning and pattern recognition operations simultaneously, by setting two matching scores in the learning algorithm. By using a slower learning rate, the construction of the memory matrix becomes more organized topologically. Moreover, the introduction of forbidden regions in the memory space enables the neural network to learn new patterns without erasing the old ones.

Neural Networks In Dynamical Systems

Abstract: Multilayer networks and recurrent neural networks have proved extremely successful in pattern recognition problems as well as in associative learning. In this paper an attempt is made to demonstrate that both types of networks, combined in arbitrary configurations, will find application in complex dynamical systems. Well known results in linear systems theory and their extensions to conventional adaptive control theory are used to suggest models for the identification and control of nonlinear dynamic systems. The use of neural networks in dynamical systems raises many theoretical questions, some of which are discussed in the paper.

Neural Networks in Structural Engineering

Abstract: In the past few years literature on computational civil engineering has concentrated primarily on artificial intelligence (Al) applications involving expert system technology. This article discusses a different Al approach involving neural networks. Unlike their expert system counterparts, neural networks can be trained based on observed information. These systems exhibit a learning and memory capability similar to that of the human brain, a fact due to their simplified modeling of the brain's biological function. This article presents an introduction to neural network technology as it applies to structural engineering applications. Differing network types are discussed. A back-propagation learning algorithm is presented. The article concludes with a demonstration of the potential of the neural network approach. The demonstration involves three structural engineering problems. The first problem involves pattern recognition; the second, a simple concrete beam design; and the third, a rectangular plate analysis. The pattern recognition problem demonstrates a solution which would otherwise be difficult to code in a conventional program. The concrete beam problem indicates that typical design decisions can be made by neural networks. The last problem demonstrates that numerically complex solutions can be estimated almost instantaneously with a neural network.

Method and procedure for neural control of dynamic processes

Abstract: A neural network control based on a general multi-variable nonlinear dynamic model incorporating time delays is disclosed. The inverse dynamics of the process being controlled is learned represented by a multi-layer neural network which is used as a feedforward control to achieve a specified closed loop response under varying conditions. The weights between the layers in the neural network are adjusted during the learning process. The learning process is based on minimizing the combined error between the desired process value and the actual process output and the error between the desired process value and the inverse process neural network output.

Neural network models of sensory integration for improved vowel recognition

Abstract: It is demonstrated that multiple sources of speech information can be integrated at a subsymbolic level to improve vowel recognition. Feedforward and recurrent neural networks are trained to estimate the acoustic characteristics of a vocal tract from images of the speaker's mouth. These estimates are then combined with the noise-degraded acoustic information, effectively increasing the signal-to-noise ratio and improving the recognition of these noise-degraded signals. Alternative symbolic strategies such as direct categorization of the visual signals into vowels are also presented. The performances of these neural networks compare favorably with human performance and with other pattern-matching and estimation techniques. >

Analysis and synthesis techniques for Hopfield type synchronous discrete time neural networks with application to associative memory

Abstract: A qualitative theory for synchronous discrete time Hopfield-type neural networks is established. The authors' objectives are accomplished in two phases. First, they address the analysis of the class of neural networks considered. Next, making use of these results, they develop a synthesis procedure for the class of neural networks considered. In the analysis section, techniques from the theory of large-scale interconnected dynamical systems are used to derive tests for the asymptotic stability of an equilibrium of the neural network. Estimates for the rate at which the trajectories of the network will converge from an initial condition to a final state are presented. In the synthesis section the stability tests from the analysis section are used as constraints to develop a design algorithm for associative memories. The algorithm presented guarantees that each desired memory will be stored as an equilibrium, and that each desired memory will be asymptotically stable. The applicability of these results is demonstrated by means of two specific examples. >

Theory of morphological neural networks

Abstract: The theory of classical artificial neural networks has been used to solve pattern recognition problems in image processing that is different from traditional pattern recognition approaches. In standard neural network theory, the first step in performing a neural network calculation involves the linear operation of multiplying neural values by their synaptic strengths and adding the results. Thresholding usually follows the linear operation in order to provide for non-linearity of the network. This paper presents the fundamental theory for a morphological neural network which, instead of multiplication and summation, uses the non-linear operation of addition and maximum. Several basic applications which are distinctly different from pattern recognition techniques are given, including a net which performs a sieving algorithm.

Combining hidden Markov model and neural network classifiers

Abstract: An architecture for a neural network that implements a hidden Markov model (HMM) is presented. This HMM net suggests integrating signal preprocessing (such as vector quantization) with the classifier. A minimum mean-squared-error training criterion for the HMM/neural net is presented and compared to maximum-likelihood and maximum-mutual-information criteria. The HMM forward-backward algorithm is shown to be the same as the neural net backpropagation algorithm. The implications of probability constraints on the HMM parameters are discussed. Relaxing these constraints allows negative probabilities, equivalent to inhibitory connections. A probabilistic interpretation is given for a network with negative, and even complex-valued, parameters. >

Associative memory in an analog iterated-map neural network.

Abstract: The behavior of an analog neural network with parallel dynamics is studied analytically and numerically for two associative-memory learning algorithms, the Hebb rule and the pseudoinverse rule. Phase diagrams in the parameter space of analog gain \ensuremath{\beta} and storage ratio \ensuremath{\alpha} are presented. For both learning rules, the networks have large ``recall'' phases in which retrieval states exist and convergence to a fixed point is guaranteed by a global stability criterion. We also demonstrate numerically that using a reduced analog gain increases the probability of recall starting from a random initial state. This phenomenon is comparable to thermal annealing used to escape local minima but has the advantage of being deterministic, and therefore easily implemented in electronic hardware. Similarities and differences between analog neural networks and networks with two-state neurons at finite temperature are also discussed.

A learning algorithm to teach spatiotemporal patterns to recurrent neural networks

Abstract: A new supervised learning algorithm is proposed. It teaches spatiotemporal patterns to the recurrent neural network with arbitrary feedback connections. In this method the network with fixed connection weights is run for a given period of time under a given external input and initial condition. Then the weights are changed so that the total error from the time dependent teacher signal in this period is maximally decreased. This algorithm is equivalent to the back propagation method for the recurrent network if the discrete time prescription is adopted. However, continuous time formalism seems suited for temporal processing application.

Sunspot numbers forecasting using neural networks

Abstract: A recurrent connectionist network has been designed to model sunspot data. The network architecture, sunspot data, and statistical models are described, and experimental results are provided. This preliminary experimental work shows that the network can produce competitive prediction results that compare with those of traditional autoregressive models. The method is not problem specific and could be applied to other problems in dynamical system modeling, recognition, prediction, and control fields. >

The ART of adaptive pattern recognition by a self-organizing neural network

Abstract: The adaptive resonance theory (ART) suggests a solution to the stability-plasticity dilemma facing designers of learning systems, namely how to design a learning system that will remain plastic, or adaptive, in response to significant events and yet remain stable in response to irrelevant events. ART architectures are discussed that are neural networks that self-organize stable recognition codes in real time in response to arbitrary sequences of input patterns. Within such an ART architecture, the process of adaptive pattern recognition is a special case of the more general cognitive process of hypothesis discovery, testing, search, classification, and learning. This property opens up the possibility of applying ART systems to more general problems of adaptively processing large abstract information sources and databases. The main computational properties of these ART architectures are outlined and contrasted with those of alternative learning and recognition systems. >

Associative memory in neural networks with binary synapses

Abstract: The simple learning algorithm in the neural network with binary synapses, which take one step for storing one pattern is considered. The resulting model turns out to be palimpsestic, and the number of patterns which can be effectively retrieved is L~N1/2.