Showing papers on "Recurrent neural network published in 1989"

Approximation by superpositions of a sigmoidal function

Abstract: In this paper we demonstrate that finite linear combinations of compositions of a fixed, univariate function and a set of affine functionals can uniformly approximate any continuous function ofn real variables with support in the unit hypercube; only mild conditions are imposed on the univariate function. Our results settle an open question about representability in the class of single hidden layer neural networks. In particular, we show that arbitrary decision regions can be arbitrarily well approximated by continuous feedforward neural networks with only a single internal, hidden layer and any continuous sigmoidal nonlinearity. The paper discusses approximation properties of other possible types of nonlinearities that might be implemented by artificial neural networks.

A learning algorithm for continually running fully recurrent neural networks

Abstract: The exact form of a gradient-following learning algorithm for completely recurrent networks running in continually sampled time is derived and used as the basis for practical algorithms for temporal supervised learning tasks. These algorithms have (1) the advantage that they do not require a precisely defined training interval, operating while the network runs; and (2) the disadvantage that they require nonlocal communication in the network being trained and are computationally expensive. These algorithms allow networks having recurrent connections to learn complex tasks that require the retention of information over time periods having either fixed or indefinite length.

Adaptive pattern recognition and neural networks

Abstract: It's coming again, the new collection that this site has. To complete your curiosity, we offer the favorite adaptive pattern recognition and neural networks book as the choice today. This is a book that will show you even new to old thing. Forget it; it will be right for you. Well, when you are really dying of adaptive pattern recognition and neural networks, just pick it. You know, this book is always making the fans to be dizzy if not to find.

Modelling Brain Function: The World of Attractor Neural Networks

Abstract: From the Publisher: Substantial progress in understanding memory, the learning process, and self-organization by studying the properties of models of neural networks have resulted in discoveries of important parallels between the properties of statistical, nonlinear cooperative systems in physics and neural networks.

Learning state space trajectories in recurrent neural networks

Abstract: Many neural network learning procedures compute gradients of the errors on the output layer of units after they have settled to their final values. We describe a procedure for finding E/wij, where E is an error functional of the temporal trajectory of the states of a continuous recurrent network and wij are the weights of that network. Computing these quantities allows one to perform gradient descent in the weights to minimize E. Simulations in which networks are taught to move through limit cycles are shown. This type of recurrent network seems particularly suited for temporally continuous domains, such as signal processing, control, and speech.

Learning state space trajectories in recurrent neural networks

Abstract: A number of procedures are described for finding delta E/ delta W/sub ij/ where E is an error functional of the temporal trajectory of the states of a continuous recurrent network and w/sub ij/ are the weights of that network. Computing these quantities allows one to perform gradient descent in the weights to minimize E, so these procedures form the kernels of connectionist learning algorithms. Simulations in which networks are taught to move through limit cycles are shown, along with some empirical perturbation sensitivity tests. The author describes a number of elaborations of the basic idea, including mutable time delays and teacher forcing. He includes a complexity analysis of the various learning procedures discussed and analyzed. Temporally continuous recurrent networks seems particularly suited for temporally continuous domains, such as signal processing, control, and speech. >

Neural Networks for System Identification

Abstract: Recent advances in the software and hardware technologies of neural networks have motivated new studies in architecture and applications of these networks. Neural networks have potentially powerful characteristics which can be utilized in the development of our research goal, namely, a true autonomous machine. Machine learning is a major step in this development. This paper presents the results of our recent study on neural-network-based machine learning. Two approaches for learning and identification of dynamical systems are presented. A Hopfield network is used in a new identification structure for learning of time varying and time invariant systems. This time domain approach results in system parameters in terms of activation levels of the network neurons. The second technique, which is in frequency domain, utilizes a set of orthogonal basis functions and Fourier analysis network to construct a dynamic system in terms of its Fourier coefficients. Mathematical formulations of each technique and simulation results of the networks are presented.

An Approach to Combining Explanation-based and Neural Learning Algorithms

Abstract: Machine learning is an area where both symbolic and neural approaches to artificial intelligence have been heavily investigated. However, there has been little research into the synergies achievable by combining these two learning paradigms. A hybrid system that combines the symbolically-oriented explanation-based learning paradigm with the neural backpropagation algorithm is described. In the presented EBL-ANN algorithm, the initial neural network configuration is determined by the generalized explanation of the solution to a specific classification task. This approach overcomes problems that arise when using imperfect theories to build explanations and addresses the problem of choosing a good initial neural network configuration. Empirical results show that the hybrid system more accurately learns a concept than the explanation-based system by itself and learns faster and generalizes better than the neural learning system by itself.

Bayesian neural networks

Abstract: A neural network that uses the basic Hebbian learning rule and the Bayesian combination function is defined. Analogously to Hopfield's neural network, the convergence for the Bayesian neural network that asynchronously updates its neurons' states is proved. The performance of the Bayesian neural network in four medical domains is compared with various classification methods. The Bayesian neural network uses more sophisticated combination function than Hopfield's neural network and uses more economically the available information. The "naive" Bayesian classifier typically outperforms the basic Bayesian neural network since iterations in network make too many mistakes. By restricting the number of iterations and increasing the number of fixed points the network performs better than the naive Bayesian classifier. The Bayesian neural network is designed to learn very quickly and incrementally.

A neural controller

Abstract: Error back-propagation is a method whereby a neural net can learn to control a plant in an autonomous way, without a specific learning stage. The paper presents an evaluation method for it, based on qualitative knowledge of the plant. The field of application of the method is specified. The proposed method is applied to three different problems. These three simulations investigate the possibility of online learning with dynamic targets. The results show that dynamic control based on the proposed learning scheme is possible for a neural network. >

BPS: a learning algorithm for capturing the dynamic nature of speech

Abstract: A novel backpropagation learning algorithm for a particular class of dynamic neural networks in which some units have a local feedback is proposed. Hence these networks can be trained to respond to sequences of input patterns. This algorithm has the same order of space and time requirements as backpropagation applied to feedforward networks. The authors present experimental results and comparisons with a speech recognition problem. >

A learning algorithm for analog, fully recurrent neural networks

Abstract: A learning algorithm for recurrent neural networks is derived. This algorithm allows a network to learn specified trajectories in state space in response to various input sequences. The network dynamics are described by a system of coupled differential equations that specify the continuous change of the unit activities and weights over time. The algorithm is nonlocal, in that a change in the connection weight between two units may depend on the values for some of the weights between different units. However, the operation of a learned network (fixed weights) is local. If the network units are specified to behave like electronic amplifiers, then an analog implementation of the learned network is straightforward. An example demonstrates the use of the algorithm in a completely connected network of four units. The network creates a limit cycle attractor in order to perform the specified task. >

Rapid training of higher-order neural networks for invariant pattern recognition

Abstract: The authors demonstrate a second-order neural network that has learned to distinguish between two objects, regardless of their size or translational position, after being trained on only one view of each object. Using an image size of 16*16 pixels, the training took less than 1 min of run time on a Sun 3 workstation. A recognition accuracy of 100% was achieved by the resulting network for several test-object pairs, including the standard T-C problem, for any translational position and over a scale factor of five. The second-order network takes advantage of known relationships between input pixels to build invariance into the network architecture. The use of a third-order neural network to achieve simultaneous rotation, scale, and position invariance is described. Because of the high level of invariance and rapid, efficient training, initial results show higher order neural networks to be vastly superior to multilevel first-order networks trained by backpropagation for applications where invariant pattern recognition is required. >

Shift-invariant, multi-category phoneme recognition using Kohonen's LVQ2

Abstract: The authors describe a shift-tolerant neural network architecture for phoneme recognition. The system is based on LVQ2, an algorithm which pays close attention to approximating the optimal Bayes decision line in a discrimination task. Recognition performances in the 98-99% correct range were obtained for LVQ2 networks aimed at speaker-dependent recognition of phonemes in small but ambiguous Japanese phonemic classes. A correct recognition rate of 97.7% was achieved by a single, larger LVQ2 network covering all Japanese consonants. These recognition results are at least as high as those obtained in the time delay neural network system and suggest that LVQ2 could be the basis for a successful speech recognition system. >

Neural Network Models In Simulation: A Comparison With Traditional Modeling Approaches

Abstract: Neural models are enjoying a resurgence in systems research primarily due to a general interest in the connectionist approach to modeling in artificial intelligence and to the availability of faster and cheaper hardware on which neural net simulations can be executed. We have experimented with using a multi-layer neural network model as a simulation model for a basic ballistics model. In an effort to evaluate the efficiency of the neural net implementation for simulation modeling, we have compared its performance with traditional methods for geometric data fitting such as linear regression and surface response methods. Both of the latter approaches are standard features in many statistical software packages. We have found that the neural net model appears to be inadequate in most respects and we hypothesize that accuracy problems arise, primarily, because the neural network model does not capture the system structure characteristic of all physical models. We discuss the experimental procedure, issues and problems, and finally consider possible future research directions.

Handwritten numeral recognition by multilayered neural network with improved learning algorithm

Abstract: Pattern recognition using multilayered feedforward neural networks is described. The backpropagation learning algorithm for multilayered neural networks was investigated. Results showed that there is a state in which neural networks can learn no more patterns, in spite of there being large errors. The learning algorithm was improved to avoid this problem. In order to estimate the multilayered neural network's ability for pattern recognition, an experiment was carried out on handwritten numeral recognition. Three kinds of neural networks were used. One is a basic multilayered neural network in which each hidden unit is connected to all input units. Another has each hidden unit connected to input units in a local area of a character. The last is a neural network into which feature vectors to be extracted from characters are input. Recognition rates achieved are 98.3%, 98.8%, and 99.1%, respectively. >

Neural networks for high-storage content-addressable memory: VLSI circuit and learning algorithm

Abstract: An implementation of a VLSI fully interconnected neural network with only two binary memory points per synapse is described. The small area of single synaptic cells allows implementation of neural networks with hundreds of neurons. Classical learning algorithms like the Hebb's rule show a poor storage capacity, especially in VLSI neural networks where the range of the synapse weights is limited by the number of memory points contained in each connection; an algorithm for programming a Hopfield neural network as a high-storage content-addressable memory is proposed. The storage capacity obtained with this algorithm is very promising for pattern-recognition applications. >

A subgrouping strategy that reduces complexity and speeds up learning in recurrent networks

Abstract: An algorithm, called RTRL, for training fully recurrent neural networks has recently been studied by Williams and Zipser (1989a, b). Whereas RTRL has been shown to have great power and generality, it has the disadvantage of requiring a great deal of computation time. A technique is described here for reducing the amount of computation required by RTRL without changing the connectivity of the networks. This is accomplished by dividing the original network into subnets for the purpose of error propagation while leaving them undivided for activity propagation. An example is given of a 12-unit network that learns to be the finite-state part of a Turing machine and runs 10 times faster using the subgrouping strategy than the original algorithm.

Neural network and system

Abstract: Neural network systems (100) with learning and recall are applied to clustered multiple-featured data (122,124,126) and analog data

Comparing different neural network architectures for classifying handwritten digits

Abstract: An evaluation is made of several neural network classifiers, comparing their performance on a typical problem, namely handwritten digit recognition. For this purpose, the authors use a database of handwritten digits, with relatively uniform handwriting styles. The authors propose a novel way of organizing the network architectures by training several small networks so as to deal separately with subsets of the problem, and then combining the results. This approach works in conjunction with various techniques including: layered networks with one or several layers of adaptive connections, fully connected recursive networks, ad hoc networks with no adaptive connections, and architectures with second-degree polynomial decision surfaces. >

Non-Linear Systems Identification Using Neural Networks

Abstract: Multi-layered neural networks offer an exciting alternative for modelling complex non-linear systems This paper investigates the identification of discrete-time non-linear systems using neural networks with a single hidden layer New parameter estimation algorithms are derived for the neural network model based on a prediction error formulation and the application to both simulated and real data is included to demonstrate the effectiveness of the neural network approach

A modular neural network architecture for sequential paraphrasing of script-based stories

Abstract: Sequential recurrent neural networks have been applied to a fairly high-level cognitive task, i.e. paraphrasing script-based stories. Using hierarchically organized modular subnetworks, which are trained separately and in parallel, the complexity of the task is reduced by effectively dividing it into subgoals. The system uses sequential natural language input and output and develops its own I/O representations for the words. The representations are stored in an external global lexicon and are adjusted in the course of training by all four subnetworks simultaneously, according to the FGREP-method. By concatenating a unique identification with the resulting representation, an arbitrary number of instances of the same word type can be created and used in the stories. The system is able to produce a fully expanded paraphrase of the story from only a few sentences, i.e. the unmentioned events are inferred. The word instances are correctly bound to their roles, and simple plausible inferences of the variable content of the story are made in the process. >

Speech dynamics and recurrent neural networks

Abstract: Recently, connectionist models have been recognized as an interesting alternative tool to hidden Markov models for speech recognition. Their main property lies in their combination of good discriminating power and the ability to capture input-output relations. They have also been proved useful in dealing with statistical data. However, the serial aspect remains difficult to handle in that kind of model, and several authors have proposed original architectures to deal with this problem. This study establishes links among them and compares their respective advantages. Relations with hidden Markov models are explained. >