Showing papers on "Recurrent neural network published in 1991"

Time series forecasting using neural networks vs. Box- Jenkins methodology

Abstract: We discuss the results of a comparative study of the performance of neural networks and conventional methods in forecasting time series Our work was initially inspired by previously published work

Gesture recognition using recurrent neural networks

Abstract: A gesture recognition method for Japanese sign language is presented. We have developed a posture recognition system using neural networks which could recognize a finger alphabet of 42 symbols. We then developed a gesture recognition system where each gesture specifies a word. Gesture recognition is more difficult than posture recognition because it has to handle dynamic processes. To deal with dynamic processes we use a recurrent neural network. Here, we describe a gesture recognition method which can recognize continuous gesture. We then discuss the results of our research.

The Induction of Dynamical Recognizers

Abstract: A higher order recurrent neural network architecture learns to recognize and generate languages after being “trained” on categorized exemplars. Studying these networks from the perspective of dynamical systems yields two interesting discoveries: First, a longitudinal examination of the learning process illustrates a new form of mechanical inference: Induction by phase transition. A small weight adjustment causes a “bifurcation” in the limit behavior of the network. This phase transition corresponds to the onset of the network's capacity for generalizing to arbitrary-length strings. Second, a study of the automata resulting from the acquisition of previously published training sets indicates that while the architecture is not guaranteed to find a minimal finite automaton consistent with the given exemplars, which is an NP-Hard problem, the architecture does appear capable of generating non-regular languages by exploiting fractal and chaotic dynamics. I end the paper with a hypothesis relating linguistic generative capacity to the behavioral regimes of non-linear dynamical systems.

Evolving neural network controllers for unstable systems

Abstract: The author describes how genetic algorithms (GAs) were used to create recurrent neural networks to control a series of unstable systems. The systems considered are variations of the pole balancing problem: network controllers with two, one, and zero inputs, variable length pole, multiple poles on one cart, and a jointed pole. GAs were able to quickly evolve networks for the one- and two-input pole balancing problems. Networks with zero inputs were only able to valance poles for a few seconds of simulated time due to the network's inability to maintain accurate estimates of their position and pole angle. Also, work in progress on a two-legged walker is briefly described. >

Fault diagnosis in complex chemical plants using artificial neural networks

Abstract: We illustrate a neural network approach to fault detection and diagnosis in a large complex chemical plant. We demonstrate the ability of a neural network to learn nonlinear mappings in the presence of noisy inputs. We also show that neural network models can exhibit the rule-following behavior of knowledge-based expert systems without containing any explicit representations of the rules. We conclude that, given a valid database, fault detection and diagnosis is a promising area for the application of artificial neural networks in industry

Speech recognition using time-warping neural networks

Abstract: The author proposes a time-warping neural network (TWNN) for phoneme-based speech recognition. The TWNN is designed to accept phonemes with arbitrary duration, whereas conventional phoneme recognition networks have a fixed-length input window. The purpose of this network is to cope with not only variability of phoneme duration but also time warping in a phoneme. The proposed network is composed of several time-warping units which each have a time-warping function. The TWNN is characterized by time-warping functions embedded between the input layer and the first hidden layer in the network. The proposed network demonstrates higher phoneme recognition accuracy than a baseline recognizer based on conventional feedforward neural networks and linear time alignment. The recognition accuracy is even higher than that achieved with discrete hidden Markov models. >

Recurrent network model of the neural mechanism of short-term active memory

Abstract: Two decades of single unit recording in monkeys performing short-term memory tasks has established that information can be stored as sustained neural activity. The mechanism of this information storage is unknown. The learning-based model described here demonstrates that a mechanism using only the dynamic activity in recurrent networks is sufficient to account for the observed phenomena. The temporal activity patterns of neurons in the model match those of real memory-associated neurons, while the model's gating properties and attractor dynamics provide explanations for puzzling aspects of the experimental data.

Optimization using neural networks

Abstract: The design of feedback (or recurrent) neural networks to produce good solutions to complex optimization problems is discussed. The theoretical basis for applying neural networks to optimization problems is reviewed, and a design rule that serves as a primitive for constructing a wide class of constraints is introduced. The use of the design rule is illustrated by developing a neural network for producing high-quality solutions to a probabilistic resource allocation task. The resulting neural network has been simulated on a high-performance parallel processor that has been optimized for neural network simulation. >

Recurrent correlation associative memories

Abstract: A model for a class of high-capacity associative memories is presented. Since they are based on two-layer recurrent neural networks and their operations depend on the correlation measure, these associative memories are called recurrent correlation associative memories (RCAMs). The RCAMs are shown to be asymptotically stable in both synchronous and asynchronous (sequential) update modes as long as their weighting functions are continuous and monotone nondecreasing. In particular, a high-capacity RCAM named the exponential correlation associative memory (ECAM) is proposed. The asymptotic storage capacity of the ECAM scales exponentially with the length of memory patterns, and it meets the ultimate upper bound for the capacity of associative memories. The asymptotic storage capacity of the ECAM with limited dynamic range in its exponentiation nodes is found to be proportional to that dynamic range. Design and fabrication of a 3-mm CMOS ECAM chip is reported. The prototype chip can store 32 24-bit memory patterns, and its speed is higher than one associative recall operation every 3 mu s. An application of the ECAM chip to vector quantization is also described. >

Interpretation of Artificial Neural Networks: Mapping Knowledge-Based Neural Networks into Rules

Abstract: We propose and empirically evaluate a method for the extraction of expert-comprehensible rules from trained neural networks. Our method operates in the context of a three-step process for learning that uses rule-based domain knowledge in combination with neural networks. Empirical tests using real-worlds problems from molecular biology show that the rules our method extracts from trained neural networks: closely reproduce the accuracy of the network from which they came, are superior to the rules derived by a learning system that directly refines symbolic rules, and are expert-comprehensible.

Integrating time alignment and neural networks for high performance continuous speech recognition

Abstract: The authors describe two systems in which neural network classifiers are merged with dynamic programming (DP) time alignment methods to produce high-performance continuous speech recognizers. One system uses the connectionist Viterbi-training (CVT) procedure, in which a neural network with frame-level outputs is trained using guidance from a time alignment procedure. The other system uses multi-state time-delay neural networks (MS-TDNNs), in which embedded DP time alignment allows network training with only word-level external supervision. The CVT results on the, TI Digits are 99.1% word accuracy and 98.0% string accuracy. The MS-TDNNs are described in detail, with attention focused on their architecture, the training procedure, and results of applying the MS-TDNNs to continuous speaker-dependent alphabet recognition: on two speakers, word accuracy is respectively 97.5% and 89.7%. >

Extracting and Learning an Unknown Grammar with Recurrent Neural Networks

Abstract: Simple second-order recurrent networks are shown to readily learn small known regular grammars when trained with positive and negative strings examples. We show that similar methods are appropriate for learning unknown grammars from examples of their strings. The training algorithm is an incremental real-time, recurrent learning (RTRL) method that computes the complete gradient and updates the weights at the end of each string. After or during training, a dynamic clustering algorithm extracts the production rules that the neural network has learned. The methods are illustrated by extracting rules from unknown deterministic regular grammars. For many cases the extracted grammar outperforms the neural net from which it was extracted in correctly classifying unseen strings.

An overview of neural networks for control

Abstract: The potential of neural network controllers is considered in general terms Some specific methods are then examined They are supervised control, direct inverse control, neural adaptive control, backpropagation of utility, and adaptive critic methods >

Neural Networks: Concepts, Applications, and Implementations

Abstract: An algebraic deterministic theory of neural nets pattern recognition and neuro engineering self-organizing neural network architectures for adaptive pattern recognition and robotics neural network applications to speech signal/image processing and underatanding with neural nwtworks networks for learning - a view from the theory of approximations of functions analog electronic neural networks for pattern recognition applications VLSI architectures for neural networks mapping the neural networks on the honeycomb architecture.

Recurrent neural networks and time series prediction

Abstract: Uses a parametric statistical framework to understand the effect of input representation on performance for nonlinear prediction of time series. In particular, considerations of input representation lead directly to choices between feedforward and recurrent neural networks. It is shown that feedforward networks are nonlinear autoregressive models and that recurrent networks can model a larger class of processes, including nonlinear autoregressive moving average models. For some processes, feedback allows recurrent networks to achieve better predictions than can be made with a feedforward network with a finite number of inputs. The results are confirmed on a problem in power system regional load forecasting. >

Neural network based design of cellular manufacturing systems

Abstract: A neural network based on a competitive learning rule, when trained with the part machine incidence matrix of a large number of parts, classifies the parts and machines into part families and machine cells, respectively. This classification compares well with the classical clustering techniques. The steady state values of the activations and interconnecting strengths enable easier identification of the part families, machine cells, overlapping parts and bottleneck machines. Neural networks are mostly applied by treating them as a blackbox, i.e. the interaction with the environment and the information acquisition and retrieval occurs at the input and the output level of the network. This paper presents an approach where knowledge is extracted from the external and internal structure of the neural network.

Second-order recurrent neural networks for grammatical inference

Abstract: It is shown that a recurrent, second-order neural network using a real-time, feedforward training algorithm readily learns to infer regular grammars from positive and negative string training samples. Numerous simulations which show the effect of initial conditions, training set size and order, and neuron architecture are presented. All simulations were performed with random initial weight strengths and usually converge after approximately a hundred epochs of training. The authors discuss a quantization algorithm for dynamically extracting finite-state automata during and after training. For a well-trained neural net, the extracted automata constitute an equivalence class of state machines that are reducible to the minimal machine of the inferred grammar. It is then shown through simulations that many of the neural net state machines are dynamically stable and correctly classify long unseen strings. >

Recurrent Networks and NARMA Modeling

Abstract: There exist large classes of time series, such as those with nonlinear moving average components, that are not well modeled by feedforward networks or linear models, but can be modeled by recurrent networks. We show that recurrent neural networks are a type of nonlinear autoregressive-moving average (NARMA) model. Practical ability will be shown in the results of a competition sponsored by the Puget Sound Power and Light Company, where the recurrent networks gave the best performance on electric load forecasting.

A multilayer neural network with piecewise-linear structure and back-propagation learning

Abstract: A multilayer neural network which is given a two-layer piecewise-linear structure for every cascaded section is proposed. The neural networks have nonlinear elements that are neither sigmoidal nor of a signum type. Each nonlinear element is an absolute value operator. It is almost everywhere differentiable, which makes back-propagation feasible in a digital setting. Both the feedforward signal propagation and the backward coefficient update rules belong to the class of regular iterative algorithms. This form of neural network specializes in functional approximation and is anticipated to have applications in control, communications, and pattern recognition. >

A structure by which a recurrent neural network can approximate a nonlinear dynamic system

Abstract: It is shown that the structure of the standard recurrent neural network has the capacity to model a broad class of nonlinear dynamic systems. The key result is that the structure of the recurrent neural network permits the internal formation of a single hidden layer/linear output layer feedforward neural network to approximate the next system state as a function of the current system state and the inputs. The recurrent nature of the network allows the single weight matrix to serve as both the input and output weight matrices of the internal feedforward network. >

High performance connected digit recognition using maximum mutual information estimation

Abstract: The authors describe the latest development by the speech research group at CRIM (Centre de Recherche Informatique de Montreal) in speaker-independent connected digit recognition, using hidden Markov Models (HMMs) trained with maximum mutual information estimation, in conjunction with connectionist models. The experiments described were all done on the complete adult portion of the 10 kHz speaker-independent TI/NIST connected digit database. The baseline system, using discrete HMMs and maximum likelihood estimation, has a 98.6% word recognition rate and a 96.1% string recognition rate. The authors describe techniques that made it possible to improve greatly the baseline system recognition rate. The 99.3% recognition rate and 98.0% string recognition rate were obtained with a single model per unit using discrete HMMs and recurrent neural networks. Using semi-continuous HMMs with two models per digit (one for male and one for female speakers), a 99.5% word recognition rate and a 98.4% string recognition rate were achieved. >

Neural Networks: Advances and Applications II

Abstract: Learning in the Recurrent Random Neural Network (E. Gelenbe). Generalization Performance of Feed-Forward Neural Networks (S. Shekhar et al.). The Nature of Intracortical Inhibitory Effects (J.A. Reggia et al.). Random Neural Networks with Multiple Classes of Signals (J.-M. Fourneau, E. Gelenbe). The MicroCircuit Associative Memory, muAM: A Biologically Motivated Memory Architecture (C.F. Miles, D. Rogers). Generalised Associative Memory and the Computation of Membership Functions (E. Gelenbe). Layered Neural Network for Stereo Disparity Detection (E. Maeda et al.). Storage and Recognition Methods for the Random Neural Network (M. Mokhtari). Neural Networks for Image Compression (S. Carrato). Autoassociative Memory with the Random Neural Network using Gelenbe's Learning Algorithm (C. Hubert). Minimum Graph Covering with the Random Neural Network Model (E. Gelenbe, F. Batty).

Pattern recognition properties of neural networks

Abstract: Artificial neural networks have been applied largely to solving pattern recognition problems. The authors point out that a firm understanding of the statistical properties of neural nets is important for using them in an effective manner for pattern recognition problems. The author gives an overview of pattern recognition properties for feedforward neural nets, with emphasis on two topics: partitioning of the input space into classes and the estimation of posterior probabilities for each of the classes. >

Identification of Chemical Processes using Recurrent Networks

Abstract: Neurl networks have been widely used in many research areas including nonlinear system identification. In the present study, a recurrent neural network, as an alternative to feed-forward networks, has been used successfully to identify the dynamic behavior of a biological wastewater treatment plant. An approach to deriving the learning algorithm for recurrent networks is discussed. In comparison to a feed-forward network, the recurrent network produces superior results for long-term predictions.

Optimization and training of feedforward neural networks by genetic algorithms

Abstract: The training of feedforward neural networks by backpropagation requires much time-consuming experimentation by the network designer. The authors use the genetic algorithm formalism to optimize network structure and training parameters automatically, so as to allow successful back-propagation learning. Additionally, they describe a method to optimize network weights directly using the genetic algorithm, removing any need for a gradient-descent algorithm such as back-propagation.

Nonlinear prediction of speech signals using memory neuron networks

Abstract: The authors present a feed-forward neural network architecture that can be used for nonlinear autoregressive prediction of multivariate time-series. It uses specialized neurons (called memory neurons) to store past activations of the network in an efficient fashion. The network learns to be a nonlinear predictor of the appropriate order to model temporal waveforms of speech signals. Arrays of such networks can be used to build real-time classifiers of speech sounds. Experiments where memory-neuron networks are trained to predict speech waveforms and sequences of spectral frames are described. Performance of the network for prediction of time-series with minimal a priori assumptions of its statistical properties is shown to be better than linear autoregressive models. >

A study on neural networks for short-term load forecasting

Abstract: A study is made on the application of the artificial neural network (ANN) method to forecast the short-term load for a large power system. The load has two distinct patterns: weekday and weekend-day patterns. The weekend-day pattern include Saturday, Sunday, and Monday loads. Three different ANN models are proposed, including two feedforward neural networks and one recurrent neural network. Inputs to the ANN are past loads and the output is the predicted load for a given day. The standard deviation and percent error of each model are compared. >

Recurrent neural network with variable size intermediate layer

Abstract: The present invention is concerned with a signal processing system having a learning function pursuant to the back-propagation learning rule by the neural network, in which the learning rate is dynamically changed as a function of input values to effect high-speed stable learning. The signal processing system of the present invention is so arranged that, by executing signal processing for the input signals by the recurrent network formed by units each corresponding to a neuron, the features of the sequential time series pattern such as voice signals fluctuating on the time axis can be extracted through learning the coupling state of the recurrent network. The present invention is also concerned with a learning processing system adapted to cause the signal processing section formed by a neural network to undergo signal processing pursuant to the back-propagation learning rule, wherein the local minimum state in the course of the learning processing may be avoided by learning the coefficient of coupling strength while simultaneously increasing the number of the unit of the intermediate layer.

Neural networks that teach themselves through genetic discovery of novel examples

Abstract: The authors introduce an active learning paradigm for neural networks. In contrast to the passive paradigm, the learning in the active paradigm is initiated by the machine learner instead of its environment or teacher. The authors present a learning algorithm that uses a genetic algorithm for creating novel examples to teach multilayer feedforward networks. The creative learning networks, based on their own knowledge, discover new examples, criticize and select useful ones, train themselves, and thereby extend their existing knowledge. Experiments on function extrapolation show that the self-teaching neural networks not only reduce the teaching efforts of the human, but the genetically created examples also contribute robustly to the improvement of generalization performance and the interpretation of the connectionist knowledge. >