Showing papers on "Recurrent neural network published in 2009"

A Novel Connectionist System for Unconstrained Handwriting Recognition

Abstract: Recognizing lines of unconstrained handwritten text is a challenging task. The difficulty of segmenting cursive or overlapping characters, combined with the need to exploit surrounding context, has led to low recognition rates for even the best current recognizers. Most recent progress in the field has been made either through improved preprocessing or through advances in language modeling. Relatively little work has been done on the basic recognition algorithms. Indeed, most systems rely on the same hidden Markov models that have been used for decades in speech and handwriting recognition, despite their well-known shortcomings. This paper proposes an alternative approach based on a novel type of recurrent neural network, specifically designed for sequence labeling tasks where the data is hard to segment and contains long-range bidirectional interdependencies. In experiments on two large unconstrained handwriting databases, our approach achieves word recognition accuracies of 79.7 percent on online data and 74.1 percent on offline data, significantly outperforming a state-of-the-art HMM-based system. In addition, we demonstrate the network's robustness to lexicon size, measure the individual influence of its hidden layers, and analyze its use of context. Last, we provide an in-depth discussion of the differences between the network and HMMs, suggesting reasons for the network's superior performance.

Error Minimized Extreme Learning Machine With Growth of Hidden Nodes and Incremental Learning

Abstract: One of the open problems in neural network research is how to automatically determine network architectures for given applications. In this brief, we propose a simple and efficient approach to automatically determine the number of hidden nodes in generalized single-hidden-layer feedforward networks (SLFNs) which need not be neural alike. This approach referred to as error minimized extreme learning machine (EM-ELM) can add random hidden nodes to SLFNs one by one or group by group (with varying group size). During the growth of the networks, the output weights are updated incrementally. The convergence of this approach is proved in this brief as well. Simulation results demonstrate and verify that our new approach is much faster than other sequential/incremental/growing algorithms with good generalization performance.

Neural Network for Graphs: A Contextual Constructive Approach

Abstract: This paper presents a new approach for learning in structured domains (SDs) using a constructive neural network for graphs (NN4G). The new model allows the extension of the input domain for supervised neural networks to a general class of graphs including both acyclic/cyclic, directed/undirected labeled graphs. In particular, the model can realize adaptive contextual transductions, learning the mapping from graphs for both classification and regression tasks. In contrast to previous neural networks for structures that had a recursive dynamics, NN4G is based on a constructive feedforward architecture with state variables that uses neurons with no feedback connections. The neurons are applied to the input graphs by a general traversal process that relaxes the constraints of previous approaches derived by the causality assumption over hierarchical input data. Moreover, the incremental approach eliminates the need to introduce cyclic dependencies in the definition of the system state variables. In the traversal process, the NN4G units exploit (local) contextual information of the graphs vertices. In spite of the simplicity of the approach, we show that, through the compositionality of the contextual information developed by the learning, the model can deal with contextual information that is incrementally extended according to the graphs topology. The effectiveness and the generality of the new approach are investigated by analyzing its theoretical properties and providing experimental results.

Neural network architectures and learning algorithms

Abstract: Neural networks are the topic of this paper. Neural networks are very powerful as nonlinear signal processors, but obtained results are often far from satisfactory. The purpose of this article is to evaluate the reasons for these frustrations and show how to make these neural networks successful. The following are the main challenges of neural network applications: (1) Which neural network architectures should be used? (2) How large should a neural network be? (3) Which learning algorithms are most suitable? The multilayer perceptron (MLP) architecture is unfortunately the preferred neural network topology of most researchers. It is the oldest neural network architecture, and it is compatible with all training softwares. However, the MLP topology is less powerful than other topologies such as bridged multilayer perceptron (BMLP), where connections across layers are allowed. The error-back propagation (EBP) algorithm is the most popular learning algorithm, but it is very slow and seldom gives adequate results. The EBP training process requires 100-1,000 times more iterations than the more advanced algorithms such as Levenberg-Marquardt (LM) or neuron by neuron (NBN) algorithms. What is most important is that the EBP algorithm is not only slow but often it is not able to find solutions for close-to-optimum neural networks. The paper describes and compares several learning algorithms.

Adaptive Neural Sliding Mode Control of Nonholonomic Wheeled Mobile Robots With Model Uncertainty

Abstract: This brief proposes an adaptive neural sliding mode control method for trajectory tracking of nonholonomic wheeled mobile robots with model uncertainties and external disturbances. The dynamic model with model uncertainties and the kinematic model represented by polar coordinates are considered to design a robust control system. Self recurrent wavelet neural networks (SRWNNs) are used for approximating arbitrary model uncertainties and external disturbances in dynamics of the mobile robot. From the Lyapunov stability theory, we derive online tuning algorithms for all weights of SRWNNs and prove that all signals of a closed-loop system are uniformly ultimately bounded. Finally, we perform computer simulations to demonstrate the robustness and performance of the proposed control system.

SORN: a self-organizing recurrent neural network

Abstract: Understanding the dynamics of recurrent neural networks is crucial for explaining how the brain processes information. In the neocortex, a range of different plasticity mechanisms are shaping recurrent networks into effective information processing circuits that learn appropriate representations for time-varying sensory stimuli. However, it has been difficult to mimic these abilities in artificial neural network models. Here we introduce SORN, a self-organizing recurrent network. It combines three distinct forms of local plasticity to learn spatio-temporal patterns in its input while maintaining its dynamics in a healthy regime suitable for learning. The SORN learns to encode information in the form of trajectories through its high-dimensional state space reminiscent of recent biological findings on cortical coding. All three forms of plasticity are shown to be essential for the network's success.

Computational Capabilities of Graph Neural Networks

Abstract: In this paper, we will consider the approximation properties of a recently introduced neural network model called graph neural network (GNN), which can be used to process-structured data inputs, e.g., acyclic graphs, cyclic graphs, and directed or undirected graphs. This class of neural networks implements a function tau(G, n) isin R m that maps a graph G and one of its nodes n onto an m-dimensional Euclidean space. We characterize the functions that can be approximated by GNNs, in probability, up to any prescribed degree of precision. This set contains the maps that satisfy a property called preservation of the unfolding equivalence, and includes most of the practically useful functions on graphs; the only known exception is when the input graph contains particular patterns of symmetries when unfolding equivalence may not be preserved. The result can be considered an extension of the universal approximation property established for the classic feedforward neural networks (FNNs). Some experimental examples are used to show the computational capabilities of the proposed model.

A Recurrent Neural-Network-Based Sensor and Actuator Fault Detection and Isolation for Nonlinear Systems With Application to the Satellite's Attitude Control Subsystem

Abstract: This paper presents a robust fault detection and isolation (FDI) scheme for a general class of nonlinear systems using a neural-network-based observer strategy. Both actuator and sensor faults are considered. The nonlinear system considered is subject to both state and sensor uncertainties and disturbances. Two recurrent neural networks are employed to identify general unknown actuator and sensor faults, respectively. The neural network weights are updated according to a modified backpropagation scheme. Unlike many previous methods developed in the literature, our proposed FDI scheme does not rely on availability of full state measurements. The stability of the overall FDI scheme in presence of unknown sensor and actuator faults as well as plant and sensor noise and uncertainties is shown by using the Lyapunov's direct method. The stability analysis developed requires no restrictive assumptions on the system and/or the FDI algorithm. Magnetorquer-type actuators and magnetometer-type sensors that are commonly employed in the attitude control subsystem (ACS) of low-Earth orbit (LEO) satellites for attitude determination and control are considered in our case studies. The effectiveness and capabilities of our proposed fault diagnosis strategy are demonstrated and validated through extensive simulation studies.

PID-Like Neural Network Nonlinear Adaptive Control for Uncertain Multivariable Motion Control Systems

Abstract: A mix locally recurrent neural network was used to create a proportional-integral-derivative (PID)-like neural network nonlinear adaptive controller for uncertain multivariable single-input/multi-output system. It is composed of a neural network with no more than three neural nodes in hidden layer, and there are included an activation feedback and an output feedback, respectively, in a hidden layer. Such a special structure makes the exterior feature of the neural network controller able to become a P, PI, PD, or PID controller as needed. The closed-loop error between directly measured output and expected value of the system is chosen to be the input of the controller. Only a group of initial weights values, which can run the controlled closed-loop system stably, are required to be determined. The proposed controller can update weights of the neural network online according to errors caused by uncertain factors of system such as modeling error and external disturbance, based on stable learning rate. The resilient back-propagation algorithm with sign instead of the gradient is used to update the network weights. The basic ideas, techniques, and system stability proof were presented in detail. Finally, actual experiments both of single and double inverted pendulums were implemented, and the comparison of effectiveness between the proposed controller and the linear optimal regulator were given.

From Zhang Neural Network to Newton Iteration for Matrix Inversion

Abstract: Different from gradient-based neural networks, a special kind of recurrent neural network (RNN) has recently been proposed by Zhang for online matrix inversion. Such an RNN is designed based on a matrix-valued error function instead of a scalar-valued error function. In addition, it was depicted in an implicit dynamics instead of an explicit dynamics. In this paper, we develop and investigate a discrete-time model of Zhang neural network (termed as such and abbreviated as ZNN for presentation convenience), which is depicted by a system of difference equations. Comparing with Newton iteration for matrix inversion, we find that the discrete-time ZNN model incorporates Newton iteration as its special case. Noticing this relation, we perform numerical comparisons on different situations of using ZNN and Newton iteration for matrix inversion. Different kinds of activation functions and different step-size values are examined for superior convergence and better stability of ZNN. Numerical examples demonstrate the efficacy of both ZNN and Newton iteration for online matrix inversion.

Almost Sure Exponential Stability of Recurrent Neural Networks With Markovian Switching

Abstract: This paper presents new stability results for recurrent neural networks with Markovian switching. First, algebraic criteria for the almost sure exponential stability of recurrent neural networks with Markovian switching and without time delays are derived. The results show that the almost sure exponential stability of such a neural network does not require the stability of the neural network at every individual parametric configuration. Next, both delay-dependent and delay-independent criteria for the almost sure exponential stability of recurrent neural networks with time-varying delays and Markovian-switching parameters are derived by means of a generalized stochastic Halanay inequality. The results herein include existing ones for recurrent neural networks without Markovian switching as special cases. Finally, simulation results in three numerical examples are discussed to illustrate the theoretical results.

Evolving Memory Cell Structures for Sequence Learning

Abstract: Long Short-Term Memory (LSTM) is one of the best recent supervised sequence learning methods. Using gradient descent, it trains memory cells represented as differentiable computational graph structures. Interestingly, LSTM's cell structure seems somewhat arbitrary. In this paper we optimize its computational structure using a multi-objective evolutionary algorithm. The fitness function reflects the structure's usefulness for learning various formal languages. The evolved cells help to understand crucial features that aid sequence learning.

Embedding Multiple Trajectories in Simulated Recurrent Neural Networks in a Self-Organizing Manner

Abstract: Complex neural dynamics produced by the recurrent architecture of neocortical circuits is critical to the cortex's computational power. However, the synaptic learning rules underlying the creation of stable propagation and reproducible neural trajectories within recurrent networks are not understood. Here, we examined synaptic learning rules with the goal of creating recurrent networks in which evoked activity would: (1) propagate throughout the entire network in response to a brief stimulus while avoiding runaway excitation; (2) exhibit spatially and temporally sparse dynamics; and (3) incorporate multiple neural trajectories, i.e., different input patterns should elicit distinct trajectories. We established that an unsupervised learning rule, termed presynaptic-dependent scaling (PSD), can achieve the proposed network dynamics. To quantify the structure of the trained networks, we developed a recurrence index, which revealed that presynaptic-dependent scaling generated a functionally feedforward network when training with a single stimulus. However, training the network with multiple input patterns established that: (1) multiple non-overlapping stable trajectories can be embedded in the network; and (2) the structure of the network became progressively more complex (recurrent) as the number of training patterns increased. In addition, we determined that PSD and spike-timing-dependent plasticity operating in parallel improved the ability of the network to incorporate multiple and less variable trajectories, but also shortened the duration of the neural trajectory. Together, these results establish one of the first learning rules that can embed multiple trajectories, each of which recruits all neurons, within recurrent neural networks in a self-organizing manner.

Pruning and regularization in reservoir computing

Abstract: Reservoir computing is a new paradigm for using recurrent neural network with a much simpler training method. The key idea is to use a large but fixed recurrent part as a reservoir of dynamic features and to train only the output layer to extract the desired information. We propose to study how pruning some connections from the reservoir to the output layer can help on the one hand to increase the generalization ability, in much the same way as regularization techniques do, and on the other hand to improve the implementability of reservoirs in hardware.

A Neuro-Sliding-Mode Control With Adaptive Modeling of Uncertainty for Control of Movement in Paralyzed Limbs Using Functional Electrical Stimulation

Abstract: During the past several years, several strategies have been proposed for control of joint movement in paraplegic subjects using functional electrical stimulation (FES), but developing a control strategy that provides satisfactory tracking performance, to be robust against time-varying properties of muscle-joint dynamics, day-to-day variations, subject-to-subject variations, muscle fatigue, and external disturbances, and to be easy to apply without any re-identification of plant dynamics during different experiment sessions is still an open problem. In this paper, we propose a novel control methodology that is based on synergistic combination of neural networks with sliding-mode control (SMC) for controlling FES. The main advantage of SMC derives from the property of robustness to system uncertainties and external disturbances. However, the main drawback of the standard sliding modes is mostly related to the so-called chattering caused by the high-frequency control switching. To eliminate the chattering, we couple two neural networks with online learning without any offline training into the SMC. A recurrent neural network is used to model the uncertainties and provide an auxiliary equivalent control to keep the uncertainties to low values, and consequently, to use an SMC with lower switching gain. The second neural network consists of a single neuron and is used as an auxiliary controller. The control law will be switched from the SMC to neural control, when the state trajectory of system enters in some boundary layer around the sliding surface. Extensive simulations and experiments on healthy and paraplegic subjects are provided to demonstrate the robustness, stability, and tracking accuracy of the proposed neuroadaptive SMC. The results show that the neuro-SMC provides accurate tracking control with fast convergence for different reference trajectories and could generate control signals to compensate the muscle fatigue and reject the external disturbance.

Discrete-Time Recurrent Neural Networks With Complex-Valued Linear Threshold Neurons

Abstract: This brief discusses a class of discrete-time recurrent neural networks with complex-valued linear threshold neurons. It addresses the boundedness, global attractivity, and complete stability of such networks. Some conditions for those properties are also derived. Examples and simulation results are used to illustrate the theory.

Using Artificial Neural Networks to Model Nonlinearity: The Case of the Job Satisfaction--Job Performance Relationship

Abstract: Neural networks are advanced pattern recognition algorithms capable of extracting complex, nonlinear relationships among variables. This study examines those capabilities by modeling nonlinearities in the job satisfaction—job performance relationship with multilayer perceptron and radial basis function neural networks. A framework for studying nonlinear relationships with neural networks is offered. It is implemented using the job satisfaction—job performance relationship with results indicative of pervasive patterns of nonlinearity.

Multistability and New Attraction Basins of Almost-Periodic Solutions of Delayed Neural Networks

Abstract: In this paper, we investigate multistability of almost-periodic solutions of recurrently connected neural networks with delays (simply called delayed neural networks). We will reveal that under some conditions, the space Rn can be divided into 2n subsets, and in each subset, the delayed n -neuron neural network has a locally stable almost-periodic solution. Furthermore, we also investigate the attraction basins of these almost-periodic solutions. We reveal that the attraction basin of almost-periodic trajectory is larger than the subset, where the corresponding almost-periodic trajectory is located. In addition, several numerical simulations are presented to corroborate the theoretical results.

Comparison of feedforward and feedback neural network architectures for short term wind speed prediction

Abstract: This paper compares three types of neural networks trained using particle swarm optimization (PSO) for use in the short term prediction of wind speed. The three types of neural networks compared are the multi-layer perceptron (MLP) neural network, Elman recurrent neural network, and simultaneous recurrent neural network (SRN). Each network is trained and tested using meteorological data of one week measured at the National Renewable Energy Laboratory National Wind Technology Center near Boulder, CO. Results show that while the recurrent neural networks outperform the MLP in the best and average case with a lower overall mean squared error, the MLP performance is comparable. The better performance of the feedback architectures is also shown using the mean absolute relative error. While the SRN performance is superior, the increase in required training time for the SRN over the other networks may be a constraint, depending on the application.

Cognitive Computation with Autonomously Active Neural Networks: An Emerging Field

Abstract: The human brain is autonomously active. To understand the functional role of this self-sustained neural activity, and its interplay with the sensory data input stream, is an important question in cognitive system research and we review here the present state of theoretical modeling. This review will start with a brief overview of the experimental efforts, together with a discussion of transient versus self-sustained neural activity in the framework of reservoir computing. The main emphasis will be then on two paradigmal neural network architectures showing continuously ongoing transient-state dynamics: saddle point networks and networks of attractor relics. Self-active neural networks are confronted with two seemingly contrasting demands: a stable internal dynamical state and sensitivity to incoming stimuli. We show, that this dilemma can be solved by networks of attractor relics based on competitive neural dynamics, where the attractor relics compete on one side with each other for transient dominance, and on the other side with the dynamical influence of the input signals. Unsupervised and local Hebbian-style online learning then allows the system to build up correlations between the internal dynamical transient states and the sensory input stream. An emergent cognitive capability results from this set-up. The system performs online, and on its own, a nonlinear independent component analysis of the sensory data stream, all the time being continuously and autonomously active. This process maps the independent components of the sensory input onto the attractor relics, which acquire in this way a semantic meaning.

Neural Networks and Its Application in Engineering

Abstract: Neural Network (NN) has emerged over the years and has made remarkable contribution to the advancement of various fields of endeavor. The purpose of this work is to examine neural net-works and their emerging applications in the field of engineering, focusing more on Controls. In this work, we have examined the various architectures of NN and the learning process. The needs for neural networks, training of neural networks, and important algorithms used in realizing neu-ral networks have also been briefly discussed. Neural network application in control engineering has been extensively discussed, whereas its applications in electrical, civil and agricultural engi-neering were also examined. We concluded by identifying limitations, recent advances and prom-ising future research directions . Keywords. Neural Network, Artificial Neural Network Introduction Whenever we talk about a neural network, we should more properly say "artificial neural net-work" (ANN), because that is what we mean most of the time. Artificial neural networks are computers whose architecture is modeled after the brain. They typically consist of many hundreds of simple processing units which are wired together in a complex communication network. Each unit or node is a simplified model of a real neuron which fires (sends off a new signal) if it re-ceives a sufficiently strong input signal from the other nodes to which it is connected. An Artificial Neural Network (ANN) is an information processing paradigm that is inspired by the way biological nervous systems, such as the brain, process information. The key element of this paradigm is the novel structure of the information processing system. It is composed of a large number of highly interconnected processing elements (neurons) working in unison to solve specific problems. ANNs, like people, learn by example . According to Michael Mozer of the Uni-versity of Colorado, “ The neural network is structured to perform nonlinear Bayesian classifica-tion”. A neural network could be also be de-scribed as a system composed of many simple processing elements operating in parallel whose function is determined by network structure , connection strengths , and the processing performed at computing elements or nodes (DARPA Neural Network Study, 1988). It resembles the brain in two respects: