scispace - formally typeset
Search or ask a question

Showing papers on "Recurrent neural network published in 2008"


Book ChapterDOI
08 Dec 2008
TL;DR: This paper introduces a globally trained offline handwriting recogniser that takes raw pixel data as input and does not require any alphabet specific preprocessing, and can therefore be used unchanged for any language.
Abstract: Offline handwriting recognition—the automatic transcription of images of handwritten text—is a challenging task that combines computer vision with sequence learning. In most systems the two elements are handled separately, with sophisticated preprocessing techniques used to extract the image features and sequential models such as HMMs used to provide the transcriptions. By combining two recent innovations in neural networks—multidimensional recurrent neural networks and connectionist temporal classification—this paper introduces a globally trained offline handwriting recogniser that takes raw pixel data as input. Unlike competing systems, it does not require any alphabet specific preprocessing, and can therefore be used unchanged for any language. Evidence of its generality and power is provided by data from a recent international Arabic recognition competition, where it outperformed all entries (91.4% accuracy compared to 87.2% for the competition winner) despite the fact that neither author understands a word of Arabic.

729 citations


Proceedings ArticleDOI
F.O. Heimes1
12 Dec 2008
TL;DR: This paper presents an approach and solution to the IEEE 2008 Prognostics and Health Management conference challenge problem that utilizes an advanced recurrent neural network architecture to estimate the remaining useful life of the system.
Abstract: This paper presents an approach and solution to the IEEE 2008 Prognostics and Health Management conference challenge problem. The solution utilizes an advanced recurrent neural network architecture to estimate the remaining useful life of the system. The recurrent neural network is trained with back-propagation through time gradient calculations, an Extended Kalman Filter training method, and evolutionary algorithms to generate an accurate and compact algorithm. This solution placed second overall in the competition with a very small margin between the first and second place finishers.

470 citations


Journal ArticleDOI
TL;DR: The findings suggest that while recurrent neural networks and support vector machines show the best performance, their forecasting accuracy was not statistically significantly better than that of the regression model.

419 citations


Journal ArticleDOI
TL;DR: The Lyapunov-Krasovskii stability theory for functional differential equations and the linear matrix inequality (LMI) approach are employed and the results are shown to be generalizations of some previously published results and are less conservative than existing results.
Abstract: In this paper, several sufficient conditions are established for the global asymptotic stability of recurrent neural networks with multiple time-varying delays. The Lyapunov-Krasovskii stability theory for functional differential equations and the linear matrix inequality (LMI) approach are employed in our investigation. The results are shown to be generalizations of some previously published results and are less conservative than existing results. The present results are also applied to recurrent neural networks with constant time delays.

337 citations


Journal ArticleDOI
01 Apr 2008
TL;DR: A general array model of coupled delayed neural networks with hybrid coupling, which is composed of constant coupling, discrete- delay coupling, and distributed-delay coupling is proposed and studied.
Abstract: In this paper, we propose and study a general array model of coupled delayed neural networks with hybrid coupling, which is composed of constant coupling, discrete-delay coupling, and distributed-delay coupling. Based on the Lyapunov functional method and Kronecker product properties, several sufficient conditions are established to ensure global exponential synchronization based on the design of the coupling matrices, the inner linking matrices, and/or some free matrices representing the relationships between the system matrices. The conditions are expressed within the framework of linear matrix inequalities, which can be easily computed by the interior-point method. In addition, a typical chaotic cellular neural network is used as the node in the array to illustrate the effectiveness and advantages of the theoretical results.

336 citations


Journal Article
TL;DR: A way to use the classic statistical methodologies (R/S Rescaled Range analysis and Hurst exponent) to obtain new methods of improving the process efficiency of the prediction chaotic time series with NARX is identified.
Abstract: The prediction of chaotic time series with neural networks is a traditional practical problem of dynamic systems. This paper is not intended for proposing a new model or a new methodology, but to study carefully and thoroughly several aspects of a model on which there are no enough communicated experimental data, as well as to derive conclusions that would be of interest. The recurrent neural networks (RNN) models are not only important for the forecasting of time series but also generally for the control of the dynamical system. A RNN with a sufficiently large number of neurons is a nonlinear autoregressive and moving average (NARMA) model, with "moving average" referring to the inputs. The prediction can be assimilated to identification of dynamic process. An architectural approach of RNN with embedded memory, "Nonlinear Autoregressive model process with eXogenous input" (NARX), showing promising qualities for dynamic system applications, is analyzed in this paper. The performances of the NARX model are verified for several types of chaotic or fractal time series applied as input for neural network, in relation with the number of neurons, the training algorithms and the dimensions of his embedded memory. In addition, this work has attempted to identify a way to use the classic statistical methodologies (R/S Rescaled Range analysis and Hurst exponent) to obtain new methods of improving the process efficiency of the prediction chaotic time series with NARX.

333 citations


Journal Article
TL;DR: This paper compares a neuroevolution method called Cooperative Synapse Neuroevolution (CoSyNE), that uses cooperative coevolution at the level of individual synaptic weights, to a broad range of reinforcement learning algorithms on very difficult versions of the pole balancing problem that involve large state spaces and hidden state.
Abstract: Many complex control problems require sophisticated solutions that are not amenable to traditional controller design. Not only is it difficult to model real world systems, but often it is unclear what kind of behavior is required to solve the task. Reinforcement learning (RL) approaches have made progress by using direct interaction with the task environment, but have so far not scaled well to large state spaces and environments that are not fully observable. In recent years, neuroevolution, the artificial evolution of neural networks, has had remarkable success in tasks that exhibit these two properties. In this paper, we compare a neuroevolution method called Cooperative Synapse Neuroevolution (CoSyNE), that uses cooperative coevolution at the level of individual synaptic weights, to a broad range of reinforcement learning algorithms on very difficult versions of the pole balancing problem that involve large (continuous) state spaces and hidden state. CoSyNE is shown to be significantly more efficient and powerful than the other methods on these tasks.

327 citations


Journal ArticleDOI
TL;DR: The resulting learning theory predicts that even difficult credit-assignment problems can be solved in a self-organizing manner through reward-modulated STDP, and provides a possible functional explanation for trial-to-trial variability, which is characteristic for cortical networks of neurons but has no analogue in currently existing artificial computing systems.
Abstract: Reward-modulated spike-timing-dependent plasticity (STDP) has recently emerged as a candidate for a learning rule that could explain how behaviorally relevant adaptive changes in complex networks of spiking neurons could be achieved in a self-organizing manner through local synaptic plasticity. However, the capabilities and limitations of this learning rule could so far only be tested through computer simulations. This article provides tools for an analytic treatment of reward-modulated STDP, which allows us to predict under which conditions reward-modulated STDP will achieve a desired learning effect. These analytical results imply that neurons can learn through reward-modulated STDP to classify not only spatial but also temporal firing patterns of presynaptic neurons. They also can learn to respond to specific presynaptic firing patterns with particular spike patterns. Finally, the resulting learning theory predicts that even difficult credit-assignment problems, where it is very hard to tell which synaptic weights should be modified in order to increase the global reward for the system, can be solved in a self-organizing manner through reward-modulated STDP. This yields an explanation for a fundamental experimental result on biofeedback in monkeys by Fetz and Baker. In this experiment monkeys were rewarded for increasing the firing rate of a particular neuron in the cortex and were able to solve this extremely difficult credit assignment problem. Our model for this experiment relies on a combination of reward-modulated STDP with variable spontaneous firing activity. Hence it also provides a possible functional explanation for trial-to-trial variability, which is characteristic for cortical networks of neurons but has no analogue in currently existing artificial computing systems. In addition our model demonstrates that reward-modulated STDP can be applied to all synapses in a large recurrent neural network without endangering the stability of the network dynamics.

288 citations


Journal ArticleDOI
TL;DR: In this paper, the global exponential stability and periodicity for a class of reaction-diffusion delayed recurrent neural networks with Dirichlet boundary conditions are addressed by constructing suitable Lyapunov functionals and utilizing some inequality techniques.
Abstract: In this paper, the global exponential stability and periodicity for a class of reaction–diffusion delayed recurrent neural networks with Dirichlet boundary conditions are addressed by constructing suitable Lyapunov functionals and utilizing some inequality techniques. We first prove global exponential converge to 0 of the difference between any two solutions of the original reaction–diffusion delayed recurrent neural networks with Dirichlet boundary conditions, the existence and uniqueness of equilibrium is the direct results of this procedure. This approach is different from the usually used one where the existence, uniqueness of equilibrium and stability are proved in two separate steps. Furthermore, we prove periodicity of the reaction–diffusion delayed recurrent neural networks with Dirichlet boundary conditions. Sufficient conditions ensuring the global exponential stability and the existence of periodic oscillatory solutions for the reaction–diffusion delayed recurrent neural networks with Dirichlet boundary conditions are given. These conditions are easy to check and have important leading significance in the design and application of reaction–diffusion recurrent neural networks with delays. Finally, two numerical examples are given to show the effectiveness of the obtained results.

253 citations


Journal ArticleDOI
TL;DR: This neural network is capable of solving a large class of quadratic programming problems and is proven to be globally stable and to be convergent to optimal solutions as long as the objective function is strictly convex on a set defined by the equality constraints.
Abstract: In this paper, a one-layer recurrent neural network with a discontinuous hard-limiting activation function is proposed for quadratic programming. This neural network is capable of solving a large class of quadratic programming problems. The state variables of the neural network are proven to be globally stable and the output variables are proven to be convergent to optimal solutions as long as the objective function is strictly convex on a set defined by the equality constraints. In addition, a sequential quadratic programming approach based on the proposed recurrent neural network is developed for general nonlinear programming. Simulation results on numerical examples and support vector machine (SVM) learning show the effectiveness and performance of the neural network.

219 citations


Book
02 Oct 2008
TL;DR: Introduction to Neural Networks with C#, Second Edition, introduces the C# programmer to the world of Neural Networks and Artificial Intelligence.
Abstract: Introduction to Neural Networks with C#, Second Edition, introduces the C# programmer to the world of Neural Networks and Artificial Intelligence Neural network architectures, such as the feedforward, Hopfield, and self-organizing map architectures are discussed Training techniques, such as backpropagation, genetic algorithms and simulated annealing are also introduced Practical examples are given for each neural network Examples include the traveling salesman problem, handwriting recognition, financial prediction, game strategy, mathematical functions, and Internet bots All C# source code is available online for easy downloading

Journal ArticleDOI
TL;DR: A novel recurrent neural network for solving a class of convex quadratic programming (QP) problems, in which the quadRatic term in the objective function is the square of the Euclidean norm of the variable, based on which the neural network model is formulated.
Abstract: This paper presents a novel recurrent neural network for solving a class of convex quadratic programming (QP) problems, in which the quadratic term in the objective function is the square of the Euclidean norm of the variable. This special structure leads to a set of simple optimality conditions for the problem, based on which the neural network model is formulated. Compared with existing neural networks for general convex QP, the new model is simpler in structure and easier to implement. The new model can be regarded as an improved version of the dual neural network in the literature. Based on the new model, a simple neural network capable of solving the $k$ -winners-take-all ( $k$ -WTA) problem is formulated. The stability and global convergence of the proposed neural network is proved rigorously and substantiated by simulation results.

Journal ArticleDOI
TL;DR: A diagonal recurrent wavelet neural network (DRWNN) is newly established in this paper to perform fine forecasting of hourly and daily global solar irradiance and comparisons between irradiation models show that the DRWNN models are definitely more accurate.

Journal ArticleDOI
TL;DR: It is shown that the proposed neural network is stable at a Karush-Kuhn-Tucker point in the sense of Lyapunov and its output trajectory is globally convergent to a minimum solution and there is no restriction on the initial point.
Abstract: This paper presents a novel recurrent neural network for solving nonlinear optimization problems with inequality constraints. Under the condition that the Hessian matrix of the associated Lagrangian function is positive semidefinite, it is shown that the proposed neural network is stable at a Karush-Kuhn-Tucker point in the sense of Lyapunov and its output trajectory is globally convergent to a minimum solution. Compared with variety of the existing projection neural networks, including their extensions and modification, for solving such nonlinearly constrained optimization problems, it is shown that the proposed neural network can solve constrained convex optimization problems and a class of constrained nonconvex optimization problems and there is no restriction on the initial point. Simulation results show the effectiveness of the proposed neural network in solving nonlinearly constrained optimization problems.

Book
24 Jun 2008
TL;DR: Modelling Issue in Fault Diagnosis, Locally Recurrent Neural Networks, and Stability and Stabilization of Locally recurrent Networks.
Abstract: Modelling Issue in Fault Diagnosis.- Locally Recurrent Neural Networks.- Approximation Abilities of Locally Recurrent Networks.- Stability and Stabilization of Locally Recurrent Networks.- Optimum Experimental Design for Locally Recurrent Networks.- Decision Making in Fault Detection.- Industrial Applications.- Concluding Remarks and Further Research Directions.

Journal ArticleDOI
01 Dec 2008
TL;DR: The proposed procedure enables both hetero- and autoassociative memories to be synthesized with high storage capacity and assured global asymptotic stability.
Abstract: This paper presents a design method for synthesizing associative memories based on discrete-time recurrent neural networks. The proposed procedure enables both hetero- and autoassociative memories to be synthesized with high storage capacity and assured global asymptotic stability. The stored patterns are retrieved by feeding probes via external inputs rather than initial conditions. As typical representatives, discrete-time cellular neural networks (CNNs) designed with space-invariant cloning templates are examined in detail. In particular, it is shown that procedure herein can determine the input matrix of any CNN based on a space-invariant cloning template which involves only a few design parameters. Two specific examples and many experimental results are included to demonstrate the characteristics and performance of the designed associative memories.

Journal Article
TL;DR: It is concluded that modulatory neurons evolve autonomously in the proposed learning tasks, allowing for increased learning and memory capabilities.
Abstract: Neuromodulation is considered a key factor for learning and memory in biological neural networks. Similarly, artificial neural networks could benefit from modulatory dynamics when facing certain types of learning problem. Here we test this hypothesis by introducing modulatory neurons to enhance or dampen neural plasticity at target neural nodes. Simulated evolution is employed to design neural control networks for T-maze learning problems, using both standard and modulatory neurons. The results show that experiments where modulatory neurons are enabled achieve better learning in comparison to those where modulatory neurons are disabled. We conclude that modulatory neurons evolve autonomously in the proposed learning tasks, allowing for increased learning and memory capabilities.

Journal ArticleDOI
TL;DR: The overall results obtained through this ensemble method are more accurate than those obtained through the standard method, backpropagation through time, on these datasets and perform significantly better even when long-range dependencies play an important role.

Journal ArticleDOI
TL;DR: A delay-dependent condition guaranteeing the global exponential stability of the concerned neural network is obtained in terms of a linear matrix inequality, which is less conservative than some previous ones in the literature.

Journal ArticleDOI
TL;DR: This paper presents a recursive algorithm for extracting classification rules from feedforward neural networks that have been trained on data sets having both discrete and continuous attributes and shows that for three real-life credit scoring data sets, the algorithm generates rules that are not only more accurate but also more comprehensible than those generated by other NN rule extraction methods.
Abstract: In this paper, we present a recursive algorithm for extracting classification rules from feedforward neural networks (NNs) that have been trained on data sets having both discrete and continuous attributes. The novelty of this algorithm lies in the conditions of the extracted rules: the rule conditions involving discrete attributes are disjoint from those involving continuous attributes. The algorithm starts by first generating rules with discrete attributes only to explain the classification process of the NN. If the accuracy of a rule with only discrete attributes is not satisfactory, the algorithm refines this rule by recursively generating more rules with discrete attributes not already present in the rule condition, or by generating a hyperplane involving only the continuous attributes. We show that for three real-life credit scoring data sets, the algorithm generates rules that are not only more accurate but also more comprehensible than those generated by other NN rule extraction methods.

Journal ArticleDOI
TL;DR: Delays-independent and delay-dependent stability conditions are proposed to ensure the asymptotic stability of the neural network to reduce the conservatism and the effectiveness of the proposed result.
Abstract: This brief is concerned with the stability for static neural networks with time-varying delays. Delay-independent conditions are proposed to ensure the asymptotic stability of the neural network. The delay-independent conditions are less conservative than existing ones. To further reduce the conservatism, delay-dependent conditions are also derived, which can be applied to fast time-varying delays. Expressed in linear matrix inequalities, both delay-independent and delay-dependent stability conditions can be checked using the recently developed algorithms. Examples are provided to illustrate the effectiveness and the reduced conservatism of the proposed result.

Journal ArticleDOI
TL;DR: Experimental results for temperature control of a variable-frequency oil-cooling process show the efficacy of the proposed adaptive predictive control with recurrent neural network prediction for industrial processes with set-points changes and load disturbances.
Abstract: An adaptive predictive control with recurrent neural network prediction for industrial processes is presented. The neural predictive control law with integral action is derived based on the minimization of a modified predictive performance criterion. The stability and steady-state performance of the closed-loop control system are well studied. Numerical simulations reveal that the proposed control gives satisfactory tracking and disturbance rejection performance for two illustrative nonlinear systems with time-delay. Experimental results for temperature control of a variable-frequency oil-cooling process show the efficacy of the proposed method for industrial processes with set-points changes and load disturbances.

Journal ArticleDOI
TL;DR: A comparison with other approaches shows that the new algorithm can considerably reduce network training time and improve solution accuracy.

Journal ArticleDOI
TL;DR: By employing Lyapunov functional and the free-weighting matrix method, several sufficient conditions in linear matrix inequality form are obtained to ensure the existence, uniqueness and global exponential stability of equilibrium point for the neural networks.

Journal ArticleDOI
TL;DR: An M-matrix-based algebraic criterion for the global exponential stability of a class of recurrent neural networks with decreasing time-varying delays is presented and the rate of exponential convergence can be estimated via a simple computation based on the criterion herein.
Abstract: This brief paper presents an M-matrix-based algebraic criterion for the global exponential stability of a class of recurrent neural networks with decreasing time-varying delays. The criterion improves some previous criteria based on M-matrix and is easy to be verified with the connection weights of the recurrent neural networks with decreasing time-varying delays. In addition, the rate of exponential convergence can be estimated via a simple computation based on the criterion herein.

Proceedings ArticleDOI
01 Feb 2008
TL;DR: The use of Multilayered Feedforward Neural Networks is proposed as an effective technique for real-time characterization of the communication performance which is based on measurements carried out by the device and therefore offers some interesting learning capabilities.
Abstract: The estimation of the communication performance achievable with respect to environmental factors and configuration parameters plays a key role in the optimization process performed by a Cognitive Radio according to the original definition by Mitola [1]. In this paper we propose the use of Multilayered Feedforward Neural Networks as an effective technique for real-time characterization of the communication performance which is based on measurements carried out by the device and therefore offers some interesting learning capabilities.

Proceedings ArticleDOI
31 Oct 2008
TL;DR: This paper has introduced the steps of the proposed system and presented the Elmanpsilas model as a partially recurrent architecture and a fully connected network with recurrent links that is believed to help the network to converge and gain stability.
Abstract: The objective of this paper is to introduce the use of two different recurrent neural networks in human hand gesture recognition for static images. Because neural networks are a promising tool for many human computer interaction applications, this paper focuses on the ability of neural networks to assist in Arabic Sign Language(ArSL) hand gesture recognition. We have introduced the steps of our proposed system and have presented the Elmanpsilas model as a partially recurrent architecture and a fully connected network with recurrent links that is believed to help the network to converge and gain stability, then we have tested it in an experiment held for this; the results of the experiment have showed that the suggested system with the fully recurrent architecture has had a performance with an accuracy rate 95%.

Journal ArticleDOI
TL;DR: A recurrent neural network trained with the backpropagation through time training algorithm is used to find a way of distinguishing between the so-called load harmonics and supply harmonics, without disconnecting the load from the network.
Abstract: Generation of harmonics and the existence of waveform pollution in power system networks are important problems facing the power utilities. The determination of harmonic currents injected into a power network by a nonlinear load is complicated when the supply voltage waveform to the load is distorted by other loads and not a pure sinusoid. This paper proposes a neural network solution to this problem. A recurrent neural network trained with the backpropagation through time training algorithm is used to find a way of distinguishing between the so-called load harmonics and supply harmonics, without disconnecting the load from the network. The advantage of this method is that only waveforms of voltages and currents have to be measured. This method is applicable for both single and three phase loads and could be fabricated into a commercial instrument that could be installed in substations of large customer loads, or used as a hand-held clip on instrument. This paper is particularly useful in determining whether the utility or the customer side has a higher contribution to harmonic pollution in a network. Hence, this method would be helpful in settling utility-customer disputes over who is responsible for harmonic pollution.

Journal ArticleDOI
TL;DR: By utilizing the Lyapunov functional and combining with the linear matrix inequality (LMI) approach, the global asymptotic stability of stochastic delayed recurrent neural networks with time varying delays is analyzed.

Journal ArticleDOI
TL;DR: The results support the use of neural networks with a dynamic framework to forecast the gold price sign variations, recalculating the weights of the network on a period-by-period basis, through a rolling process.