scispace - formally typeset
Search or ask a question

Showing papers on "Recurrent neural network published in 2005"


Proceedings ArticleDOI
27 Dec 2005
TL;DR: A new neural model, called graph neural network (GNN), capable of directly processing graphs, which extends recursive neural networks and can be applied on most of the practically useful kinds of graphs, including directed, undirected, labelled and cyclic graphs.
Abstract: In several applications the information is naturally represented by graphs. Traditional approaches cope with graphical data structures using a preprocessing phase which transforms the graphs into a set of flat vectors. However, in this way, important topological information may be lost and the achieved results may heavily depend on the preprocessing stage. This paper presents a new neural model, called graph neural network (GNN), capable of directly processing graphs. GNNs extends recursive neural networks and can be applied on most of the practically useful kinds of graphs, including directed, undirected, labelled and cyclic graphs. A learning algorithm for GNNs is proposed and some experiments are discussed which assess the properties of the model.

1,569 citations


Book ChapterDOI
11 Sep 2005
TL;DR: In this paper, two experiments on the TIMIT speech corpus with bidirectional and unidirectional Long Short Term Memory networks are carried out and it is found that a hybrid BLSTM-HMM system improves on an equivalent traditional HMM system.
Abstract: In this paper, we carry out two experiments on the TIMIT speech corpus with bidirectional and unidirectional Long Short Term Memory (LSTM) networks. In the first experiment (framewise phoneme classification) we find that bidirectional LSTMoutperforms both unidirectional LSTMand conventional Recurrent Neural Networks (RNNs). In the second (phoneme recognition) we find that a hybrid BLSTM-HMM system improves on an equivalent traditional HMM system, as well as unidirectional LSTM-HMM.

669 citations


Journal ArticleDOI
TL;DR: The obtained results demonstrated that the proposed RNNs employing the Lyapunov exponents can be useful in analyzing long-term EEG signals for early detection of the electroencephalographic changes.
Abstract: There are a number of different quantitative models that can be used in a medical diagnostic decision support system including parametric methods, non-parametric methods and several neural network models. Unfortunately, there is no theory available to guide model selection. The aim of this study is to evaluate the diagnostic accuracy of the recurrent neural networks (RNNs) employing Lyapunov exponents trained with Levenberg-Marquardt algorithm on the electroencephalogram (EEG) signals. An approach based on the consideration that the EEG signals are chaotic signals was used in developing a reliable classification method for electroencephalographic changes. This consideration was tested successfully using the non-linear dynamics tools, like the computation of Lyapunov exponents. We explored the ability of designed and trained Elman RNNs, combined with the Lyapunov exponents, to discriminate the EEG signals (EEG signals recorded from healthy volunteers with eyes open, epilepsy patients in the epileptogenic zone during a seizure-free interval, and epilepsy patients during epileptic seizures). The RNNs achieved accuracy rates which were higher than that of the feedforward neural network models. The obtained results demonstrated that the proposed RNNs employing the Lyapunov exponents can be useful in analyzing long-term EEG signals for early detection of the electroencephalographic changes.

500 citations


Journal ArticleDOI
TL;DR: Several sufficient conditions are derived for the existence, uniqueness, and GRS of equilibria for interval neural networks with time delays by use of a new Lyapunov function and matrix inequality.
Abstract: In this paper, two related problems, global asymptotic stability (GAS) and global robust stability (GRS) of neural networks with time delays, are studied. First, GAS of delayed neural networks is discussed based on Lyapunov method and linear matrix inequality. New criteria are given to ascertain the GAS of delayed neural networks. In the designs and applications of neural networks, it is necessary to consider the deviation effects of bounded perturbations of network parameters. In this case, a delayed neural network must be formulated as a interval neural network model. Several sufficient conditions are derived for the existence, uniqueness, and GRS of equilibria for interval neural networks with time delays by use of a new Lyapunov function and matrix inequality. These results are less restrictive than those given in the earlier references.

498 citations


Journal ArticleDOI
TL;DR: Simulation results substantiate the theoretical analysis and demonstrate the efficacy of the neural model on time-varying matrix inversion, especially when using a power-sigmoid activation function.
Abstract: Following the idea of using first-order time derivatives, this paper presents a general recurrent neural network (RNN) model for online inversion of time-varying matrices. Different kinds of activation functions are investigated to guarantee the global exponential convergence of the neural model to the exact inverse of a given time-varying matrix. The robustness of the proposed neural model is also studied with respect to different activation functions and various implementation errors. Simulation results, including the application to kinematic control of redundant manipulators, substantiate the theoretical analysis and demonstrate the efficacy of the neural model on time-varying matrix inversion, especially when using a power-sigmoid activation function.

466 citations


Journal ArticleDOI
TL;DR: This article proposes a freeway travel time prediction framework that exploits a recurrent neural network topology, the so-called state-space neural network (SSNN), with preprocessing strategies based on imputation that appears to be robust to the “damage” done by these imputation schemes.
Abstract: Accuracy and robustness with respect to missing or corrupt input data are two key characteristics for any travel time prediction model that is to be applied in a real-time environment (e.g. for display on variable message signs on freeways). This article proposes a freeway travel time prediction framework that exhibits both qualities. The framework exploits a recurrent neural network topology, the so-called state-space neural network (SSNN), with preprocessing strategies based on imputation. Although the SSNN model is a neural network, its design (in terms of input- and model selection) is not “black box” nor location-specific. Instead, it is based on the lay-out of the freeway stretch of interest. In this sense, the SSNN model combines the generality of neural network approaches, with traffic related (“white-box”) design. Robustness to missing data is tackled by means of simple imputation (data replacement) schemes, such as exponential forecasts and spatial interpolation. Although there are clear theoretical shortcomings to “simple” imputation schemes to remedy input failure, our results indicate that their use is justified in this particular application. The SSNN model appears to be robust to the “damage” done by these imputation schemes. This is true for both incidental (random) and structural input failure. We demonstrate that the SSNN travel time prediction framework yields good accurate and robust travel time predictions on both synthetic and real data.

461 citations


Proceedings ArticleDOI
27 Dec 2005
TL;DR: It is found that bidirectional LSTM outperforms both RNNs and unidirectionalLSTM, and the significance of framewise phoneme classification to continuous speech recognition and the validity of usingbidirectional networks for online causal tasks is discussed.
Abstract: In this paper, we apply bidirectional training to a long short term memory (LSTM) network for the first time. We also present a modified, full gradient version of the LSTM learning algorithm. We discuss the significance of framewise phoneme classification to continuous speech recognition, and the validity of using bidirectional networks for online causal tasks. On the TIMIT speech database, we measure the framewise phoneme classification scores of bidirectional and unidirectional variants of both LSTM and conventional recurrent neural networks (RNNs). We find that bidirectional LSTM outperforms both RNNs and unidirectional LSTM.

456 citations


Journal ArticleDOI
TL;DR: Experimental results show that Elman network yields epileptic detection accuracy rates as high as 99.6% with a single input feature which is better than the results obtained by using other types of neural networks with two and more input features.
Abstract: Electroencephalogram (EEG) signal plays an important role in the diagnosis of epilepsy. The long-term EEG recordings of an epileptic patient obtained from the ambulatory recording systems contain a large volume of EEG data. Detection of the epileptic activity requires a time consuming analysis of the entire length of the EEG data by an expert. The traditional methods of analysis being tedious, many automated diagnostic systems for epilepsy have emerged in recent years. This paper discusses an automated diagnostic method for epileptic detection using a special type of recurrent neural network known as Elman network. The experiments are carried out by using time-domain as well as frequency-domain features of the EEG signal. Experimental results show that Elman network yields epileptic detection accuracy rates as high as 99.6% with a single input feature which is better than the results obtained by using other types of neural networks with two and more input features.

400 citations


Journal ArticleDOI
TL;DR: New criteria are found to ascertain the global exponential stability and periodicity of the recurrent neural networks with time delays, and are also shown to be different from and improve upon existing ones.
Abstract: In this paper, the global exponential stability and periodicity of a class of recurrent neural networks with time delays are addressed by using Lyapunov functional method and inequality techniques. The delayed neural network includes the well-known Hopfield neural networks, cellular neural networks, and bidirectional associative memory networks as its special cases. New criteria are found to ascertain the global exponential stability and periodicity of the recurrent neural networks with time delays, and are also shown to be different from and improve upon existing ones.

306 citations


Journal ArticleDOI
TL;DR: A dynamic neural network model for forecasting time series events that uses a different architecture than traditional models is presented and shows that this approach is more accurate and performs significantly better than the traditional neural network and autoregressive integrated moving average (ARIMA) models.

251 citations


Journal ArticleDOI
TL;DR: In this article, a direct identification and inverse dynamic modeling for magnetorheological (MR) fluid dampers using feedforward and recurrent neural networks are studied. And the trained neural network models are applied to predict and control the damping force of the MR fluid damper.
Abstract: Due to the inherent nonlinear nature of magnetorheological (MR) fluid dampers, one of the challenging aspects for utilizing these devices to achieve high system performance is the development of accurate models and control algorithms that can take advantage of their unique characteristics. In this paper, the direct identification and inverse dynamic modeling for MR fluid dampers using feedforward and recurrent neural networks are studied. The trained direct identification neural network model can be used to predict the damping force of the MR fluid damper on line, on the basis of the dynamic responses across the MR fluid damper and the command voltage, and the inverse dynamic neural network model can be used to generate the command voltage according to the desired damping force through supervised learning. The architectures and the learning methods of the dynamic neural network models and inverse neural network models for MR fluid dampers are presented, and some simulation results are discussed. Finally, the trained neural network models are applied to predict and control the damping force of the MR fluid damper. Moreover, validation methods for the neural network models developed are proposed and used to evaluate their performance. Validation results with different data sets indicate that the proposed direct identification dynamic model using the recurrent neural network can be used to predict the damping force accurately and the inverse identification dynamic model using the recurrent neural network can act as a damper controller to generate the command voltage when the MR fluid damper is used in a semi-active mode.

Journal ArticleDOI
TL;DR: It is shown here that the proposed neural network is stable in the sense of Lyapunov and globally convergent to an optimal solution within a finite time under the condition that the objective function is strictly convex.
Abstract: In this paper, we propose a recurrent neural network for solving nonlinear convex programming problems with linear constraints. The proposed neural network has a simpler structure and a lower complexity for implementation than the existing neural networks for solving such problems. It is shown here that the proposed neural network is stable in the sense of Lyapunov and globally convergent to an optimal solution within a finite time under the condition that the objective function is strictly convex. Compared with the existing convergence results, the present results do not require Lipschitz continuity condition on the objective function. Finally, examples are provided to show the applicability of the proposed neural network.

Proceedings Article
05 Dec 2005
TL;DR: Training multi-layer neural networks in which the number of hidden units is learned can be viewed as a convex optimization problem, which involves an infinite number of variables but can be solved by incrementally inserting a hidden unit at a time.
Abstract: Convexity has recently received a lot of attention in the machine learning community, and the lack of convexity has been seen as a major disadvantage of many learning algorithms, such as multi-layer artificial neural networks. We show that training multi-layer neural networks in which the number of hidden units is learned can be viewed as a convex optimization problem. This problem involves an infinite number of variables, but can be solved by incrementally inserting a hidden unit at a time, each time finding a linear classifier that minimizes a weighted sum of errors.

Journal ArticleDOI
TL;DR: This article reviews the progress of techniques for extraction of rules from RNNs and develops a taxonomy specifically designed for this purpose, and identifies important open research issues that can give the field a significant push forward.
Abstract: Rule extraction (RE) from recurrent neural networks (RNNs) refers to finding models of the underlying RNN, typically in the form of finite state machines, that mimic the network to a satisfactory degree while having the advantage of being more transparent. RE from RNNs can be argued to allow a deeper and more profound form of analysis of RNNs than other, more or less ad hoc methods. RE may give us understanding of RNNs in the intermediate levels between quite abstract theoretical knowledge of RNNs as a class of computing devices and quantitative performance evaluations of RNN instantiations. The development of techniques for extraction of rules from RNNs has been an active field since the early 1990s. This article reviews the progress of this development and analyzes it in detail. In order to structure the survey and evaluate the techniques, a taxonomy specifically designed for this purpose has been developed. Moreover, important open research issues are identified that, if addressed properly, possibly can give the field a significant push forward.

Journal ArticleDOI
TL;DR: In this paper, an integrated recurrent neural network (IRNN) is introduced to identify the nonstationarity of the thermo-elastic process with a deterministic linear trend.
Abstract: This paper presents a new modeling methodology for nonstationary machine tool thermal errors. The method uses the dynamic neural network model to track nonlinear time-varying machine tool errors under various thermal conditions. To accommodate the nonstationary nature of the thermo-elastic process, an Integrated Recurrent Neural Network (IRNN) is introduced to identify the nonstationarity of the thermo-elastic process with a deterministic linear trend. Experiments on spindle thermal deformation are conducted to evaluate the model performance in terms of model estimation accuracy and robustness. The comparison indicates that the IRNN performs better than other modeling methods, such as, multi-variable regression analysis (MRA), multi-layer feedforward neural network (MFN), and recurrent neural network (RNN), in terms of model robustness under a variety of working conditions.

Journal Article
TL;DR: In this paper, a new method aimed at approximating missing data in a database using a combination of genetic algorithms and neural networks is introduced, which uses genetic algorithm to minimise an error function derived from an auto-associative neural network.
Abstract: Missing data creates various problems in analysing and processing data in databases. In this paper we introduce a new method aimed at approximating missing data in a database using a combination of genetic algorithms and neural networks. The proposed method uses genetic algorithm to minimise an error function derived from an auto-associative neural network. Multi-layer perceptron (MLP) and radial basis function (RBF) networks are employed to train the neural networks. Our focus also lies on the investigation of using the proposed method in accurately predicting missing data as the number of missing cases within a single record increases. It is observed that there is no significant reduction in accuracy of results as the number of missing cases in a single record increases. It is also found that results obtained using RBF are superior to MLP.

Journal ArticleDOI
TL;DR: A batch-to-batch model-based iterative optimal control strategy for batch processes is proposed, where a quadratic objective function is introduced to track the desired qualities at the end-point of a batch.

Journal ArticleDOI
01 Jan 2005
TL;DR: Simulation results from turbine blade optimizations using the structurally optimized neural network models are presented to show that the performance of the models can be improved significantly through structure optimization.
Abstract: We study the use of neural networks as approximate models for the fitness evaluation in evolutionary design optimization. To improve the quality of the neural network models, structure optimization of these networks is performed with respect to two different criteria: One is the commonly used approximation error with respect to all available data, and the other is the ability of the networks to learn different problems of a common class of problems fast and with high accuracy. Simulation results from turbine blade optimizations using the structurally optimized neural network models are presented to show that the performance of the models can be improved significantly through structure optimization.

Proceedings Article
01 Jan 2005
TL;DR: Attempts to evolve a Pac-Man player, where the control algorithm uses a neural network to evaluate the possible next moves, show that useful behaviours can be evolved that are frequently capable of clearing the first level, but are still susceptible to making poor decisions.
Abstract: Ms. Pac-Man is a challenging, classic arcade game with a certain cult status. This paper reports attempts to evolve a Pac-Man player, where the control algorithm uses a neural network to evaluate the possible next moves. The evolved neural network takes a handcrafted feature vector based on a candidate maze location as input, and produces a score for that location as output. Results are reported on two simulated versions of the game: deterministic and nondeterministic. The results show that useful behaviours can be evolved that are frequently capable of clearing the first level, but are still susceptible to making poor decisions. Currently, the best evolved players play at the level of a reasonable human novice.

Journal ArticleDOI
TL;DR: A control law is derived and a delay independent sufficient exponential synchronization condition is derived based on the Lyapunov stability method and the Halanay inequality lemma to achieve the state synchronization of two identical chaotic neural networks.
Abstract: This paper deals with the synchronization problem of a class of chaotic neural networks with or without delays. This class of chaotic neural networks covers several well-known neural networks, such as Hopfield neural networks, cellular neural networks, and bidirectional associative memory networks with or without delays. Using the drive-response concept, a control law is derived to achieve the state synchronization of two identical chaotic neural networks. Furthermore, based on the Lyapunov stability method and the Halanay inequality lemma, a delay independent sufficient exponential synchronization condition is derived. The synchronization condition is easy to verify and relies on the connection matrix in the driven networks and the suitable designed controller gain matrix in the response networks. Finally, some illustrative examples are given to demonstrate the effectiveness of the presented synchronization scheme.

Journal ArticleDOI
TL;DR: A sufficient condition for the existence, uniqueness and global asymptotic stability of the equilibrium point for bidirectional associative memory (BAM) neural networks with fixed time delays is presented.

Journal ArticleDOI
01 Feb 2005
TL;DR: Simulation results show that the proposed primal-dual neural network has a faster convergence rate than the dual neural network in effectively tracking for the motion control of kinematically redundant manipulators.
Abstract: This paper proposes a primal-dual neural network with a one-layer structure for online resolution of constrained kinematic redundancy in robot motion control. Unlike the Lagrangian network, the proposed neural network can handle physical constraints, such as joint limits and joint velocity limits. Compared with the existing primal-dual neural network, the proposed neural network has a low complexity for implementation. Compared with the existing dual neural network, the proposed neural network has no computation of matrix inversion. More importantly, the proposed neural network is theoretically proved to have not only a finite time convergence, but also an exponential convergence rate without any additional assumption. Simulation results show that the proposed neural network has a faster convergence rate than the dual neural network in effectively tracking for the motion control of kinematically redundant manipulators.

Journal ArticleDOI
TL;DR: This paper introduces a model that uses a different architecture compared to the traditional neural network, to capture and forecast nonlinear processes, and shows that this approach performs well when compared with traditional models and established research.

01 Jan 2005
TL;DR: This paper uses monotone dynamical systems theory to show that, generically, every orbit of the RNN is asymptotic to a periodic orbit, and investigates whether RNN's of this class can adapt their internal parameters so as to "learn" and then replicate autonomously (in feedback) certain external periodic signals.
Abstract: We study a particular class of n-node recurrent neural networks (RNNs). In the 3-node case we use monotone dynamical systems theory to show, for a well-defined set of parameters, that, generically, every orbit of the RNN is asymptotic to a periodic orbit. We then investigate whether RNNs of this class can adapt their internal parameters so as to ?learn? and then replicate autonomously (in feedback) certain external periodic signals. Our learning algorithm is similar to the identification algorithms in adaptive control theory. The main feature of the algorithm is that global exponential convergence of parameters is guaranteed. We also obtain partial convergence results in the n-node case

Journal ArticleDOI
01 Jun 2005
TL;DR: This paper proposes the combination of an EA, a clustering process, and a local-search procedure to the evolutionary design of product-units neural networks and shows a favorable performance when the regression method proposed is compared to other standard methods.
Abstract: This paper presents a hybrid evolutionary algorithm (EA) to solve nonlinear-regression problems. Although EAs have proven their ability to explore large search spaces, they are comparatively inefficient in fine tuning the solution. This drawback is usually avoided by means of local optimization algorithms that are applied to the individuals of the population. The algorithms that use local optimization procedures are usually called hybrid algorithms. On the other hand, it is well known that the clustering process enables the creation of groups (clusters) with mutually close points that hopefully correspond to relevant regions of attraction. Local-search procedures can then be started once in every such region. This paper proposes the combination of an EA, a clustering process, and a local-search procedure to the evolutionary design of product-units neural networks. In the methodology presented, only a few individuals are subject to local optimization. Moreover, the local optimization algorithm is only applied at specific stages of the evolutionary process. Our results show a favorable performance when the regression method proposed is compared to other standard methods.

Journal ArticleDOI
TL;DR: It is found that the network of small-world connectivity reduces the learning error and learning time when compared to the networks of regular or random connectivity.

Journal ArticleDOI
TL;DR: This paper considers the hardware implementation of Hopfield neural networks using a field-programmable gate array (FPGA) and shows that the learning scheme proposed here is feasible.
Abstract: Recurrent neural networks have interesting properties and can handle dynamic information processing unlike ordinary feedforward neural networks. However, they are generally difficult to use because there is no convenient learning scheme. In this paper, a recursive learning scheme for recurrent neural networks using the simultaneous perturbation method is described. The detailed procedure of the scheme for recurrent neural networks is explained. Unlike ordinary correlation learning, this method is applicable to analog learning and the learning of oscillatory solutions of recurrent neural networks. Moreover, as a typical example of recurrent neural networks, we consider the hardware implementation of Hopfield neural networks using a field-programmable gate array (FPGA). The details of the implementation are described. Two examples of a Hopfield neural network system for analog and oscillatory targets are shown. These results show that the learning scheme proposed here is feasible.

Journal ArticleDOI
TL;DR: It is proved that any finite time trajectory of a given dynamical time-variant system can be approximated by the internal state of a continuous-time recurrent neural network.
Abstract: This paper studies the approximation ability of continuous-time recurrent neural networks to dynamical time-variant systems. It proves that any finite time trajectory of a given dynamical time-variant system can be approximated by the internal state of a continuous-time recurrent neural network. Given several special forms of dynamical time-variant systems or trajectories, this paper shows that they can all be approximately realized by the internal state of a simple recurrent neural network.

Proceedings ArticleDOI
27 Jun 2005
TL;DR: A general neural system for matrix inversion is presented which can be constructed by using monotonically-increasing odd activation functions, and an application example on inverse-kinematic control of redundant manipulators via online pseudoinverse solution is presented.
Abstract: As inspired by revising (Zhang and Ge, 2003), the traditional gradient-based neural system (also termed analog computer (Manherz et al., 1968)) for matrix inversion is re-visited by examining different activation functions and various implementation errors. A general neural system for matrix inversion is thus presented which can be constructed by using monotonically-increasing odd activation functions. For superior convergence and robustness of such a system, the power-sigmoid activation function is preferred to be in use if the hardware permits. In addition to investigating the singular case, this paper also presents an application example on inverse-kinematic control of redundant manipulators via online pseudoinverse solution

Journal ArticleDOI
TL;DR: A novel evolutionary method for learning DFA that evolves only the transition matrix and uses a simple deterministic procedure to optimally assign state labels is described, which consistently outperforms EDSM, as well as other significant methods submitted to two recent competitions.
Abstract: Learning a deterministic finite automaton (DFA) from a training set of labeled strings is a hard task that has been much studied within the machine learning community. It is equivalent to learning a regular language by example and has applications in language modeling. In this paper, we describe a novel evolutionary method for learning DFA that evolves only the transition matrix and uses a simple deterministic procedure to optimally assign state labels. We compare its performance with the evidence driven state merging (EDSM) algorithm, one of the most powerful known DFA learning algorithms. We present results on random DFA induction problems of varying target size and training set density. We also study the effects of noisy training data on the evolutionary approach and on EDSM. On noise-free data, we find that our evolutionary method outperforms EDSM on small sparse data sets. In the case of noisy training data, we find that our evolutionary method consistently outperforms EDSM, as well as other significant methods submitted to two recent competitions.