Showing papers on "Recurrent neural network published in 2006"
TL;DR: A new learning algorithm called ELM is proposed for feedforward neural networks (SLFNs) which randomly chooses hidden nodes and analytically determines the output weights of SLFNs which tends to provide good generalization performance at extremely fast learning speed.
10,217 citations
25 Jun 2006
TL;DR: This paper presents a novel method for training RNNs to label unsegmented sequences directly, thereby solving both problems of sequence learning and post-processing.
Abstract: Many real-world sequence learning tasks require the prediction of sequences of labels from noisy, unsegmented input data. In speech recognition, for example, an acoustic signal is transcribed into words or sub-word units. Recurrent neural networks (RNNs) are powerful sequence learners that would seem well suited to such tasks. However, because they require pre-segmented training data, and post-processing to transform their outputs into label sequences, their applicability has so far been limited. This paper presents a novel method for training RNNs to label unsegmented sequences directly, thereby solving both problems. An experiment on the TIMIT speech corpus demonstrates its advantages over both a baseline HMM and a hybrid HMM-RNN.
5,188 citations
TL;DR: A linear matrix inequality (LMI) approach is developed to establish sufficient conditions for the RNNs to be globally exponentially stable, and the existence and uniqueness of the equilibrium point under mild conditions is proved.
671 citations
TL;DR: Simulation results demonstrate that the recurrent models, trained by the suggested methods, outperform the static ones while they exhibit significant improvement over the persistent method.
Abstract: This paper deals with the problem of long-term wind speed and power forecasting based on meteorological information. Hourly forecasts up to 72-h ahead are produced for a wind park on the Greek island of Crete. As inputs our models use the numerical forecasts of wind speed and direction provided by atmospheric modeling system SKIRON for four nearby positions up to 30 km away from the wind turbine cluster. Three types of local recurrent neural networks are employed as forecasting models, namely, the infinite impulse response multilayer perceptron (IIR-MLP), the local activation feedback multilayer network (LAF-MLN), and the diagonal recurrent neural network (RNN). These networks contain internal feedback paths, with the neuron connections implemented by means of IIR synaptic filters. Two novel and optimal on-line learning schemes are suggested for the update of the recurrent network's weights based on the recursive prediction error algorithm. The methods assure continuous stability of the network during the learning phase and exhibit improved performance compared to the conventional dynamic back propagation. Extensive experimentation is carried out where the three recurrent networks are additionally compared to two static models, a finite-impulse response NN (FIR-NN) and a conventional static-MLP network. Simulation results demonstrate that the recurrent models, trained by the suggested methods, outperform the static ones while they exhibit significant improvement over the persistent method.
534 citations
TL;DR: In this article, the global exponential stability analysis problem for a class of recurrent neural networks (RNNs) with time delays and Markovian jumping parameters is considered, and a new Lyapunov-Krasovskii functional is developed to establish the desired sufficient conditions, which can be easily checked by utilizing the numerically efficient Matlab LMI toolbox.
344 citations
TL;DR: An alternative model is presented, according to which sequence information is encoded through sustained patterns of activation within a recurrent neural network architecture, which provides a parsimonious account for numerous benchmark characteristics of immediate serial recall.
Abstract: Despite a century of research, the mechanisms underlying short-term or working memory for serial order remain uncertain. Recent theoretical models have converged on a particular account, based on transient associations between independent item and context representations. In the present article, the authors present an alternative model, according to which sequence information is encoded through sustained patterns of activation within a recurrent neural network architecture. As demonstrated through a series of computer simulations, the model provides a parsimonious account for numerous benchmark characteristics of immediate serial recall, including data that have been considered to preclude the application of recurrent neural networks in this domain. Unlike most competing accounts, the model deals naturally with findings concerning the role of background knowledge in serial recall and makes contact with relevant neuroscientific data. Furthermore, the model gives rise to numerous testable predictions that differentiate it from competing theories. Taken together, the results presented indicate that recurrent neural networks may offer a useful framework for understanding short-term memory for serial order.
340 citations
TL;DR: The proposed Lyapunov-Krasovskii functional and linear matrix inequality result is computationally efficient as it can be solved numerically using standard commercial software.
Abstract: By employing the Lyapunov-Krasovskii functional and linear matrix inequality (LMI) approach, the problem of global asymptotical stability is studied for recurrent neural networks with both discrete time-varying delays and distributed time-varying delays. Some sufficient conditions are given for checking the global asymptotical stability of recurrent neural networks with mixed time-varying delay. The proposed LMI result is computationally efficient as it can be solved numerically using standard commercial software. Two examples are given to show the usefulness of the results
296 citations
TL;DR: It is shown theoretically that the recently developed extreme learning machine (ELM) algorithm can be used to train the neural networks with threshold functions directly instead of approximating them with sigmoid functions.
Abstract: Neural networks with threshold activation functions are highly desirable because of the ease of hardware implementation. However, the popular gradient-based learning algorithms cannot be directly used to train these networks as the threshold functions are nondifferentiable. Methods available in the literature mainly focus on approximating the threshold activation functions by using sigmoid functions. In this paper, we show theoretically that the recently developed extreme learning machine (ELM) algorithm can be used to train the neural networks with threshold functions directly instead of approximating them with sigmoid functions. Experimental results based on real-world benchmark regression problems demonstrate that the generalization performance obtained by ELM is better than other algorithms used in threshold networks. Also, the ELM method does not need control variables (manually tuned parameters) and is much faster.
268 citations
10 Sep 2006
TL;DR: A proof for the universal approximation ability of RNN in state space model form and even extend it to Error Correction and Normalized Recurrent Neural Networks is given.
Abstract: Neural networks represent a class of functions for the efficient identification and forecasting of dynamical systems. It has been shown that feedforward networks are able to approximate any (Borel-)measurable function on a compact domain [1,2,3]. Recurrent neural networks (RNNs) have been developed for a better understanding and analysis of open dynamical systems. Compared to feedforward networks they have several advantages which have been discussed extensively in several papers and books, e.g. [4]. Still the question often arises if RNNs are able to map every open dynamical system, which would be desirable for a broad spectrum of applications. In this paper we give a proof for the universal approximation ability of RNNs in state space model form. The proof is based on the work of Hornik, Stinchcombe, and White about feedforward neural networks [1].
239 citations
14 May 2006
TL;DR: This paper deals with phoneme recognition based on neural networks (NN), and focuses on temporal patterns (TRAPs) and novel split temporal context (STC) phoneme recognizers and investigates into tandem NN architectures.
Abstract: This paper deals with phoneme recognition based on neural networks (NN). First, several approaches to improve the phoneme error rate are suggested and discussed. In the experimental part, we concentrate on TempoRAl Patterns (TRAPs) and novel split temporal context (STC) phoneme recognizers. We also investigate into tandem NN architectures. The results of the final system reported on standard TIMIT database compare favorably to the best published results.
236 citations
TL;DR: A stable neural network (NN)-based observer for general multivariable nonlinear systems is presented in this paper and the stability of the recurrent neural network observer is shown by Lyapunov's direct method.
Abstract: A stable neural network (NN)-based observer for general multivariable nonlinear systems is presented in this paper. Unlike most previous neural network observers, the proposed observer uses a nonlinear-in-parameters neural network (NLPNN). Therefore, it can be applied to systems with higher degrees of nonlinearity without any a priori knowledge about system dynamics. The learning rule for the neural network is a novel approach based on the modified backpropagation (BP) algorithm. An e-modification term is added to guarantee robustness of the observer. No strictly positive real (SPR) or any other strong assumption is imposed on the proposed approach. The stability of the recurrent neural network observer is shown by Lyapunov's direct method. Simulation results for a flexible-joint manipulator are presented to demonstrate the enhanced performance achieved by utilizing the proposed neural network observer.
TL;DR: The simplified dual neural network is shown to be globally convergent to the exact optimal solution of k-winners-take-all (KWTA) operation.
Abstract: The design, analysis, and application of a new recurrent neural network for quadratic programming, called simplified dual neural network, are discussed. The analysis mainly concentrates on the convergence property and the computational complexity of the neural network. The simplified dual neural network is shown to be globally convergent to the exact optimal solution. The complexity of the neural network architecture is reduced with the number of neurons equal to the number of inequality constraints. Its application to k-winners-take-all (KWTA) operation is discussed to demonstrate how to solve problems with this neural network
TL;DR: It is shown that the projection neural network can also be used to solve pseudomonotone variational inequalities and related pseudoconvex optimization problems, and a new concept, called componentwise pseudomononicity, different from pseudomon onicity in general is introduced.
Abstract: In recent years, a recurrent neural network called projection neural network was proposed for solving monotone variational inequalities and related convex optimization problems. In this paper, we show that the projection neural network can also be used to solve pseudomonotone variational inequalities and related pseudoconvex optimization problems. Under various pseudomonotonicity conditions and other conditions, the projection neural network is proved to be stable in the sense of Lyapunov and globally convergent, globally asymptotically stable, and globally exponentially stable. Since monotonicity is a special case of pseudomononicity, the projection neural network can be applied to solve a broader class of constrained optimization problems related to variational inequalities. Moreover, a new concept, called componentwise pseudomononicity, different from pseudomononicity in general, is introduced. Under this new concept, two stability results of the projection neural network for solving variational inequalities are also obtained. Finally, numerical examples show the effectiveness and performance of the projection neural network
01 Oct 2006
TL;DR: Results obtained using long short-term memory RNNs trained by the recent Evolino algorithm show that this approach can significantly increase the efficiency of suture knot tying in MIS over preprogrammed control.
Abstract: Tying suture knots is a time-consuming task performed frequently during Minimally Invasive Surgery (MIS). Automating this task could greatly reduce total surgery time for patients. Current solutions to this problem replay manually programmed trajectories, but a more general and robust approach is to use supervised machine learning to smooth surgeon-given training trajectories and generalize from them. Since knottying generally requires a controller with internal memory to distinguish between identical inputs that require different actions at different points along a trajectory, it would be impossible to teach the system using traditional feedforward neural nets or support vector machines. Instead we exploit more powerful, recurrent neural networks (RNNs) with adaptive internal states. Results obtained using LSTM RNNs trained by the recent Evolino algorithm show that this approach can significantly increase the efficiency of suture knot tying in MIS over preprogrammed control.
TL;DR: The proposed procedure can be applied to construct controllers with arbitrary structures, such as optimal reduced-order controllers and decentralized controllers, to minimize a control-relevant cost function.
TL;DR: In this paper, the authors investigated the existence of multiple stable stationary solutions for Hopfield-type neural networks with delay and without delay and established a scenario of dynamics through formulating parameter conditions based on a geometrical setting.
Abstract: Stable stationary solutions correspond to memory capacity in the application of associative memory for neural networks. In this presentation, existence of multiple stable stationary solutions for Hopfield-type neural networks with delay and without delay is investigated. Basins of attraction for these stationary solutions are also estimated. Such a scenario of dynamics is established through formulating parameter conditions based on a geometrical setting. The present theory is demonstrated by two numerical simulations on the Hopfield neural networks with delays.
TL;DR: Using new theoretical results, the global exponential stability of recurrent neural networks can be derived, and the estimated location of the equilibrium point can be obtained.
Abstract: This paper presents new theoretical results on global exponential stability of recurrent neural networks with bounded activation functions and time-varying delays. The stability conditions depend on external inputs, connection weights, and time delays of recurrent neural networks. Using these results, the global exponential stability of recurrent neural networks can be derived, and the estimated location of the equilibrium point can be obtained. As typical representatives, the Hopfield neural network (HNN) and the cellular neural network (CNN) are examined in detail
Journal Article•
TL;DR: In this paper, the universal approximation ability of recurrent neural networks in state space model form was proved based on the work of Hornik, Stinchcombe, and White about feed-forward neural networks.
Abstract: Neural networks represent a class of functions for the efficient identification and forecasting of dynamical systems. It has been shown that feedforward networks are able to approximate any (Borel-)measurable function on a compact domain [1,2,3]. Recurrent neural networks (RNNs) have been developed for a better understanding and analysis of open dynamical systems. Compared to feed-forward networks they have several advantages which have been discussed extensively in several papers and books, e.g. [4]. Still the question often arises if RNNs are able to map every open dynamical system, which would be desirable for a broad spectrum of applications. In this paper we give a proof for the universal approximation ability of RNNs in state space model form. The proof is based on the work of Hornik, Stinchcombe, and White about feedforward neural networks [1].
TL;DR: This study presents experiments on the learning of object handling behaviors by a small humanoid robot using a dynamic neural network model, the recurrent neural network with parametric bias (RNNPB), and showed that entrainment of the internal memory structures of the RNNPB through the interactions of the objects and the human supporters are the essential mechanisms for those observed situated behaviors of the robot.
16 Jul 2006
TL;DR: A straightforward improvement of the previous LSM-based implementation is presented that results in an outperformance of a state-of-the-art Hidden Markov Model (HMM) based recognizer.
Abstract: A solution for the slow convergence of most learning rules for Recurrent Neural Networks (RNN) has been proposed under the terms Liquid State Machines (LSM) and Echo State Networks (ESN). These methods use a RNN as a reservoir that is not trained. For this article we build upon previous work, where we used reservoir-based techniques to solve the task of isolated digit recognition. We present a straightforward improvement of our previous LSM-based implementation that results in an outperformance of a state-of-the-art Hidden Markov Model (HMM) based recognizer. Also, we apply the Echo State approach to the problem, which allows us to investigate the impact of several interconnection parameters on the performance of our speech recognizer.
TL;DR: Experimental results on the wind prediction problem demonstrate that the proposed algorithms exhibit enhanced performance, in terms of convergence speed and the accuracy of the attained solutions, compared to conventional gradient-based methods.
TL;DR: An artificial neural network, namely recurrent neural network with rprop update algorithm and is applied for wave forecasting and shows that the wave forecasting using recurrent neuralnetwork yields better results than the previous neural network application.
TL;DR: This paper presents a novel approach for the simultaneous modelling and forecasting of wind signal components by using novel neural network algorithms and architectures based on modular complex-valued recurrent neural networks (RNNs).
TL;DR: A wavelet multiscale decomposition-based autoregressive approach for the prediction of 1-h ahead load based on historical electricity load data, based on a multiple resolution decomposition of the signal using the non-decimated or redundant Haar a trous wavelet transform whose advantage is taking into account the asymmetric nature of the time-varying data.
TL;DR: In this article, the authors considered both exponential stability and periodic oscillatory solutions for reaction-diffusion recurrent neural networks with continuously distributed delays and provided sufficient conditions to ensure global exponential stability.
Abstract: Both exponential stability and periodic oscillatory solutions are considered for reaction–diffusion recurrent neural networks with continuously distributed delays. By constructing suitable Lyapunov functional, using M-matrix theory and some analysis techniques, some simple sufficient conditions are given ensuring the global exponential stability and the existence of periodic oscillatory solutions for reaction–diffusion recurrent neural networks with continuously distributed delays. Moreover, the exponential convergence rate is estimated. These results have leading significance in the design and applications of globally exponentially stable and periodic oscillatory neural circuits for reaction–diffusion recurrent neural networks with continuously distributed delays. Two examples are given to illustrate the correctness of the obtained results.
TL;DR: This paper presents a fully automated recurrent neural network (FARNN) that is capable of self-structuring its network in a minimal representation with satisfactory performance for unknown dynamic system identification and control.
Abstract: This paper presents a fully automated recurrent neural network (FARNN) that is capable of self-structuring its network in a minimal representation with satisfactory performance for unknown dynamic system identification and control. A novel recurrent network, consisting of a fully-connected single-layer neural network and a feedback interconnected dynamic network, was developed to describe an unknown dynamic system as a state-space representation. Next, a fully automated construction algorithm was devised to construct a minimal state-space representation with the essential dynamics captured from the input-output measurements of the unknown system. The construction algorithm integrates the methods of minimal model determination, parameter initialization and performance optimization into a systematic framework that totally exempt trial-and-error processes on the selections of network sizes and parameters. Computer simulations on benchmark examples of unknown nonlinear dynamic system identification and control have successfully validated the effectiveness of the proposed FARNN in constructing a parsimonious network with superior performance
TL;DR: In this paper, sufficient conditions are obtained for ascertaining the global asymptotic stability of the unique equilibrium of the network based on linear matrix inequality (LMI), which can be checked easily by various recently developed convex optimization algorithms.
Abstract: This paper investigates the stability for a delayed recurrent neural network. Sufficient conditions are obtained for ascertaining the global asymptotic stability of the unique equilibrium of the network based on LMI technique. The results are computationally efficient, since they are in the form of linear matrix inequality (LMI), which can be checked easily by various recently developed convex optimization algorithms. Besides, the analysis approach allows one to consider different types of activation functions, such as piecewise linear, sigmoids with bounded activations as well as C1-smooth sigmoids. In the end of this paper two illustrative examples are also provided to show the effectiveness of our results.
TL;DR: It is shown that the Cohen-Grossberg neural network is globally exponentially stable, if the absolute value of the input vector exceeds a criterion.
TL;DR: This letter begins a systematic study of the global parameter space structure of continuous-time recurrent neural networks (CTRNNs), a class of neural models that is simple but dynamically universal.
Abstract: A fundamental challenge for any general theory of neural circuits is how to characterize the structure of the space of all possible circuits over a given model neuron. As a first step in this direction, this letter begins a systematic study of the global parameter space structure of continuous-time recurrent neural networks (CTRNNs), a class of neural models that is simple but dynamically universal. First, we explicitly compute the local bifurcation manifolds of CTRNNs. We then visualize the structure of these manifolds in net input space for small circuits. These visualizations reveal a set of extremal saddle node bifurcation manifolds that divide CTRNN parameter space into regions of dynamics with different effective dimensionality. Next, we completely characterize the combinatorics and geometry of an asymptotically exact approximation to these regions for circuits of arbitrary size. Finally, we show how these regions can be used to calculate estimates of the probability of encountering different kinds of dynamics in CTRNN parameter space.
01 Apr 2006
TL;DR: Using the proposed ARNN motion control system for a biaxial motion mechanism driven by two field-oriented control permanent magnet synchronous motors in the CNC machine, the position tracking performance is substantially improved and the robustness to uncertainties including cross-coupled interference and friction torque can be obtained.
Abstract: In this paper, an adaptive recurrent-neural-network (ARNN) motion control system for a biaxial motion mechanism driven by two field-oriented control permanent magnet synchronous motors (PMSMs) in the computer numerical control (CNC) machine is proposed. In the proposed ARNN control system, a RNN with accurate approximation capability is employed to approximate an unknown dynamic function, and the adaptive learning algorithms that can learn the parameters of the RNN on line are derived using Lyapunov stability theorem. Moreover, a robust controller is proposed to confront the uncertainties including approximation error, optimal parameter vectors, higher-order terms in Taylor series, external disturbances, cross-coupled interference and friction torque of the system. To relax the requirement for the value of lumped uncertainty in the robust controller, an adaptive lumped uncertainty estimation law is investigated. Using the proposed control, the position tracking performance is substantially improved and the robustness to uncertainties including cross-coupled interference and friction torque can be obtained as well. Finally, some experimental results of the tracking of various reference contours demonstrate the validity of the proposed design for practical applications.