Showing papers on "Recurrent neural network published in 2004"
TL;DR: A method for learning nonlinear systems, echo state networks (ESNs), which employ artificial recurrent neural networks in a way that has recently been proposed independently as a learning mechanism in biological brains is presented.
Abstract: We present a method for learning nonlinear systems, echo state networks (ESNs). ESNs employ artificial recurrent neural networks in a way that has recently been proposed independently as a learning mechanism in biological brains. The learning method is computationally efficient and easy to use. On a benchmark task of predicting a chaotic time series, accuracy is improved by a factor of 2400 over previous techniques. The potential for engineering applications is illustrated by equalizing a communication channel, where the signal error rate is improved by two orders of magnitude.
3,122 citations
01 Apr 2004
TL;DR: An evolutionary recurrent network which automates the design of recurrent neural/fuzzy networks using a new evolutionary learning algorithm based on a hybrid of genetic algorithm (GA) and particle swarm optimization (PSO), and is thus called HGAPSO.
Abstract: An evolutionary recurrent network which automates the design of recurrent neural/fuzzy networks using a new evolutionary learning algorithm is proposed in this paper. This new evolutionary learning algorithm is based on a hybrid of genetic algorithm (GA) and particle swarm optimization (PSO), and is thus called HGAPSO. In HGAPSO, individuals in a new generation are created, not only by crossover and mutation operation as in GA, but also by PSO. The concept of elite strategy is adopted in HGAPSO, where the upper-half of the best-performing individuals in a population are regarded as elites. However, instead of being reproduced directly to the next generation, these elites are first enhanced. The group constituted by the elites is regarded as a swarm, and each elite corresponds to a particle within it. In this regard, the elites are enhanced by PSO, an operation which mimics the maturing phenomenon in nature. These enhanced elites constitute half of the population in the new generation, whereas the other half is generated by performing crossover and mutation operation on these enhanced elites. HGAPSO is applied to recurrent neural/fuzzy network design as follows. For recurrent neural network, a fully connected recurrent neural network is designed and applied to a temporal sequence production problem. For recurrent fuzzy network design, a Takagi-Sugeno-Kang-type recurrent fuzzy network is designed and applied to dynamic plant control. The performance of HGAPSO is compared to both GA and PSO in these recurrent networks design problems, demonstrating its superiority.
961 citations
TL;DR: It is shown that only near the critical boundary can recurrent networks of threshold gates perform complex computations on time series, which strongly supports conjectures that dynamical systems that are capable of doing complex computational tasks should operate near the edge of chaos.
Abstract: Depending on the connectivity, recurrent networks of simple computational units can show very different types of dynamics, ranging from totally ordered to chaotic. We analyze how the type of dynamics (ordered or chaotic) exhibited by randomly connected networks of threshold gates driven by a time-varying input signal depends on the parameters describing the distribution of the connectivity matrix. In particular, we calculate the critical boundary in parameter space where the transition from ordered to chaotic dynamics takes place. Employing a recently developed framework for analyzing real-time computations, we show that only near the critical boundary can such networks perform complex computations on time series. Hence, this result strongly supports conjectures that dynamical systems that are capable of doing complex computational tasks should operate near the edge of chaos, that is, the transition from ordered to chaotic dynamics.
746 citations
TL;DR: Empirical results indicate that HFM is relatively less accurate and RBFN is relatively more reliable for the weather forecasting problem, while the ensemble of neural networks produced the most accurate forecasts.
Abstract: This study presents the applicability of an ensemble of artificial neural networks (ANNs) and learning paradigms for weather forecasting in southern Saskatchewan, Canada. The proposed ensemble method for weather forecasting has advantages over other techniques like linear combination. Generally, the output of an ensemble is a weighted sum, which are weight-fixed, with the weights being determined from the training or validation data. In the proposed approach, weights are determined dynamically from the respective certainties of the network outputs. The more certain a network seems to be of its decision, the higher the weight. The proposed ensemble model performance is contrasted with multi-layered perceptron network (MLPN), Elman recurrent neural network (ERNN), radial basis function network (RBFN), Hopfield model (HFM) predictive models and regression techniques. The data of temperature, wind speed and relative humidity are used to train and test the different models. With each model, 24-h-ahead forecasts are made for the winter, spring, summer and fall seasons. Moreover, the performance and reliability of the seven models are then evaluated by a number of statistical measures. Among the direct approaches employed, empirical results indicate that HFM is relatively less accurate and RBFN is relatively more reliable for the weather forecasting problem. In comparison, the ensemble of neural networks produced the most accurate forecasts.
299 citations
TL;DR: In this paper, a sufficient condition for the existence, uniqueness, and global robust stability of equilibria for interval neural networks with time delays by constructing Lyapunov functional and using matrix-norm inequality is presented.
Abstract: This paper is concerned with the global robust stability of a class of delayed interval recurrent neural networks which contain time-invariant uncertain parameters whose values are unknown but bounded in given compact sets. A new sufficient condition is presented for the existence, uniqueness, and global robust stability of equilibria for interval neural networks with time delays by constructing Lyapunov functional and using matrix-norm inequality. An error is corrected in an earlier publication, and an example is given to show the effectiveness of the obtained results.
266 citations
TL;DR: Under various mild conditions, the proposed general projection neural network is shown to be globally convergent, globally asymptotically stable, and globally exponentially stable.
Abstract: Recently, a projection neural network for solving monotone variational inequalities and constrained optimization problems was developed. In this paper, we propose a general projection neural network for solving a wider class of variational inequalities and related optimization problems. In addition to its simple structure and low complexity, the proposed neural network includes existing neural networks for optimization, such as the projection neural network, the primal-dual neural network, and the dual neural network, as special cases. Under various mild conditions, the proposed general projection neural network is shown to be globally convergent, globally asymptotically stable, and globally exponentially stable. Furthermore, several improved stability criteria on two special cases of the general projection neural network are obtained under weaker conditions. Simulation results demonstrate the effectiveness and characteristics of the proposed neural network.
254 citations
TL;DR: A connectionist model, the recurrent neural network with parametric biases (RNNPB), in which multiple behavior schemata can be learned by the network in a distributed manner is reviewed, explaining how self-organizing internal structures can contribute to generalization in learning, and diversity in behavior generation, in the proposed distributed representation scheme.
236 citations
TL;DR: This paper studies a new methodology to model and predict chaotic time series based on a new recurrent predictor neural network (RPNN) that realizes long-term prediction by making accurate multistep predictions.
Abstract: Chaos limits predictability so that the long-term prediction of chaotic time series is very difficult. The main purpose of this paper is to study a new methodology to model and predict chaotic time series based on a new recurrent predictor neural network (RPNN). This method realizes long-term prediction by making accurate multistep predictions. This RPNN consists of nonlinearly operated nodes whose outputs are only connected with the inputs of themselves and the latter nodes. The connections may contain multiple branches with time delays. An extended algorithm of self-adaptive back-propagation through time (BPTT) learning algorithm is used to train the RPNN. In simulation, two performance measures [root-mean-square error (RMSE) and prediction accuracy (PA)] show that the proposed method is more effective and accurate for multistep prediction. It can identify the systems characteristics quite well and provide a new way to make long-term prediction of the chaotic time series.
233 citations
TL;DR: The results herein answer a question: if a neural network without any delay is absolutely exponentially stable, then under what additional conditions, the neural networks with delay is alsoabsolutely exponentially stable.
220 citations
25 Jul 2004
TL;DR: A new learning rule for fully recurrent neural networks is introduced which combines important principles: one-step backpropagation of errors and the usage of temporal memory in the network dynamics by means of decorrelation of activations.
Abstract: We introduce a new learning rule for fully recurrent neural networks which we call backpropagation-decorrelation rule (BPDC). It combines important principles: one-step backpropagation of errors and the usage of temporal memory in the network dynamics by means of decorrelation of activations. The BPDC rule is derived and theoretically justified from regarding learning as a constraint optimization problem and applies uniformly in discrete and continuous time. It is very easy to implement, and has a minimal complexity of 2N multiplications per time-step in the single output case. Nevertheless we obtain fast tracking and excellent performance in some benchmark problems including the Mackey-Glass time-series.
217 citations
TL;DR: Recurrent neural networks were used to train and forecast the monthly flows of a river in India, with a catchment area of 5189 km2 up to the gauging site, and performed better than the feed forward networks.
Abstract: Forecasting a hydrologic time series has been one of the most complicated tasks owing to the wide range of data, the uncertainties in the parameters influencing the time series and also due to the non availability of adequate data. Recently, Artificial Neural Networks (ANNs) have become quite popular in time series forecasting in various fields. This paper demonstrates the use of ANNs to forecast monthly river flows. Two different networks, namely the feed forward network and the recurrent neural network, have been chosen. The feed forward network is trained using the conventional back propagation algorithm with many improvements and the recurrent neural network is trained using the method of ordered partial derivatives. The selection of architecture and the training procedure for both the networks are presented. The selected ANN models were used to train and forecast the monthly flows of a river in India, with a catchment area of 5189 km2 up to the gauging site. The trained networks are used for both single step ahead and multiple step ahead forecasting. A comparative study of both networks indicates that the recurrent neural networks performed better than the feed forward networks. In addition, the size of the architecture and the training time required were less for the recurrent neural networks. The recurrent neural network gave better results for both single step ahead and multiple step ahead forecasting. Hence recurrent neural networks are recommended as a tool for river flow forecasting.
TL;DR: It is shown that the memory capacity of linear recurrent networks obeying discrete time dynamics scales with system size, and this temporal memory capacity for both distributed shift register and random orthogonal connectivity matrices is calculated.
Abstract: We study the ability of linear recurrent networks obeying discrete time dynamics to store long temporal sequences that are retrievable from the instantaneous state of the network. We calculate this temporal memory capacity for both distributed shift register and random orthogonal connectivity matrices. We show that the memory capacity of these networks scales with system size.
01 Feb 2004
TL;DR: A recurrent neural network is developed and applied for kinematic control of redundant manipulators with obstacle avoidance capability and an improved problem formulation is proposed that the collision-avoidance requirement is represented by dynamically-updated inequality constraints.
Abstract: One important issue in the motion planning and control of kinematically redundant manipulators is the obstacle avoidance. In this paper, a recurrent neural network is developed and applied for kinematic control of redundant manipulators with obstacle avoidance capability. An improved problem formulation is proposed in the sense that the collision-avoidance requirement is represented by dynamically-updated inequality constraints. In addition, physical constraints such as joint physical limits are also incorporated directly into the formulation. Based on the improved problem formulation, a dual neural network is developed for the online solution to collision-free inverse kinematics problem. The neural network is simulated for motion control of the PA10 robot arm in the presence of point and window-shaped obstacle.
TL;DR: This paper elaborate upon the claim that clustering in the recurrent layer of recurrent neural networks (RNNs) reflects meaningful information processing states even prior to training.
Abstract: In this paper, we elaborate upon the claim that clustering in the recurrent layer of recurrent neural networks (RNNs) reflects meaningful information processing states even prior to training. By concentrating on activation clusters in RNNs, while not throwing away the continuous state space network dynamics, we extract predictive models that we call neural prediction machines (NPMs). When RNNs with sigmoid activation functions are initialized with small weights (a common technique in the RNN community), the clusters of recurrent activations emerging prior to training are indeed meaningful and correspond to Markov prediction contexts. In this case, the extracted NPMs correspond to a class of Markov models, called variable memory length Markov models (VLMMs). In order to appreciate how much information has really been induced during the training, the RNN performance should always be compared with that of VLMMs and NPMs extracted before training as the "null" base models. Our arguments are supported by experiments on a chaotic symbolic sequence and a context-free language with a deep recursive structure.
TL;DR: In this article, an input-to-state stability approach is applied to access robust training algorithms of discrete-time recurrent neural networks and the authors conclude that for nonlinear system identification, the gradient descent law and the backpropagation-like algorithm for the weights adjustment are stable in the sense of L∞ and robust to any bounded uncertainties.
25 Jul 2004
TL;DR: An architecture which automates the design of recurrent neural networks using a new evolutionary learning algorithm based on a hybrid of particle swarm optimization (PSO) and evolutionary algorithm (EA) is applied.
Abstract: To predict the 100 missing values from the time series consisting of 5000 data given for the IJCNN 2004 time series prediction competition, we applied an architecture which automates the design of recurrent neural networks using a new evolutionary learning algorithm. This new evolutionary learning algorithm is based on a hybrid of particle swarm optimization (PSO) and evolutionary algorithm (EA). By combining the searching abilities of these two global optimization methods, the evolution of individuals is no longer restricted to be in the same generation, and better performed individuals may produce offspring to replace those with poor performance. The novel algorithm is then applied to the recurrent neural network for the time series prediction. The experimental results show that our approach gives good performance in predicting the missing values from the time series.
TL;DR: This paper presents a recurrent neural network for solving strict convex quadratic programming problems and related linear piecewise equations with a one-layer structure with a low model complexity and shown to have a finite-time convergence and exponential convergence.
TL;DR: This paper presents meta-learning evolutionary artificial neural network (MLEANN), an automatic computational framework for the adaptive optimization of artificial neural networks (ANNs) wherein the neural network architecture, activation function, connection weights; learning algorithm and its parameters are adapted according to the problem.
TL;DR: Simulation results demonstrate that the proposed design method can achieve favorable control performance for the wing rock system without the knowledge of system dynamic functions.
Abstract: Wing rock is a highly nonlinear phenomenon in which an aircraft undergoes limit cycle roll oscillations at high angles of attack. In this paper, a supervisory recurrent fuzzy neural network control (SRFNNC) system is developed to control the wing rock system. This SRFNNC system is comprised of a recurrent fuzzy neural network (RFNN) controller and a supervisory controller. The RFNN controller is investigated to mimic an ideal controller and the supervisory controller is designed to compensate for the approximation error between the RFNN controller and the ideal controller. The RFNN is inherently a recurrent multilayered neural network for realizing fuzzy inference using dynamic fuzzy rules. Moreover, an on-line parameter training methodology, using the gradient descent method and the Lyapunov stability theorem, is proposed to increase the learning capability. Finally, a comparison between the sliding-mode control, the fuzzy sliding control and the proposed SRFNNC of a wing rock system is presented to illustrate the effectiveness of the SRFNNC system. Simulation results demonstrate that the proposed design method can achieve favorable control performance for the wing rock system without the knowledge of system dynamic functions.
TL;DR: The comparison of the neuron state modeling and the local field modeling reveals an important stability invariance property of the two models, which lays a sound theoretical foundation of validity of a useful, cross-fertilization type stability analysis methodology for various neural network models.
TL;DR: It is proposed that the dynamics of coherence and incoherence between the robot's and the user’s movements could enhance close interactions between them, and that they could also explain the essential psychological mechanism of joint attention.
Abstract: This study presents experiments on the imitative interactions between a small humanoid robot and a user. A dynamic neural network model of a mirror system was implemented in a humanoid robot, based...
Patent•
06 Dec 2004TL;DR: In this article, a recurrent neural network is trained by inputting building thermal mass data and building occupancy data for actual weather conditions and comparing the predicted thermal load generated by the recurrent neural networks to the actual thermal load measured at the building.
Abstract: A system for forecasting predicted thermal loads for a building comprises a thermal condition forecaster for forecasting weather conditions to be compensated by a building environmental control system and a thermal load predictor for modeling building environmental management system components to generate a predicted thermal load for a building for maintaining a set of environmental conditions. The thermal load predictor of the present invention is a neural network and, preferably, the neural network is a recurrent neural network that generates the predicted thermal load from short-term data. The recurrent neural network is trained by inputting building thermal mass data and building occupancy data for actual weather conditions and comparing the predicted thermal load generated by the recurrent neural network to the actual thermal load measured at the building. Training error is attributed to weights of the neurons processing the building thermal mass data and building occupancy data. Iteratively adjusting these weights to minimize the error optimizes the design of the recurrent neural network for these non-weather inputs.
TL;DR: Using the local inhibition, conditions for nondivergence are derived, which not only guarantee nondiversgence, but also allow for the existence of multiequilibrium points.
Abstract: This paper studies the multistability of a class of discrete-time recurrent neural networks with unsaturating piecewise linear activation functions. It addresses the nondivergence, global attractivity, and complete stability of the networks. Using the local inhibition, conditions for nondivergence are derived, which not only guarantee nondivergence, but also allow for the existence of multiequilibrium points. Under these nondivergence conditions, global attractive compact sets are obtained. Complete stability is studied via constructing novel energy functions and using the well-known Cauchy Convergence Principle. Examples and simulation results are used to illustrate the theory.
TL;DR: In this article, a robust global exponential stability analysis for interval recurrent neural networks (RNNs) via the linear matrix inequality (LMI) approach is proposed. But the authors assume that the values of the time-invariant uncertain parameters are bounded within given compact sets.
01 Apr 2004
TL;DR: This paper presents a one-layer recurrent neural network for support vector machine (SVM) learning in pattern classification and regression that can converge exponentially to the optimal solution of SVM learning.
Abstract: This paper presents a one-layer recurrent neural network for support vector machine (SVM) learning in pattern classification and regression. The SVM learning problem is first converted into an equivalent formulation, and then a one-layer recurrent neural network for SVM learning is proposed. The proposed neural network is guaranteed to obtain the optimal solution of support vector classification and regression. Compared with the existing two-layer neural network for the SVM classification, the proposed neural network has a low complexity for implementation. Moreover, the proposed neural network can converge exponentially to the optimal solution of SVM learning. The rate of the exponential convergence can be made arbitrarily high by simply turning up a scaling parameter. Simulation examples based on benchmark problems are discussed to show the good performance of the proposed neural network for SVM learning.
TL;DR: A novel approach in designing neural network based adaptive controllers for a class of nonlinear discrete-time systems with simplicity in parallelism to linear generalized minimum variance (GMV) controller design and efficiency to deal with complex nonlinear dynamics.
Abstract: This paper presents a novel approach in designing neural network based adaptive controllers for a class of nonlinear discrete-time systems. This type of controllers has its simplicity in parallelism to linear generalized minimum variance (GMV) controller design and efficiency to deal with complex nonlinear dynamics. A recurrent neural network is introduced as a bridge to compensation simplify controller design procedure and efficiently to deal with nonlinearity. The network weight adaptation law is derived from Lyapunov stability analysis and the connection between convergence of the network weight and the reconstruction error of the network is established. A theorem is presented for the conditions of the stability of the closed-loop systems. Two simulation examples are provided to demonstrate the efficiency of the approach.
TL;DR: The model is computed as a flexible multi-layer feed-forward neural network and includes a modified probabilistic incremental program evolution algorithm (MPIPE) to evolve and determine a optimal structure of the neural tree and a parameter learning algorithm to optimize the free parameters embedded in the Neural tree.
Abstract: This paper introduces a flexible neural tree model. The model is computed as a flexible multi-layer feed-forward neural network. A hybrid learning/evolutionary approach to automatically optimize the neural tree model is also proposed. The approach includes a modified probabilistic incremental program evolution algorithm (MPIPE) to evolve and determine a optimal structure of the neural tree and a parameter learning algorithm to optimize the free parameters embedded in the neural tree. The performance and effectiveness of the proposed method are evaluated using function approximation, time series prediction and system identification problems and compared with the related methods.
TL;DR: Optimization of a fed-batch bioreactor using a cascade recurrent neural network (RNN) model and modified genetic algorithm and modified GA is studied, showing that the algorithm is able to generate a smooth feed rate profile, where the optimality is still maintained.
TL;DR: It is concluded that LSTM should be further investigated as a biologically plausible basis for a bottom-up, neural net-based approach to speech recognition.
Abstract: Long Short-Term Memory (LSTM) recurrent neural networks (RNNs) are local in space and time and closely related to a biological model of memory in the prefrontal cortex. Not only are they more biologically plausible than previous artificial RNNs, they also outperformed them on many artificially generated sequential processing tasks. This encouraged us to apply LSTM to more realistic problems, such as the recognition of spoken digits. Without any modification of the underlying algorithm, we achieved results comparable to state-of-the-art Hidden Markov Model (HMM) based recognisers on both the TIDIGITS and TI46 speech corpora. We conclude that LSTM should be further investigated as a biologically plausible basis for a bottom-up, neural net-based approach to speech recognition.
Book•
01 Jan 2004TL;DR: In this paper, the authors present a Neural Network approach to rainfall forecasting in urban environments.See and Kneale used the Cascade Correlation Neural Networks (CCNNs) for river flow forecasting.
Abstract: 1. Why Use Neural Networks? Pauline E.Kneale, Linda M. See & Robert J.Abrahart 2. Neural Network Modelling: Basic Tools and Broader Issues Robert J.Abrahart 3. Single Network Modelling Solutions Christian W.Dawson & Robert L.Wilby 4. Hybrid Neural Network Modelling Solutions Asaad Y.Shamseldin 5. The Application of Time Delay Neural Networks to River Level Forecasting Linda M.See & Pauline E.Kneale 6. The Application of Cascade Correlation Neural Networks to River Flow Forecasting Claire E.Imrie 7. The Use of Partial Recurrent Neural Networks for Autoregressive Modelling of Dynamic Hydrological Systems Henk F.P.van den Boogaard 8. RLF1/ Flood Forecasting via the Internet Simon A.Corne & Stan Openshaw 9. Rainfall-Runoff Modelling Anthony W.Minns & Michael J.Hall 10. A Neural Network Approach to Rainfall Forecasting in Urban Environments James E.Ball & Kin Choi Luk 11. Water Quality and Ecological Management in Freshwaters Pauline E.Kneale 12. Neural Network Modelling of Sediment Supply and Transfer Susan M.White 13. Nowcasting products from Meteorological Satellite Imagery George S.Pankiewicz 14. Mapping Land Cover from Remotely Sensed Imagery for Input to Hydrological Models Giles M.Foody 15. Towards a Hydrological Research Agenda Robert J.Abrahart, Pauline E.Kneale & Linda M.See Index