This paper investigates new learning algorithms (LF I and LF II) based on Lyapunov function for the training of feedforward neural networks. It is observed that such algorithms have interesting parallel with the popular backpropagation (BP) algorithm where the fixed learning rate is replaced by an adaptive learning rate computed using convergence theorem based on Lyapunov stability theory. LF II, a modified version of LF I, has been introduced with an aim to avoid local minima. This modification also helps in improving the convergence speed in some cases. Conditions for achieving global minimum for these kind of algorithms have been studied in detail. The performances of the proposed algorithms are compared with BP algorithm and extended Kalman filtering (EKF) on three bench-mark function approximation problems: XOR, 3-bit parity, and 8-3 encoder. The comparisons are made in terms of number of learning iterations and computational time required for convergence. It is found that the proposed algorithms (LF I and II) are much faster in convergence than other two algorithms to attain same accuracy. Finally, the comparison is made on a complex two-dimensional (2-D) Gabor function and effect of adaptive learning rate for faster convergence is verified. In a nutshell, the investigations made in this paper help us better understand the learning procedure of feedforward neural networks in terms of adaptive learning rate, convergence speed, and local minima

On Adaptive Learning Rate That Guarantees Convergence in Feedforward Networks

We present a novel event-based stereo matching algorithm that exploits the asynchronous visual events from a pair of silicon retinas. Unlike conventional frame-based cameras, recent artificial retinas transmit their outputs as a continuous stream of asynchronous temporal events, in a manner similar to the output cells of the biological retina. Our algorithm uses the timing information carried by this representation in addressing the stereo-matching problem on moving objects. Using the high temporal resolution of the acquired data stream for the dynamic vision sensor, we show that matching on the timing of the visual events provides a new solution to the real-time computation of 3-D objects when combined with geometric constraints using the distance to the epipolar lines. The proposed algorithm is able to filter out incorrect matches and to accurately reconstruct the depth of moving objects despite the low spatial resolution of the sensor. This brief sets up the principles for further event-based vision processing and demonstrates the importance of dynamic information and spike timing in processing asynchronous streams of visual events.

Asynchronous Event-Based Binocular Stereo Matching

A novel model-free adaptive control method is proposed for a class of discrete-time single input single output (SISO) nonlinear systems, where the equivalent dynamic linearization technique is used on the ideal nonlinear controller. With radial basis function neural network, the controller parameters are tuned on-line directly using the measured input and output data of the plant, when the plant model is unavailable. The stability of the proposed method is guaranteed by rigorous theoretical analysis, and the effectiveness and applicability are verified by numerical simulation and further demonstrated by the experiment on three tanks water level control process.

Data-Driven MFAC for a Class of Discrete-Time Nonlinear Systems With RBFNN

A new proposed method, i.e. the recurrent neural network (RNN), is introduced to predict chaotic time series. The effectiveness of using RNN for making one-step and multi-step predictions is tested based on remarkable few datum points by computer-generated chaotic time series. Numerical results show that the RNN proposed here is a very powerful tool for making prediction of chaotic time series.

https://iopscience.iop.org/article/10.1088/0256-307X/17/2/004/pdf

Predicting Chaotic Time Series Using Recurrent Neural Network

A novel learning algorithm is proposed for nonlinear modelling and identification using radial basis function neural networks. The proposed method simplifies neural network training through the use of an adaptive computation algorithm (ACA). In addition, the convergence of the ACA is analyzed by the Lyapunov criterion. The proposed algorithm offers two important advantages. First, the model performance can be significantly improved through ACA, and the modelling error is uniformly ultimately bounded. Secondly, the proposed ACA can reduce computational cost and accelerate the training speed. The proposed method is then employed to model classical nonlinear system with limit cycle and to identify nonlinear dynamic system, exhibiting the effectiveness of the proposed algorithm. Computational complexity analysis and simulation results demonstrate its effectiveness.

Adaptive Computation Algorithm for RBF Neural Network

The recursive least squares method (RLS) is derived for the learning of multilayer feedforward neural networks. Simulation results on the XOR, 4-2-4 encoder, and function approximation problems indicate a fast learning process in comparison to the classical and momentum backpropagation (BP) algorithms.

A fast training algorithm for neural networks

Abstract This paper presents a local modification of the Levenberg-Marquardt algorithm (LM). First, the mathematical basics of the classic LM method are shown. The classic LM algorithm is very efficient for learning small neural networks. For bigger neural networks, whose computational complexity grows significantly, it makes this method practically inefficient. In order to overcome this limitation, local modification of the LM is introduced in this paper. The main goal of this paper is to develop a more complexity efficient modification of the LM method by using a local computation. The introduced modification has been tested on the following benchmarks: the function approximation and classification problems. The obtained results have been compared to the classic LM method performance. The paper shows that the local modification of the LM method significantly improves the algorithm’s performance for bigger networks. Several possible proposals for future works are suggested.

/pdf/local-levenberg-marquardt-algorithm-for-learning-feedforwad-4cx4r69d8y.pdf

Local Levenberg-Marquardt Algorithm for Learning Feedforwad Neural Networks

A major problem encountered by researchers of dynamic neural networks is the computational complexity increasing the learning time. In this paper the parallel realization of the RTRN and the Elman networks are discussed. Both networks are examples of dynamic neural networks. Inherent parallelism of dynamic neural networks has been employed to accelerate the learning process. The proposed solution is based on a highly parallel three dimensional architecture to speed up the learning performance. The presented structures are suitable for efficient parallel realization in digital hardware or vector processors.

Parallel Architectures for Learning the RTRN and Elman Dynamic Neural Networks

This paper presents the parallel architecture of the Jordan network learning algorithm. The proposed solution is based on the high parallel three dimensional structures to speed up learning performance. Detailed parallel neural network structures are explicitly shown.

Parallel Approach to Learning of the Recurrent Jordan Neural Network

This paper presents the parallel architecture of the conjugate gradient learning algorithm for the feedforward neural networks. The proposed solution is based on the high parallel structures to speed up learning performance. Detailed parallel neural network structures are explicitly shown.

Jarosław Bilski

Papers

A fast training algorithm for neural networks

Local Levenberg-Marquardt Algorithm for Learning Feedforwad Neural Networks

Parallel Architectures for Learning the RTRN and Elman Dynamic Neural Networks

Parallel Approach to Learning of the Recurrent Jordan Neural Network

The Parallel Approach to the Conjugate Gradient Learning Algorithm for the Feedforward Neural Networks