In this letter, a new approach for the learning process of multilayer feedforward neural network is introduced. This approach minimizes a modified form of the criterion used in the standard backpropagation algorithm. This criterion is based on the sum of the linear and the nonlinear quadratic errors of the output neuron. The quadratic linear error signal is appropriately weighted. The choice of the weighted design parameter is evaluated via rank convergence series analysis and asymptotic constant error values. The new proposed modified standard backpropagation algorithm (MBP) is first derived on a single neuron-based net and then extended to a general feedforward neural network. Simulation results of the 4-b parity checker and the circle in the square problem confirm that the performance of the MBP algorithm exceed the standard backpropagation (SBP) in the reduction of the total number of iterations and in the learning time.

A fast feedforward training algorithm using a modified form of the standard backpropagation algorithm

A new approach for the learning process of multilayer perceptron neural networks using the recursive least squares (RLS) type algorithm is proposed. This method minimizes the global sum of the square of the errors between the actual and the desired output values iteratively. The weights in the network are updated upon the arrival of a new training sample and by solving a system of normal equations recursively. To determine the desired target in the hidden layers an analog of the back-propagation strategy used in the conventional learning algorithms is developed. This permits the application of the learning procedure to all the layers. Simulation results on the 4-b parity checker and multiplexer networks were obtained which indicate significant reduction in the total number of iterations when compared with those of the conventional and accelerated back-propagation algorithms. >

Fast learning process of multilayer neural networks using recursive least squares method

A simple method for improving the learning rate of the backpropagation algorithm is described and analyzed. The method is referred to as the gradient reuse algorithm (GRA). The basic idea is that ingredients which are computed using backpropagation are reused several times until the resulting weight updates no longer lead to a reduction in error. It is shown that convergence speedup is a function of the reuse rate, and that the reuse rate can be controlled by using a dynamic convergence parameter.<<ETX>>

Improving the learning rate of back-propagation with the gradient reuse algorithm.

It is noted that the training of feedforward networks using the conventional backpropagation algorithm is plagued by poor convergence and misadjustment. The authors introduce the multiple extended Kalman algorithm (MEKA) to train feedforward networks. It is based on the idea of partitioning the global problem of finding the weights into a set of manageable nonlinear subproblems. The algorithm is local at the neuron level. The superiority of MEKA over the global extended Kalman algorithm in terms of convergence and quality of solution obtained on two benchmark problems is demonstrated. The superior performance can be attributed to the nonlinear localized approach. In fact, the nonconvex nature of the local performance surface reduces the chances of getting trapped into a local minima

MEKA-a fast, local algorithm for training feedforward neural networks

In this paper, a new learning algorithm, called the Modified Recursive Least Square (MRLS), is introduced for the Hybrid Multilayered Perceptron (HMLP) network. Adopting the Recursive Least Square (RLS) algorithm as its basis, the MRLS algorithm differs from RLS in the way that the weight of the linear connections for the HMLP network is estimated. The convergence rate of the MRLS algorithm is further improved by varying the forgetting factor, optimizing the way the momentum and learning rate are assigned. To investigate its applicability, the MRLS algorithm is demonstrated on the HMLP network using six benchmark data sets obtained from the UCI repository. The classification performance of the HMLP network trained with the MRLS algorithm is compared with those of the HMLP network trained with the Modified Recursive Prediction Error (MRPE) algorithm and the MLP trained with the standard RLS algorithm as well as with other commonly adopted machine learning classifiers. The comparison results indicated that the proposed MRLS trained HMLP network provides significant improvement over RLS trained MLP network, MRPE trained HMLP network, and other machine learning classifiers in terms of accuracy, convergence rate and mean square error (MSE).

Modified Recursive Least Squares algorithm to train the Hybrid Multilayered Perceptron (HMLP) network

A new approach for the learning process of multilayer perceptron neural networks using a recursive-least-squares-(RLS) type algorithm is proposed. The weights in the network are updated recursively upon the arrival of a new training sample. To determine the desired target in the hidden layers an analog of the back-propagation strategy used in the conventional learning algorithms is developed. This permits the application of the learning procedure to all the other lower layers. Simulation results on the 4-b parity checker problem are provided. >

Supervised learning process of multi-layer perceptron neural networks using fast least squares

A 2-D diagonal block recursive representation for 2-D autoregressive (AR) image models with nonsymmetric half-plane (NSHP) regions of support that does not have noncausality problems is introduced. The relevant 2-D block Kalman filter equations are used to obtain suboptimal block filtered estimates for the blurred and noisy image. A recursive parameter identification scheme can be used online to update the model parameters at each processing window suggested. Simulation results are presented. >

A 2-D adaptive diagonal block Kalman filter for nonsymmetric half plane image models

A method of two-dimensional block Kalman filtering using a full plane model is presented. The filter uses a full plane model, yet it maintains causality within the actual filter by using multiple estimators. Using this method the sizes of the state and error covariance matrices are reduced to a computationally feasible size. Simulation results are provided for the case of filtering an image corrupted by 5-dB additive Gaussian noise. >

S. Citrin

Papers

Supervised learning process of multi-layer perceptron neural networks using fast least squares

A 2-D adaptive diagonal block Kalman filter for nonsymmetric half plane image models

A full plane block Kalman filter for image estimation