https://www2.econ.iastate.edu/tesfatsi/DeepLearningInNeuralNetworksOverview.JSchmidhuber2015.pdf

Deep learning in neural networks

The Marquardt algorithm for nonlinear least squares is presented and is incorporated into the backpropagation algorithm for training feedforward neural networks. The algorithm is tested on several function approximation problems, and is compared with a conjugate gradient algorithm and a variable learning rate algorithm. It is found that the Marquardt algorithm is much more efficient than either of the other techniques when the network contains no more than a few hundred weights. >

/pdf/training-feedforward-networks-with-the-marquardt-algorithm-e1r0710f99.pdf

Training feedforward networks with the Marquardt algorithm

https://sci2s.ugr.es/keel/pdf/algorithm/articulo/moller1990.pdf

Original Contribution: A scaled conjugate gradient algorithm for fast supervised learning

Many scientific, engineering and economic problems involve the optimisation of a set of parameters. These problems include examples like minimising the losses in a power grid by finding the optimal configuration of the components, or training a neural network to recognise images of people's faces. Numerous optimisation algorithms have been proposed to solve these problems, with varying degrees of success. The Particle Swarm Optimiser (PSO) is a relatively new technique that has been empirically shown to perform well on many of these optimisation problems. This thesis presents a theoretical model that can be used to describe the long-term behaviour of the algorithm. An enhanced version of the Particle Swarm Optimiser is constructed and shown to have guaranteed convergence on local minima. This algorithm is extended further, resulting in an algorithm with guaranteed convergence on global minima. A model for constructing cooperative PSO algorithms is developed, resulting in the introduction of two new PSO-based algorithms. Empirical results are presented to support the theoretical properties predicted by the various models, using synthetic benchmark functions to investigate specific properties. The various PSO-based algorithms are then applied to the task of training neural networks, corroborating the results obtained on the synthetic benchmark functions.

/pdf/an-analysis-of-particle-swarm-optimizers-3iz2tj39rw.pdf

An analysis of particle swarm optimizers

An artificial neural network (ANN) is a flexible mathematical structure which is capable of identifying complex nonlinear relationships between input and output data sets. ANN models have been found useful and efficient, particularly in problems for which the characteristics of the processes are difficult to describe using physical equations. This study presents a new procedure (entitled linear least squares simplex, or LLSSIM) for identifying the structure and parameters of three-layer feed forward ANN models and demonstrates the potential of such models for simulating the nonlinear hydrologic behavior of watersheds. The nonlinear ANN model approach is shown to provide a better representation of the rainfall-runoff relationship of the medium-size Leaf River basin near Collins, Mississippi, than the linear ARMAX (autoregressive moving average with exogenous inputs) time series approach or the conceptual SAC-SMA (Sacramento soil moisture accounting) model. Because the ANN approach presented here does not provide models that have physically realistic components and parameters, it is by no means a substitute for conceptual watershed modeling. However, the ANN approach does provide a viable and effective alternative to the ARMAX time series approach for developing input-output simulation and forecasting models in situations that do not require modeling of the internal structure of the watershed.

Artificial Neural Network Modeling of the Rainfall‐Runoff Process

The utility of the back-propagation method in establishing suitable weights in a distributed adaptive network has been demonstrated repeatedly. Unfortunately, in many applications, the number of iterations required before convergence can be large. Modifications to the back-propagation algorithm described by Rumelhart et al. (1986) can greatly accelerate convergence. The modifications consist of three changes:1) instead of updating the network weights after each pattern is presented to the network, the network is updated only after the entire repertoire of patterns to be learned has been presented to the network, at which time the algebraic sums of all the weight changes are applied:2) instead of keeping ź, the "learning rate" (i.e., the multiplier on the step size) constant, it is varied dynamically so that the algorithm utilizes a near-optimum ź, as determined by the local optimization topography; and3) the momentum factor ź is set to zero when, as signified by a failure of a step to reduce the total error, the information inherent in prior steps is more likely to be misleading than beneficial. Only after the network takes a useful step, i.e., one that reduces the total error, does ź again assume a non-zero value. Considering the selection of weights in neural nets as a problem in classical nonlinear optimization theory, the rationale for algorithms seeking only those weights that produce the globally minimum error is reviewed and rejected.

Accelerating the convergence of the back-propagation method

An experiment is described which compares the performance of a neural network to human performance on a visual task which consists of detecting a target in a background image of correlated noise. A three-layer, feed-forward, multi-layer perceptron is trained to indicate the presence or absence of a target in images also presented to human observers. The basis for the comparison between the network and the human observers is the receiver operating characteristic (ROC) curve. Network performance is comparable to human performance for this particular task.

W. T. Zink

Papers

Accelerating the convergence of the back-propagation method

Performance comparison of a neural network with human observers on a visual target detection task