We study and compare two types of connectionist learning methods for model-free regression problems: 1) the backpropagation learning (BPL); and 2) the projection pursuit learning (PPL) emerged in recent years in the statistical estimation literature. Both the BPL and the PPL are based on projections of the data in directions determined from interconnection weights. However, unlike the use of fixed nonlinear activations (usually sigmoidal) for the hidden neurons in BPL, the PPL systematically approximates the unknown nonlinear activations. Moreover, the BPL estimates all the weights simultaneously at each iteration, while the PPL estimates the weights cyclically (neuron-by-neuron and layer-by-layer) at each iteration. Although the BPL and the PPL have comparable training speed when based on a Gauss-Newton optimization algorithm, the PPL proves more parsimonious in that the PPL requires a fewer hidden neurons to approximate the true function. To further improve the statistical performance of the PPL, an orthogonal polynomial approximation is used in place of the supersmoother method originally proposed for nonlinear activation approximation in the PPL.

Regression modeling in backpropagation and projection pursuit learning

We study and compare two types of connectionist learning methods for model-free regression problems: 1) the backpropagation learning (BPL); and 2) the projection pursuit learning (PPL) emerged in recent years in the statistical estimation literature. Both the BPL and the PPL are based on projections of the data in directions determined from interconnection weights. However, unlike the use of fixed nonlinear activations (usually sigmoidal) for the hidden neurons in BPL, the PPL systematically approximates the unknown nonlinear activations. Moreover, the BPL estimates all the weights simultaneously at each iteration, while the PPL estimates the weights cyclically (neuron-by-neuron and layer-by-layer) at each iteration. Although the BPL and the PPL have comparable training speed when based on a Gauss-Newton optimization algorithm, the PPL proves more parsimonious in that the PPL requires a fewer hidden neurons to approximate the true function. To further improve the statistical performance of the PPL, an orthogonal polynomial approximation is used in place of the supersmoother method originally proposed for nonlinear activation approximation in the PPL. >

Regression modeling in back-propagation and projection pursuit learning

Efficient algorithms that incorporate different types of spectral data and ancillary data are being developed to extract continuous vegetation variables. Inferring continuous variables implies that functional relationships must be found among the predicted variable(s), the remotely sensed data and the ancillary data. Neural networks have attributes which facilitate the extraction of vegetation variables. The advantages and power of neural networks for extracting continuous vegetation variables using optical and/or radar data and ancillary data are discussed and compared to traditional techniques. Studies that have made advances in this research area are reviewed and discussed. Neural networks can provide accurate initial models for extracting vegetation variables when an adequate amount of data is available. Networks provide a performance standard for evaluating existing physically based models. Many practical problems occur when inverting physically based models using traditional techniques and neural ne...

Review article: Attributes of neural networks for extracting continuous vegetation variables from optical and radar measurements

Modular Neural Networks (MNNs) is a rapidly growing field in artificial Neural Networks (NNs) research. This paper surveys the different motivations for creating MNNs: biological, psychological, hardware, and computational. Then, the general stages of MNN design are outlined and surveyed as well, viz., task decomposition techniques, learning schemes and multi-module decision-making strategies. Advantages and disadvantages of the surveyed methods are pointed out, and an assessment with respect to practical potential is provided. Finally, some general recommendations for future designs are presented.

Modular neural networks: a survey.

PLS/neural networks

A modular architecture for supervised learning is presented. Three ways to modularize the learning are investigated: (1) preprocessing the input by a self-organizing subnetwork to extract strong features from the data, (2) supervised feature extraction, and (3) an iterative learning cycle in which only one layer learns at a time, with the output-layer weights learned by an exact method. With this modular architecture, only a small fraction of connection weights is determined by the gradient-descent method. A series of computational experiments shows the superiority of the modular model in learning quality and speed. The author begins by considering the decomposition of a feedforward network into a feature-discovery and a supervised-learning modules, then he introduces supervised feature discovery, and finally he describes a modular backpropagation algorithm

T. Hrycej

Papers

A modular architecture for efficient learning