A wavelet network concept, which is based on wavelet transform theory, is proposed as an alternative to feedforward neural networks for approximating arbitrary nonlinear functions. The basic idea is to replace the neurons by 'wavelons', i.e., computing units obtained by cascading an affine transform and a multidimensional wavelet. Then these affine transforms and the synaptic weights must be identified from possibly noise corrupted input/output data. An algorithm of backpropagation type is proposed for wavelet network training, and experimental results are reported. >

Wavelet networks

Statistical aspects of neural networks

One of the most important aspects of any machine learning paradigm is how i t scales according to problem size and complexity. Using a task with known optimal train ing error, and a pre-specified maximum number of training updates, we investigate the convergence of th backpropagation algorithm with respect to a) the complexity of the required function approximatio n, b) the size of the network in relation to the size required for an optimal solution, and c) the degree o f noise in the training data. In general, for a) the solution found is worse when the function to be app roximated is more complex, for b) oversized networks can result in lower training and generalization error in certain cases, and for c) the use of committee or ensemble techniques can be more beneficial as the level o f noise in the training data is increased. For the experiments we performed, we do not obtain the optimal solution in any case. We further support the observation that larger networks can produce bett er training and generalization error using a face recognition example where a network with many more par ameters than training points generalizes better than smaller networks.

What Size Neural Network Gives Optimal Generalization? Convergence Properties of Backpropagation

The theory of artificial neural networks is briefly reviewed focusing on supervised and unsupervised techniques which have great impact on current chemical applications. An introduction to molecular descriptors and representation schemes is given. In addition, worked examples of recent advances in this field are highlighted and pioneering publications are discussed. Applications of several types of artificial neural networks to compound classification, modelling of structure–activity relationships, biological target identification, and feature extraction from biopolymers are presented and compared to other techniques. Advantages and limitations of neural networks for computer-aided molecular design and sequence analysis are discussed.

Artificial neural networks for computer-based molecular design.

A modified back-propagation method to avoid false local minima

Summary form only given, as follows. The possible existence of local minima in the error surfaces of backpropagation neural networks has been an important unanswered question. Evidence has demonstrated that error surface regions of small slope with a high mean square error are frequently encountered during training. Such regions are often mistakenly believed to be local minima since no significant decrease in error occurs over considerable training time. In many cases, if training is continued, the shallow region is traversed. Given these experiences, it became plausible to suggest that backpropagation error surfaces have no local minima. A discussion is presented of the results of the exploration of the error surface for two networks, and the discovery of a true local minimum is documented. >

Hecht-Nielsen

Papers

Back propagation error surfaces can have local minima