Learning and evolution are two fundamental forms of adaptation. There has been a great interest in combining learning and evolution with artificial neural networks (ANNs) in recent years. This paper: 1) reviews different combinations between ANNs and evolutionary algorithms (EAs), including using EAs to evolve ANN connection weights, architectures, learning rules, and input features; 2) discusses different search operators which have been used in various EAs; and 3) points out possible future research directions. It is shown, through a considerably large literature review, that combinations between ANNs and EAs can lead to significantly better intelligent systems than relying on ANNs or EAs alone.

/pdf/evolving-artificial-neural-networks-4kzrbu7pkn.pdf

Evolving artificial neural networks

This paper presents a new evolutionary system, i.e., EPNet, for evolving artificial neural networks (ANNs). The evolutionary algorithm used in EPNet is based on Fogel's evolutionary programming (EP). Unlike most previous studies on evolving ANN's, this paper puts its emphasis on evolving ANN's behaviors. Five mutation operators proposed in EPNet reflect such an emphasis on evolving behaviors. Close behavioral links between parents and their offspring are maintained by various mutations, such as partial training and node splitting. EPNet evolves ANN's architectures and connection weights (including biases) simultaneously in order to reduce the noise in fitness evaluation. The parsimony of evolved ANN's is encouraged by preferring node/connection deletion to addition. EPNet has been tested on a number of benchmark problems in machine learning and ANNs, such as the parity problem, the medical diagnosis problems, the Australian credit card assessment problem, and the Mackey-Glass time series prediction problem. The experimental results show that EPNet can produce very compact ANNs with good generalization ability in comparison with other algorithms.

/pdf/a-new-evolutionary-system-for-evolving-artificial-neural-56dnqjphy7.pdf

A new evolutionary system for evolving artificial neural networks

One way of simplifying neural networks so they generalize better is to add an extra term to the error function that will penalize complexity. Simple versions of this approach include penalizing the sum of the squares of the weights or penalizing the number of nonzero weights. We propose a more complicated penalty term in which the distribution of weight values is modeled as a mixture of multiple gaussians. A set of weights is simple if the weights have high probability density under the mixture model. This can be achieved by clustering the weights into subsets with the weights in each cluster having very similar values. Since we do not know the appropriate means or variances of the clusters in advance, we allow the parameters of the mixture model to adapt at the same time as the network learns. Simulations on two different problems demonstrate that this complexity term is more effective than previous complexity terms.

Simplifying neural networks by soft weight-sharing

Within this text neural networks are considered as massively interconnected nonlinear adaptive filters.

Recurrent Neural Networks for Prediction

In a radial basis function (RBF) network, the RBF centers and widths can be evolved by a cooperative-competitive genetic algorithm The set of genetic strings in one generation of the algorithm represents one REP network, not a population of competing networks This leads to moderate computation times for the algorithm as a whole Selection operates on individual RBFs rather than on whole networks Selection therefore requires a genetic fitness function that promotes competition among RBFs which are doing nearly the same job while at the same time promoting cooperation among RBFs which cover different parts of the domain of the function to be approximated Niche creation resulting from a fitness function of the form |w/sub i/|/sup /spl beta///E(|w/sub i'/|/sup /spl beta//), 1</spl beta/<2 can facilitate the desired cooperative-competitive behavior The feasibility of the resulting algorithm to evolve networks of Gaussian, inverse multiquadric, and thin-plate spline RBFs is demonstrated by predicting the Mackey-Glass time series For each type of RBF, and for networks of 25, 50, 75, 100, 125, and 150 RBF units, prediction errors for the evolved Gaussian RBF networks are 50-70% lower than RBF networks obtained by k-means clustering

Cooperative-competitive genetic evolution of radial basis function centers and widths for time series prediction

Evolutionary programming, a systematic multi-agent stochastic search technique, is used to generate recurrent perceptrons (nonlinear IIR filters). A hybrid optimization scheme is proposed that embeds a single-agent stochastic search technique, the method of Solis and Wets, into the evolutionary programming paradigm. The proposed hybrid optimization approach is further augmented by "blending" randomly selected parent vectors to create additional offspring. The first part of this work investigates the performance of the suggested hybrid stochastic search method. After demonstration on the Bohachevsky and Rosenbrock response surfaces, the hybrid stochastic optimization approach is applied in determining both the model order and the coefficients of recurrent perceptron time-series models. An information criterion is used to evaluate each recurrent perceptron structure as a candidate solution. It is speculated that the stochastic training method implemented in this study for training recurrent perceptrons can be used to train perceptron networks that have radically recurrent architectures. >

D. Waagen

Papers

Evolving recurrent perceptrons for time-series modeling