scispace - formally typeset
Search or ask a question
Journal ArticleDOI

An optimizing BP neural network algorithm based on genetic algorithm

01 Aug 2011-Artificial Intelligence Review (Springer Netherlands)-Vol. 36, Iss: 2, pp 153-162
TL;DR: A method that combines GA and BP to train the neural network works better; is less easily stuck in a local minimum; the trained network has a better generalization ability; and it has a good stabilization performance.
Abstract: A back-propagation (BP) neural network has good self-learning, self-adapting and generalization ability, but it may easily get stuck in a local minimum, and has a poor rate of convergence. Therefore, a method to optimize a BP algorithm based on a genetic algorithm (GA) is proposed to speed the training of BP, and to overcome BP's disadvantage of being easily stuck in a local minimum. The UCI data set is used here for experimental analysis and the experimental result shows that, compared with the BP algorithm and a method that only uses GA to learn the connection weights, our method that combines GA and BP to train the neural network works better; is less easily stuck in a local minimum; the trained network has a better generalization ability; and it has a good stabilization performance.
Citations
More filters
Journal ArticleDOI
01 Jan 2018
TL;DR: The qualitative and quantitative results prove that the proposed WOA-based trainer is able to outperform the current algorithms on the majority of datasets in terms of both local optima avoidance and convergence speed.
Abstract: The learning process of artificial neural networks is considered as one of the most difficult challenges in machine learning and has attracted many researchers recently. The main difficulty of training a neural network is the nonlinear nature and the unknown best set of main controlling parameters (weights and biases). The main disadvantages of the conventional training algorithms are local optima stagnation and slow convergence speed. This makes stochastic optimization algorithm reliable alternative to alleviate these drawbacks. This work proposes a new training algorithm based on the recently proposed whale optimization algorithm (WOA). It has been proved that this algorithm is able to solve a wide range of optimization problems and outperform the current algorithms. This motivated our attempts to benchmark its performance in training feedforward neural networks. For the first time in the literature, a set of 20 datasets with different levels of difficulty are chosen to test the proposed WOA-based trainer. The results are verified by comparisons with back-propagation algorithm and six evolutionary techniques. The qualitative and quantitative results prove that the proposed trainer is able to outperform the current algorithms on the majority of datasets in terms of both local optima avoidance and convergence speed.

556 citations

Journal ArticleDOI
TL;DR: This paper describes the latest progress of ELM in recent years, including the model and specific applications of ELm, and finally points out the research and development prospects ofELM in the future.
Abstract: Extreme learning machine (ELM) is a new learning algorithm for the single hidden layer feedforward neural networks Compared with the conventional neural network learning algorithm it overcomes the slow training speed and over-fitting problems ELM is based on empirical risk minimization theory and its learning process needs only a single iteration The algorithm avoids multiple iterations and local minimization It has been used in various fields and applications because of better generalization ability, robustness, and controllability and fast learning rate In this paper, we make a review of ELM latest research progress about the algorithms, theory and applications It first analyzes the theory and the algorithm ideas of ELM, then tracking describes the latest progress of ELM in recent years, including the model and specific applications of ELM, finally points out the research and development prospects of ELM in the future

429 citations


Cites background or methods from "An optimizing BP neural network alg..."

  • ...…solid theoretical basis and simple network structuremodelmake the neural network in the fields of pattern recognition, image processing, sensors, signal processing and automatic control have significant results (Ding et al. 2011a,b, 2012;Quteishat andLim2008;Zhang and Wang 2009; Ding and Jin 2013)....

    [...]

  • ...Coupled with its solid theoretical basis and simple network structuremodelmake the neural network in the fields of pattern recognition, image processing, sensors, signal processing and automatic control have significant results (Ding et al. 2011a,b, 2012;Quteishat andLim2008;Zhang and Wang 2009; Ding and Jin 2013). It is widely used in the fields of expert system (MarkowskaKaczmar and Trelak 2005), pattern recognition (Mohamed 2011), intelligent control (Ding et al. 2011a,b), combinatorial optimization (Kahramanli and Allahverdi 2009) and prediction (Hagan et al. 2002). At present, there are many common kinds of neural network model, such as BP network (Ding et al. 2011a,b; Feng et al. 2009), RBF network (Ding et al. 2011a,b), Hopfield network, CMAC cerebellar model, ART adaptive resonance theory (LeCun and Bengio 1995; Benqio 2009; Carpenter and Grossberg 2003; Li et al. 2013), etc. The powerful computing capabilities of the neural network are achieved through the propagation of information between neurons. According to the direction of the neural network internal information transfer, the neural network can be divided into two categories: feedforward neural network and feedback type neural networks. Extreme learning machine (ELM) described in this paper is for single hidden layer feedforward neural network which is one of feedforward neural networks. Feedforward neural network model has been extensively used in many fields due to its ability to approximate complex nonlinear mappings directly from the input samples. Among them, for a single hidden layer feedforward neural network learning ability the majority of studies focused on the input samples, divided into two aspects of the compact set and finite set. Hornik (1991) proved that if the activation function is continuous, bounded and nonconstant, then continuous mappings can be approximated in measure by neural networks over compact input sets....

    [...]

  • ...At present, there are many common kinds of neural network model, such as BP network (Ding et al. 2011a,b; Feng et al. 2009), RBF network (Ding et al. 2011a,b), Hopfield network, CMAC cerebellar model, ART adaptive resonance theory (LeCun and Bengio 1995; Benqio 2009; Carpenter and Grossberg 2003;…...

    [...]

  • ...It is widely used in the fields of expert system (MarkowskaKaczmar and Trelak 2005), pattern recognition (Mohamed 2011), intelligent control (Ding et al. 2011a,b), combinatorial optimization (Kahramanli and Allahverdi 2009) and prediction (Hagan et al. 2002)....

    [...]

  • ...Coupled with its solid theoretical basis and simple network structuremodelmake the neural network in the fields of pattern recognition, image processing, sensors, signal processing and automatic control have significant results (Ding et al. 2011a,b, 2012;Quteishat andLim2008;Zhang and Wang 2009; Ding and Jin 2013). It is widely used in the fields of expert system (MarkowskaKaczmar and Trelak 2005), pattern recognition (Mohamed 2011), intelligent control (Ding et al. 2011a,b), combinatorial optimization (Kahramanli and Allahverdi 2009) and prediction (Hagan et al. 2002). At present, there are many common kinds of neural network model, such as BP network (Ding et al. 2011a,b; Feng et al. 2009), RBF network (Ding et al. 2011a,b), Hopfield network, CMAC cerebellar model, ART adaptive resonance theory (LeCun and Bengio 1995; Benqio 2009; Carpenter and Grossberg 2003; Li et al. 2013), etc. The powerful computing capabilities of the neural network are achieved through the propagation of information between neurons. According to the direction of the neural network internal information transfer, the neural network can be divided into two categories: feedforward neural network and feedback type neural networks. Extreme learning machine (ELM) described in this paper is for single hidden layer feedforward neural network which is one of feedforward neural networks. Feedforward neural network model has been extensively used in many fields due to its ability to approximate complex nonlinear mappings directly from the input samples. Among them, for a single hidden layer feedforward neural network learning ability the majority of studies focused on the input samples, divided into two aspects of the compact set and finite set. Hornik (1991) proved that if the activation function is continuous, bounded and nonconstant, then continuous mappings can be approximated in measure by neural networks over compact input sets. On this basis Leshno et al. (1991) improved the results and proved that feedforward networks with a non-polynomial activation function can approximate continuous functions....

    [...]

Journal ArticleDOI
TL;DR: A broad spectrum of FNN optimization methodologies including conventional and metaheuristic approaches are summarized, which provides interesting research challenges for future research to cope-up with the present information processing era.

398 citations

Journal ArticleDOI
TL;DR: In this paper, the authors present a taxonomy of CNN acceleration methods in terms of three levels, i.e. structure level, algorithm level, and implementation level, for CNN architectures compression, algorithm optimization and hardware-based improvement.

233 citations

Journal ArticleDOI
TL;DR: The principle and algorithm of extreme learning machine (ELM), a novel learning algorithm for single-hidden-layer feedforward neural networks (SLFNs), are described, which provides extremely faster learning speed, better generalization performance and with least human intervention.
Abstract: Recently, a novel learning algorithm for single-hidden-layer feedforward neural networks (SLFNs) named extreme learning machine (ELM) was proposed by Huang et al. The essence of ELM is that the learning parameters of hidden nodes, including input weights and biases, are randomly assigned and need not be tuned while the output weights can be analytically determined by the simple generalized inverse operation. The only parameter needed to be defined is the number of hidden nodes. Compared with other traditional learning algorithms for SLFNs, ELM provides extremely faster learning speed, better generalization performance and with least human intervention. This paper firstly introduces a brief review of ELM, describing the principle and algorithm of ELM. Then, we put emphasis on the improved methods or the typical variants of ELM, especially on incremental ELM, pruning ELM, error-minimized ELM, two-stage ELM, online sequential ELM, evolutionary ELM, voting-based ELM, ordinal ELM, fully complex ELM, and symmetric ELM. Next, the paper summarized the applications of ELM on classification, regression, function approximation, pattern recognition, forecasting and diagnosis, and so on. In the last, the paper discussed several open issues of ELM, which may be worthy of exploring in the future.

187 citations

References
More filters
Journal ArticleDOI
Xin Yao1
01 Sep 1999
TL;DR: It is shown, through a considerably large literature review, that combinations between ANNs and EAs can lead to significantly better intelligent systems than relying on ANNs or EAs alone.
Abstract: Learning and evolution are two fundamental forms of adaptation. There has been a great interest in combining learning and evolution with artificial neural networks (ANNs) in recent years. This paper: 1) reviews different combinations between ANNs and evolutionary algorithms (EAs), including using EAs to evolve ANN connection weights, architectures, learning rules, and input features; 2) discusses different search operators which have been used in various EAs; and 3) points out possible future research directions. It is shown, through a considerably large literature review, that combinations between ANNs and EAs can lead to significantly better intelligent systems than relying on ANNs or EAs alone.

2,877 citations


"An optimizing BP neural network alg..." refers methods in this paper

  • ...GA is an iterative computation process, and its main steps include: encoding, initialization of the population, selection, genetic operation (crossover, mutation), evaluation and stop decision (Yao and Xu 2006)....

    [...]

Journal ArticleDOI
TL;DR: It is shown that the use of a genetic algorithm can provide better results for training a feedforward neural network than the traditional techniques of backpropagation.
Abstract: This article shows that the use of a genetic algorithm can provide better results for training a feedforward neural network than the traditional techniques of backpropagation. Using a chaotic time series as an illustration, we directly compare the genetic algorithm and backpropagation for effectiveness, ease-of-use, and efficiency for training neural networks.

224 citations

Journal ArticleDOI
TL;DR: This paper will review some of the recent work in evolutionary approaches to designing ANN ensembles and reveal that there is a deep underlying connection between evolutionary computation and ANNEnsembles.
Abstract: Using a coordinated group of simple solvers to tackle a complex problem is not an entirely new idea. Its root could be traced back hundreds of years ago when ancient Chinese suggested a team approach to problem solving. For a long time, engineers have used the divide-and-conquer strategy to decompose a complex problem into simpler sub-problems and then solve them by a group of solvers. However, knowing the best way to divide a complex problem into simpler ones relies heavily on the available domain knowledge. It is often a manual process by an experienced engineer. There have been few automatic divide-and-conquer methods reported in the literature. Fortunately, evolutionary computation provides some of the interesting avenues to automatic divide-and-conquer methods. An in-depth study of such methods reveals that there is a deep underlying connection between evolutionary computation and ANN ensembles. Ideas in one area can be usefully transferred into another in producing effective algorithms. For example, using speciation to create and maintain diversity had inspired the development of negative correlation learning for ANN ensembles, and an in-depth study of diversity in ensembles. This paper will review some of the recent work in evolutionary approaches to designing ANN ensembles.

126 citations


"An optimizing BP neural network alg..." refers methods in this paper

  • ...…such as constructing a neural network based on particle swarm optimization algorithm (Chen and Yu 2005), and using evolutionary algorithms to optimize the neural networks (Eysa and Saeed 2005; Harpham 2004; Venkatesan 2009; Yao and Islam 2008), which have been proved feasible and effective....

    [...]

Journal ArticleDOI
TL;DR: A brief overview of feedforward ANNs and GAs is given followed by a review of the current state of research in applying evolutionary techniques to training RBF networks.
Abstract: The problems associated with training feedforward artificial neural networks (ANNs) such as the multilayer perceptron (MLP) network and radial basis function (RBF) network have been well documented. The solutions to these problems have inspired a considerable amount of research, one particular area being the application of evolutionary search algorithms such as the genetic algorithm (GA). To date, the vast majority of GA solutions have been aimed at the MLP network. This paper begins with a brief overview of feedforward ANNs and GAs followed by a review of the current state of research in applying evolutionary techniques to training RBF networks.

118 citations


"An optimizing BP neural network alg..." refers methods in this paper

  • ...…such as constructing a neural network based on particle swarm optimization algorithm (Chen and Yu 2005), and using evolutionary algorithms to optimize the neural networks (Eysa and Saeed 2005; Harpham 2004; Venkatesan 2009; Yao and Islam 2008), which have been proved feasible and effective....

    [...]

Journal ArticleDOI
TL;DR: Using neural networks within the framework of VSP creates a robust tool for optimum design of structures and reduces the computational cost of standard GA.

117 citations


"An optimizing BP neural network alg..." refers methods in this paper

  • ...…such as constructing a neural network based on particle swarm optimization algorithm (Chen and Yu 2005), and using evolutionary algorithms to optimize the neural networks (Eysa and Saeed 2005; Harpham 2004; Venkatesan 2009; Yao and Islam 2008), which have been proved feasible and effective....

    [...]

  • ...To overcome the disadvantages, many optimization algorithms have been introduced in the study and design of neural networks such as constructing a neural network based on particle swarm optimization algorithm (Chen and Yu 2005), and using evolutionary algorithms to optimize the neural networks (Eysa and Saeed 2005; Harpham 2004; Venkatesan 2009; Yao and Islam 2008), which have been proved feasible and effective....

    [...]