Differential evolution (DE) is arguably one of the most powerful stochastic real-parameter optimization algorithms in current use. DE operates through similar computational steps as employed by a standard evolutionary algorithm (EA). However, unlike traditional EAs, the DE-variants perturb the current-generation population members with the scaled differences of randomly selected and distinct population members. Therefore, no separate probability distribution has to be used for generating the offspring. Since its inception in 1995, DE has drawn the attention of many researchers all over the world resulting in a lot of variants of the basic algorithm with improved performance. This paper presents a detailed review of the basic concepts of DE and a survey of its major variants, its application to multiobjective, constrained, large scale, and uncertain optimization problems, and the theoretical studies conducted on DE so far. Also, it provides an overview of the significant engineering applications that have benefited from the powerful nature of DE.

/pdf/differential-evolution-a-survey-of-the-state-of-the-art-3lz840pz2t.pdf

Differential Evolution: A Survey of the State-of-the-Art

A unified view of metaheuristics This book provides a complete background on metaheuristics and shows readers how to design and implement efficient algorithms to solve complex optimization problems across a diverse range of applications, from networking and bioinformatics to engineering design, routing, and scheduling. It presents the main design questions for all families of metaheuristics and clearly illustrates how to implement the algorithms under a software framework to reuse both the design and code. Throughout the book, the key search components of metaheuristics are considered as a toolbox for: Designing efficient metaheuristics (e.g. local search, tabu search, simulated annealing, evolutionary algorithms, particle swarm optimization, scatter search, ant colonies, bee colonies, artificial immune systems) for optimization problems Designing efficient metaheuristics for multi-objective optimization problems Designing hybrid, parallel, and distributed metaheuristics Implementing metaheuristics on sequential and parallel machines Using many case studies and treating design and implementation independently, this book gives readers the skills necessary to solve large-scale optimization problems quickly and efficiently. It is a valuable reference for practicing engineers and researchers from diverse areas dealing with optimization or machine learning; and graduate students in computer science, operations research, control, engineering, business and management, and applied mathematics.

/pdf/metaheuristics-from-design-to-implementation-1vqdfv8nk9.pdf

Metaheuristics: From Design to Implementation

Derived from the concept of self-adaptation in evolution strategies, the CMA (Covariance Matrix Adaptation) adapts the covariance matrix of a multi-variate normal search distribution. The CMA was originally designed to perform well with small populations. In this review, the argument starts out with large population sizes, reflecting recent extensions of the CMA algorithm. Commonalities and differences to continuous Estimation of Distribution Algorithms are analyzed. The aspects of reliability of the estimation, overall step size control, and independence from the coordinate system (invariance) become particularly important in small populations sizes. Consequently, performing the adaptation task with small populations is more intricate.

The CMA Evolution Strategy: A Comparing Review

Restart techniques are common in gradient-free optimization to deal with multimodal functions. Partial warm restarts are also gaining popularity in gradient-based optimization to improve the rate of convergence in accelerated gradient schemes to deal with ill-conditioned functions. In this paper, we propose a simple warm restart technique for stochastic gradient descent to improve its anytime performance when training deep neural networks. We empirically study its performance on the CIFAR-10 and CIFAR-100 datasets, where we demonstrate new state-of-the-art results at 3.14% and 16.21%, respectively. We also demonstrate its advantages on a dataset of EEG recordings and on a downsampled version of the ImageNet dataset. Our source code is available at this https URL

/pdf/sgdr-stochastic-gradient-descent-with-warm-restarts-266xcgab0r.pdf

SGDR: Stochastic Gradient Descent with Warm Restarts

Ant colony optimization for continuous domains

In this paper the performance of the CMA evolution strategy with rank-μ-update and weighted recombination is empirically investigated on eight multimodal test functions. In particular the effect of the population size λ on the performance is investigated. Increasing the population size remarkably improves the performance on six of the eight test functions. The optimal population size takes a wide range of values, but, with one exception, scales sub-linearly with the problem dimension. The global optimum can be located in all but one function. The performance for locating the global optimum scales between linear and cubic with the problem dimension. In a comparison to state-of-the-art global search strategies the CMA evolution strategy achieves superior performance on multimodal, non-separable test functions without intricate parameter tuning.

https://www.dsy.it/~wmotti/proj/cma-es/papers/Hansen,Kern_-_Evaluating_the_CMA_evolution_strategy_on_multimodal_test_functions.pdf

Evaluating the CMA Evolution Strategy on Multimodal Test Functions

The hydrodynamics of anguilliform swimming motions was investigated using three-dimensional simulations of the fluid flow past a self-propelled body. The motion of the body is not specified a priori, but is instead obtained through an evolutionary algorithm used to optimize the swimming efficiency and the burst swimming speed. The results of the present simulations support the hypothesis that anguilliform swimmers modify their kinematics according to different objectives and provide a quantitative analysis of the swimming motion and the forces experienced by the body. The kinematics of burst swimming is characterized by the large amplitude of the tail undulations while the anterior part of the body remains straight. In contrast, during efficient swimming behavior significant lateral undulation occurs along the entire length of the body. In turn, during burst swimming, the majority of the thrust is generated at the tail, whereas in the efficient swimming mode, in addition to the tail, the middle of the body contributes significantly to the thrust. The burst swimming velocity is 42% higher and the propulsive efficiency is 15% lower than the respective values during efficient swimming. The wake, for both swimming modes, consists largely of a double row of vortex rings with an axis aligned with the swimming direction. The vortex rings are responsible for producing lateral jets of fluid, which has been documented in prior experimental studies. We note that the primary wake vortices are qualitatively similar in both swimming modes except that the wake vortex rings are stronger and relatively more elongated in the fast swimming mode. The present results provide quantitative information of three-dimensional fluid-body interactions that may complement related experimental studies. In addition they enable a detailed quantitative analysis, which may be difficult to obtain experimentally, of the different swimming modes linking the kinematics of the motion with the forces acting on the self-propelled body. Finally, the optimization procedure helps to identify, in a systematic fashion, links between swimming motion and biological function.

Simulations of optimized anguilliform swimming.

We present a comparative review of Evolutionary Algorithms that generate new population members by sampling a probability distribution constructed during the optimization process. We present a unifying formulation for five such algorithms that enables us to characterize them based on the parametrization of the probability distribution, the learning methodology, and the use of historical information. The algorithms are evaluated on a number of test functions in order to assess their relative strengths and weaknesses. This comparative review helps to identify areas of applicability for the algorithms and to guide future algorithmic developments.

/pdf/learning-probability-distributions-in-continuous-5drq213hua.pdf

Learning Probability Distributions in Continuous Evolutionary Algorithms - a Comparative Review

We employ local meta-models to enhance the efficiency of evolution strategies in the optimization of computationally expensive problems. The method involves the combination of second order local regression meta-models with the Covariance Matrix Adaptation Evolution Strategy. Experiments on benchmark problems demonstrate that the proposed meta-models have the potential to reliably account for the ranking of the offspring population resulting in significant computational savings. The results show that the use of local meta-models significantly increases the efficiency of already competitive evolution strategies.

Local meta-models for optimization using evolution strategies

Direct numerical simulation (DNS) of the channel flow with an anisotropic compliant surface is performed in order to investigate its drag reduction effect in a fully developed turbulent flow. The computational domain is set to be 3δ×2δ×3δ, where δ is the channel half-width. The surface is passively driven by the pressure and wall-shear stress fluctuations, and the surface velocity provides a boundary condition for the fluid velocity field. An evolutionary optimization method (CMA-ES) is used to optimize the parameters of the anisotropic compliant surface. The optimization identifies several sets of parameters that result in a reduction of the friction drag with a maximum reduction rate of 8%. The primary mechanism for drag reduction is attributed to the decrease of the Reynolds shear stress (RSS) near the wall induced by the kinematics of the surface. The resultant wall motion is a uniform wave traveling downstream. The compliant wall, with the parameters found in the optimization study, is also tested in...

/pdf/evolutionary-optimization-of-an-anisotropic-compliant-2zoset0ji9.pdf

Stefan Kern

Papers

Evaluating the CMA Evolution Strategy on Multimodal Test Functions

Simulations of optimized anguilliform swimming.

Learning Probability Distributions in Continuous Evolutionary Algorithms - a Comparative Review

Local meta-models for optimization using evolution strategies

Evolutionary optimization of an anisotropic compliant surface for turbulent friction drag reduction