Restart techniques are common in gradient-free optimization to deal with multimodal functions. Partial warm restarts are also gaining popularity in gradient-based optimization to improve the rate of convergence in accelerated gradient schemes to deal with ill-conditioned functions. In this paper, we propose a simple warm restart technique for stochastic gradient descent to improve its anytime performance when training deep neural networks. We empirically study its performance on the CIFAR-10 and CIFAR-100 datasets, where we demonstrate new state-of-the-art results at 3.14% and 16.21%, respectively. We also demonstrate its advantages on a dataset of EEG recordings and on a downsampled version of the ImageNet dataset. Our source code is available at this https URL

SGDR: Stochastic Gradient Descent with Warm Restarts

Salp Swarm Algorithm

/pdf/sgdr-stochastic-gradient-descent-with-warm-restarts-266xcgab0r.pdf

This tutorial introduces the CMA Evolution Strategy (ES), where CMA stands for Covariance Matrix Adaptation. The CMA-ES is a stochastic, or randomized, method for real-parameter (continuous domain) optimization of non-linear, non-convex functions. We try to motivate and derive the algorithm from intuitive concepts and from requirements of non-linear, non-convex search in continuous domain.

/pdf/the-cma-evolution-strategy-a-tutorial-209ln9p7fz.pdf

The CMA Evolution Strategy: A Tutorial

This paper presents a new cooperative coevolving particle swarm optimization (CCPSO) algorithm in an attempt to address the issue of scaling up particle swarm optimization (PSO) algorithms in solving large-scale optimization problems (up to 2000 real-valued variables). The proposed CCPSO2 builds on the success of an early CCPSO that employs an effective variable grouping technique random grouping. CCPSO2 adopts a new PSO position update rule that relies on Cauchy and Gaussian distributions to sample new points in the search space, and a scheme to dynamically determine the coevolving subcomponent sizes of the variables. On high-dimensional problems (ranging from 100 to 2000 variables), the performance of CCPSO2 compared favorably against a state-of-the-art evolutionary algorithm sep-CMA-ES, two existing PSO algorithms, and a cooperative coevolving differential evolution algorithm. In particular, CCPSO2 performed significantly better than sep-CMA-ES and two existing PSO algorithms on more complex multimodal problems (which more closely resemble real-world problems), though not as well as the existing algorithms on unimodal functions. Our experimental results and analysis suggest that CCPSO2 is a highly competitive optimization algorithm for solving large-scale and complex multimodal optimization problems.

Cooperatively Coevolving Particle Swarms for Large Scale Optimization

This paper presents results of the BBOB-2009 benchmarking of 31 search algorithms on 24 noiseless functions in a black-box optimization scenario in continuous domain. The runtime of the algorithms, measured in number of function evaluations, is investigated and a connection between a single convergence graph and the runtime distribution is uncovered. Performance is investigated for different dimensions up to 40-D, for different target precision values, and in different subgroups of functions. Searching in larger dimension and multi-modal functions appears to be more difficult. The choice of the best algorithm also depends remarkably on the available budget of function evaluations.

/pdf/comparing-results-of-31-algorithms-from-the-black-box-q5wwguk3vb.pdf

Comparing results of 31 algorithms from the black-box optimization benchmarking BBOB-2009

In a companion paper, we presented a weighted negative update of the covariance matrix in the CMA-ES--weighted active CMA-ES or, in short, aCMA-ES. In this paper, we benchmark the IPOP-aCMA-ES on the BBOB-2010 noisy testbed in search space dimensions between 2 and 40 and compare its performance with the IPOP-CMA-ES.The aCMA suffers from a moderate performance loss, of less than a factor of two, on the sphere function with two different noise models. On the other hand, the aCMA enjoys a (significant) performance gain, up to a factor of four, on 13 unimodal functions in various dimensions, in particular the larger ones. Compared to the best performance observed during BBOB-2009, the IPOP-aCMA-ES sets a new record on overall ten functions. The global picture is in favor of aCMA which might establish a new standard also for noisy problems.

/pdf/benchmarking-a-weighted-negative-covariance-matrix-update-on-21tb8z2kzs.pdf

Benchmarking a weighted negative covariance matrix update on the BBOB-2010 noiseless testbed

The BFGS quasi-Newton method is benchmarked on the noiseless BBOB-2009 testbed. A multistart strategy is applied with a maximum number of function evaluations of 105 times the search space dimension, resulting in the algorithm solving six functions.

/pdf/benchmarking-the-bfgs-algorithm-on-the-bbob-2009-function-1smycltqjv.pdf

Benchmarking the BFGS algorithm on the BBOB-2009 function testbed

Differential Evolution is a popular powerful optimization algorithm for continuous problems. Part of its efficiency comes from the availability of several mutation strategies that can (and must) be chosen in a problem-dependent way. However, such flexibility also makes DE difficult to be automatically used in a new context. F-AUC-Bandit is a comparison-based Adaptive Operator Selection method that has been proposed in the GA framework. It is used here for the on-line control of DE mutation strategy, thus preserving DE invariance w.r.t. monotonous transformations of the objective function. The approach is comparatively assessed on the BBOB test suite, demonstrating significant improvement on baseline and other Adaptive Strategy Selection approaches, while presenting a very low sensitivity to hyper-parameter setting.

/pdf/comparison-based-adaptive-strategy-selection-with-bandits-in-2aknk7o4bx.pdf

Comparison-based adaptive strategy selection with bandits in differential evolution

The NEWUOA which belongs to the class of Derivative-Free optimization algorithms is benchmarked on the BBOB-2009 noisefree testbed. A multistart strategy is applied with a maximum number of function evaluations of up to 10^5 times the search space dimension resulting in the algorithm solving 11 functions in 20-D. The results of the algorithm using the recommended number of interpolation points for the underlying model and the full model are shown and discussed.

/pdf/benchmarking-the-newuoa-on-the-bbob-2009-function-testbed-434jvmkmz4.pdf

Raymond Ros

Papers

Comparing results of 31 algorithms from the black-box optimization benchmarking BBOB-2009

Benchmarking a weighted negative covariance matrix update on the BBOB-2010 noiseless testbed

Benchmarking the BFGS algorithm on the BBOB-2009 function testbed

Comparison-based adaptive strategy selection with bandits in differential evolution

Benchmarking the NEWUOA on the BBOB-2009 function testbed