Introduction to linear and nonlinear programming

We present a new algorithm for finding the global minimum of a multivariate function subject to simple bounds. The algorithm is a modification of the standard Lipschitzian approach that eliminates the need to specify a Lipschitz constant. This is done by carrying out simultaneous searches using all possible constants from zero to infinity. On nine standard test functions, the new algorithm converges in fewer function evaluations than most competing methods.

The motivation for the new algorithm stems from a different way of looking at the Lipschitz constant. In particular, the Lipschitz constant is viewed as a weighting parameter that indicates how much emphasis to place on global versus local search. In standard Lipschitzian methods, this constant is usually large because it must equal or exceed the maximum rate of change of the objective function. As a result, these methods place a high emphasis on global search and exhibit slow convergence. In contrast, the new algorithm carries out simultaneous searches using all possible constants, and therefore operates at both the global and local level. Once the global part of the algorithm finds the basin of convergence of the optimum, the local part of the algorithm quickly and automatically exploits it. This accounts for the fast convergence of the new algorithm on the test functions.

/pdf/lipschitzian-optimization-without-the-lipschitz-constant-1lc0hthf81.pdf

Lipschitzian optimization without the Lipschitz constant

Many scientific, engineering and economic problems involve the optimisation of a set of parameters. These problems include examples like minimising the losses in a power grid by finding the optimal configuration of the components, or training a neural network to recognise images of people's faces. Numerous optimisation algorithms have been proposed to solve these problems, with varying degrees of success. The Particle Swarm Optimiser (PSO) is a relatively new technique that has been empirically shown to perform well on many of these optimisation problems. This thesis presents a theoretical model that can be used to describe the long-term behaviour of the algorithm. An enhanced version of the Particle Swarm Optimiser is constructed and shown to have guaranteed convergence on local minima. This algorithm is extended further, resulting in an algorithm with guaranteed convergence on global minima. A model for constructing cooperative PSO algorithms is developed, resulting in the introduction of two new PSO-based algorithms. Empirical results are presented to support the theoretical properties predicted by the various models, using synthetic benchmark functions to investigate specific properties. The various PSO-based algorithms are then applied to the task of training neural networks, corroborating the results obtained on the synthetic benchmark functions.

/pdf/an-analysis-of-particle-swarm-optimizers-3iz2tj39rw.pdf

An analysis of particle swarm optimizers

This paper presents an overview of our most recent results concerning the Particle Swarm Optimization (PSO) method. Techniques for the alleviation of local minima, and for detecting multiple minimizers are described. Moreover, results on the ability of the PSO in tackling Multiobjective, Minimax, Integer Programming and e1 errors-in-variables problems, as well as problems in noisy and continuously changing environments, are reported. Finally, a Composite PSO, in which the heuristic parameters of PSO are controlled by a Differential Evolution algorithm during the optimization, is described, and results for many well-known and widely used test functions are given.

Recent approaches to global optimization problems through Particle Swarm Optimization

A new structural optimization method based on the harmony search algorithm

A new and dynamic method for unconstrained minimization

A new multi-start algorithm for global unconstrained minimization is presented in which the search trajectories are derived from the equation of motion of a particle in a conservative force field, where the function to be minimized represents the potential energy. The trajectories are modified to increase the probability of convergence to a comparatively low local minimum, thus increasing the region of convergence of the global minimum. A Bayesian argument is adopted by which, under mild assumptions, the confidence level that the global minimum has been attained may be computed. When applied to standard and other test functions, the algorithm never failed to yield the global minimum.

A multi-start global minimization algorithm with dynamic search trajectories

Practical Mathematical Optimization

An improved version of the original leap-frog dynamic method for unconstrained minimization: LFOP1(b)

This paper describes an accurate and reliable new algorithm (LFOPC) for solving constrained optimization problems, through a three-phase application of the well-established leap-frog method for unconstrained optimization, to penalty function formulations of the original constrained problems. The algorithm represents a considerable improvement over an earlier version (LFOPCON) which requires the judicious choice of parameter settings for efficient use. The current algorithm automatically executes normalization and scaling operations on the gradients of the constraints. This results in a robust algorithm that, apart from convergence tolerances, requires virtually no parameter settings. The method has been well tested, on both standard analytical test problems and practical engineering design problems.

Jan A. Snyman

Papers

A new and dynamic method for unconstrained minimization

A multi-start global minimization algorithm with dynamic search trajectories

Practical Mathematical Optimization

An improved version of the original leap-frog dynamic method for unconstrained minimization: LFOP1(b)

The LFOPC leap-frog algorithm for constrained optimization