From the Publisher:
This is the first comprehensive treatment of feed-forward neural networks from the perspective of statistical pattern recognition. After introducing the basic concepts, the book examines techniques for modelling probability density functions and the properties and merits of the multi-layer perceptron and radial basis function network models. Also covered are various forms of error functions, principal algorithms for error function minimalization, learning and generalization in neural networks, and Bayesian techniques and their applications. Designed as a text, with over 100 exercises, this fully up-to-date work will benefit anyone involved in the fields of neural computation and pattern recognition.

/pdf/neural-networks-for-pattern-recognition-34puigiau1.pdf

Neural networks for pattern recognition

Numerical Optimization presents a comprehensive and up-to-date description of the most effective methods in continuous optimization. It responds to the growing interest in optimization in engineering, science, and business by focusing on the methods that are best suited to practical problems. For this new edition the book has been thoroughly updated throughout. There are new chapters on nonlinear interior methods and derivative-free methods for optimization, both of which are used widely in practice and the focus of much current research. Because of the emphasis on practical methods, as well as the extensive illustrations and exercises, the book is accessible to a wide audience. It can be used as a graduate text in engineering, operations research, mathematics, computer science, and business. It also serves as a handbook for researchers and practitioners in the field. The authors have strived to produce a text that is pleasant to read, informative, and rigorous - one that reveals both the beautiful nature of the discipline and its practical side.

Numerical Optimization

The Nelder--Mead simplex algorithm, first published in 1965, is an enormously popular direct search method for multidimensional unconstrained minimization. Despite its widespread use, essentially no theoretical results have been proved explicitly for the Nelder--Mead algorithm. This paper presents convergence properties of the Nelder--Mead algorithm applied to strictly convex functions in dimensions 1 and 2. We prove convergence to a minimizer for dimension 1, and various limited convergence results for dimension 2. A counterexample of McKinnon gives a family of strictly convex functions in two dimensions and a set of initial conditions for which the Nelder--Mead algorithm converges to a nonminimizer. It is not yet known whether the Nelder--Mead method can be proved to converge to a minimizer for a more specialized class of convex functions in two dimensions.

/pdf/convergence-properties-of-the-nelder-mead-simplex-method-in-1e36h79lmd.pdf

Convergence Properties of the Nelder--Mead Simplex Method in Low Dimensions

We study the numerical performance of a limited memory quasi-Newton method for large scale optimization, which we call the L-BFGS method. We compare its performance with that of the method developed by Buckley and LeNir (1985), which combines cycles of BFGS steps and conjugate direction steps. Our numerical tests indicate that the L-BFGS method is faster than the method of Buckley and LeNir, and is better able to use additional storage to accelerate convergence. We show that the L-BFGS method can be greatly accelerated by means of a simple scaling. We then compare the L-BFGS method with the partitioned quasi-Newton method of Griewank and Toint (1982a). The results show that, for some problems, the partitioned quasi-Newton method is clearly superior to the L-BFGS method. However we find that for other problems the L-BFGS method is very competitive due to its low iteration cost. We also study the convergence properties of the L-BFGS method, and prove global convergence on uniformly convex problems.

/pdf/on-the-limited-memory-bfgs-method-for-large-scale-5f8hu2qim3.pdf

On the limited memory BFGS method for large scale optimization

In many engineering optimization problems, the number of function evaluations is severely limited by time or cost. These problems pose a special challenge to the field of global optimization, since existing methods often require more function evaluations than can be comfortably afforded. One way to address this challenge is to fit response surfaces to data collected by evaluating the objective and constraint functions at a few points. These surfaces can then be used for visualization, tradeoff analysis, and optimization. In this paper, we introduce the reader to a response surface methodology that is especially good at modeling the nonlinear, multimodal functions that often occur in engineering. We then show how these approximating functions can be used to construct an efficient global optimization algorithm with a credible stopping rule. The key to using response surfaces for global optimization lies in balancing the need to exploit the approximating surface (by sampling where it is minimized) with the need to improve the approximation (by sampling where prediction error may be high). Striking this balance requires solving certain auxiliary problems which have previously been considered intractable, but we show how these computational obstacles can be overcome.

/pdf/efficient-global-optimization-of-expensive-black-box-4m8fv6oubz.pdf

Efficient Global Optimization of Expensive Black-Box Functions

A standard engineering practice is the use of approximation models in place of expensive simulations to drive an optimal design process based on nonlinear programming algorithms. The use of approximation techniques is intended to reduce the number of detailed, costly analyses required during optimization while maintaining the salient features of the design problem. The question we address is how to manage the interplay between the optimization and the fidelity of the approximation models to ensure that the process converges to a solution of the original design problem. Using well-established notions from the literature on trust-region methods and a powerful global convergence theory for pattern search methods, we can ensure that the optimization process converges to a solution of the original design problem.

/pdf/managing-approximation-models-in-optimization-2sovn7amop.pdf

Managing approximation models in optimization

: Finding the unconstrained minimizer of a function of more than one variable is an important problem with many practical applications, including data fitting, engineering design, and process control. In addition, techniques for solving unconstrained optimization problems form the basis for most methods for solving constrained optimization problems. This paper surveys the state of the art for solving unconstrained optimization problems and the closely related problem of solving systems of nonlinear equations. First the authors briefly give some mathematical background. Then they discuss Newton's method, the fundamental method underlying most approaches to these problems, as well as the inexact Newton method. The two main practical deficiencies of Newton's method, the need for analytic derivatives and the possible failure to converge to the solution from poor starting points, are the key issues in unconstrained optimization, and the extension of these techniques to solving large, sparse problems. Then this document discusses the main methods used to ensure convergence from poor starting points, line search methods and trust region methods. Also discusses are two rather different approaches to unconstrained optimization, the Nelder-Meade simplex method and conjugate direction methods.

A view of unconstrained optimization

This paper presents a convergence rate analysis for interior point primal-dual linear programming algorithms. Conditions that guarantee Q-superlinear convergence are identified in two distinct theories. Both state that, under appropriate assumptions, Q-superlinear convergence is achieved by asymptotically taking the step to the boundary of the positive orthant and letting the barrier parameter approach zero at a rate that is superlinearly faster than the convergence of the duality gap to zero. The first theory makes no nondegeneracy assumption and explains why in recent numerical experimentation Q-superlinear convergence was always observed. The second theory requires the restrictive assumption of primal nondegeneracy. However, it gives the surprising result that Q-superlinear convergence can still be attained even if centering is not phased out, provided the iterates asymptotically approach the central path. The latter theory is extended to produce a satisfactory Q-quadratic convergence theory. It requir...

/pdf/on-the-superlinear-and-quadratic-convergence-of-primal-dual-291kanvps7.pdf

On the Superlinear and Quadratic Convergence of Primal-Dual Interior Point Linear Programming Algorithms

The class of Mesh Adaptive Direct Search (Mads) algorithms is designed for the optimization of constrained black-box problems. The purpose of this paper is to compare instantiations of Mads under different strategies to handle constraints. Intensive numerical tests are conducted from feasible and/or infeasible starting points on three real engineering applications.

https://www.gerad.ca/Charles.Audet/Slides/2009-chicago.pdf

Globalization strategies for Mesh Adaptive Direct Search

In the literature, thermal insulation systems with a fixed number of heat intercepts have been optimized with respect to intercept locations and temperatures. The number of intercepts and the types of insulators that surround them were chosen by parametric studies. This was because the optimization methods used could not treat such categorical variables. Discrete optimization variables are categorical if the objective function or the constraints can not be evaluated unless the variables take one of a prescribed enumerable set of values. The key issue is that categorical variables can not be treated as ordinary discrete variables are treated by relaxing them to continuous variables with a side constraint that they be discrete at the solution.

/pdf/mixed-variable-optimization-of-the-number-and-composition-of-20sh3nxfbr.pdf

John E. Dennis

Papers

Managing approximation models in optimization

A view of unconstrained optimization

On the Superlinear and Quadratic Convergence of Primal-Dual Interior Point Linear Programming Algorithms

Globalization strategies for Mesh Adaptive Direct Search

Mixed Variable Optimization of the Number and Composition of Heat Intercepts in a Thermal Insulation System