From the Publisher:
This is the first comprehensive treatment of feed-forward neural networks from the perspective of statistical pattern recognition. After introducing the basic concepts, the book examines techniques for modelling probability density functions and the properties and merits of the multi-layer perceptron and radial basis function network models. Also covered are various forms of error functions, principal algorithms for error function minimalization, learning and generalization in neural networks, and Bayesian techniques and their applications. Designed as a text, with over 100 exercises, this fully up-to-date work will benefit anyone involved in the fields of neural computation and pattern recognition.

/pdf/neural-networks-for-pattern-recognition-34puigiau1.pdf

Neural networks for pattern recognition

Numerical Optimization presents a comprehensive and up-to-date description of the most effective methods in continuous optimization. It responds to the growing interest in optimization in engineering, science, and business by focusing on the methods that are best suited to practical problems. For this new edition the book has been thoroughly updated throughout. There are new chapters on nonlinear interior methods and derivative-free methods for optimization, both of which are used widely in practice and the focus of much current research. Because of the emphasis on practical methods, as well as the extensive illustrations and exercises, the book is accessible to a wide audience. It can be used as a graduate text in engineering, operations research, mathematics, computer science, and business. It also serves as a handbook for researchers and practitioners in the field. The authors have strived to produce a text that is pleasant to read, informative, and rigorous - one that reveals both the beautiful nature of the discipline and its practical side.

Numerical Optimization

The Nelder--Mead simplex algorithm, first published in 1965, is an enormously popular direct search method for multidimensional unconstrained minimization. Despite its widespread use, essentially no theoretical results have been proved explicitly for the Nelder--Mead algorithm. This paper presents convergence properties of the Nelder--Mead algorithm applied to strictly convex functions in dimensions 1 and 2. We prove convergence to a minimizer for dimension 1, and various limited convergence results for dimension 2. A counterexample of McKinnon gives a family of strictly convex functions in two dimensions and a set of initial conditions for which the Nelder--Mead algorithm converges to a nonminimizer. It is not yet known whether the Nelder--Mead method can be proved to converge to a minimizer for a more specialized class of convex functions in two dimensions.

/pdf/convergence-properties-of-the-nelder-mead-simplex-method-in-1e36h79lmd.pdf

Convergence Properties of the Nelder--Mead Simplex Method in Low Dimensions

We study the numerical performance of a limited memory quasi-Newton method for large scale optimization, which we call the L-BFGS method. We compare its performance with that of the method developed by Buckley and LeNir (1985), which combines cycles of BFGS steps and conjugate direction steps. Our numerical tests indicate that the L-BFGS method is faster than the method of Buckley and LeNir, and is better able to use additional storage to accelerate convergence. We show that the L-BFGS method can be greatly accelerated by means of a simple scaling. We then compare the L-BFGS method with the partitioned quasi-Newton method of Griewank and Toint (1982a). The results show that, for some problems, the partitioned quasi-Newton method is clearly superior to the L-BFGS method. However we find that for other problems the L-BFGS method is very competitive due to its low iteration cost. We also study the convergence properties of the L-BFGS method, and prove global convergence on uniformly convex problems.

/pdf/on-the-limited-memory-bfgs-method-for-large-scale-5f8hu2qim3.pdf

On the limited memory BFGS method for large scale optimization

In many engineering optimization problems, the number of function evaluations is severely limited by time or cost. These problems pose a special challenge to the field of global optimization, since existing methods often require more function evaluations than can be comfortably afforded. One way to address this challenge is to fit response surfaces to data collected by evaluating the objective and constraint functions at a few points. These surfaces can then be used for visualization, tradeoff analysis, and optimization. In this paper, we introduce the reader to a response surface methodology that is especially good at modeling the nonlinear, multimodal functions that often occur in engineering. We then show how these approximating functions can be used to construct an efficient global optimization algorithm with a credible stopping rule. The key to using response surfaces for global optimization lies in balancing the need to exploit the approximating surface (by sampling where it is minimized) with the need to improve the approximation (by sampling where prediction error may be high). Striking this balance requires solving certain auxiliary problems which have previously been considered intractable, but we show how these computational obstacles can be overcome.

/pdf/efficient-global-optimization-of-expensive-black-box-4m8fv6oubz.pdf

Efficient Global Optimization of Expensive Black-Box Functions

This paper considers the problem of minimizing the $L_p$-norm of the truncation error for consistent multistep methods for integrating an ordinary differential equation. It is shown that the minimum is attained uniquely by the trapezoidal rule. For a given left-hand side, the right-hand side which gives a minimum is constructed, and it is proved that this method is essentially the trapezoidal rule.

Some Minimum Properties of the Trapezoidal Rule

Author(s): Meza, Juan C.; Garcia-Lekue, Arantzazu; Abramson, Mark A.; Dennis, John E. | Abstract: Many properties of nanostructures depend on the atomic configuration at the surface. One common technique used for determining this surface structure is based on the low energy electron diffraction (LEED) method, which uses a high-fidelity physics model to compare experimental results with spectra computed via a computer simulation. While this approach is highly effective, the computational cost of the simulations can be prohibitive for large systems. In this work, we propose the use of a direct search method in conjunction with an additive surrogate. This surrogate is constructed from a combination of a simplified physics model and an interpolation that is based on the differences between the simplified physics model and the full fidelity model.

/pdf/derivative-free-optimization-methods-for-surface-structure-y9pdyyfxv8.pdf

Derivative-free optimization methods for surface structure determination of nanosystems

: A very important problem in numerical optimization is to find a way to update a sparse Hessian approximation so that it will be positive-definite under reasonable circumstances. This problem has motivated research -- which is yet to show much progress -- toward a "sparse BFGS method." In this paper, the authors suggest a different approach to the problem based on using a sparse Broyden, or Schubert, update directly on the Cholesky factor of the current Hessian approximation to define the next Hessian approximation implicitly in terms of its Cholesky factorization. This approach has the added advantage of being able to cheaply find the Newton step, since no factorization step is required. The difficulty with the approach is in finding a satisfactory secant or quasi-Newton condition to use in the update.

/pdf/toward-direct-sparse-updates-of-cholesky-factors-33m25lzfqt.pdf

Toward Direct Sparse Updates of Cholesky Factors

The paper proposes a framework for sensitivity analyses of blackbox constrained optimization problems for which Lagrange multipliers are not available. Two strategies are developed to analyze the sensitivity of the optimal objective function value to general constraints. These are a simple method which may be performed immediately after a single optimization, and a detailed method performing biobjective optimization on the minimization of the objective versus the constraint of interest. The detailed method provides points on the Pareto front of the objective versus a chosen constraint. The proposed methods are tested on an academic test case and on an engineering problem using the mesh adaptive direct search algorithm.

Sensitivity to Constraints in Blackbox Optimization

: CONCLUSIONS: We have found an effective method for mixed variable problems ... BUT ... It requires serious thinking by the user as to what constitutes an acceptable solution for each problem * It is not easy to incorporate surrogates with categorical variables * It is unlikely to be effective for many categorical variables * We have analyzed the method, BUT..., better results depend on the filter method for continuous variables.

John E. Dennis

Papers

Some Minimum Properties of the Trapezoidal Rule

Derivative-free optimization methods for surface structure determination of nanosystems

Toward Direct Sparse Updates of Cholesky Factors

Sensitivity to Constraints in Blackbox Optimization

Pattern Search for Mixed Variable Optimization Problems