From the Publisher:
This is the first comprehensive treatment of feed-forward neural networks from the perspective of statistical pattern recognition. After introducing the basic concepts, the book examines techniques for modelling probability density functions and the properties and merits of the multi-layer perceptron and radial basis function network models. Also covered are various forms of error functions, principal algorithms for error function minimalization, learning and generalization in neural networks, and Bayesian techniques and their applications. Designed as a text, with over 100 exercises, this fully up-to-date work will benefit anyone involved in the fields of neural computation and pattern recognition.

/pdf/neural-networks-for-pattern-recognition-34puigiau1.pdf

Neural networks for pattern recognition

Numerical Optimization presents a comprehensive and up-to-date description of the most effective methods in continuous optimization. It responds to the growing interest in optimization in engineering, science, and business by focusing on the methods that are best suited to practical problems. For this new edition the book has been thoroughly updated throughout. There are new chapters on nonlinear interior methods and derivative-free methods for optimization, both of which are used widely in practice and the focus of much current research. Because of the emphasis on practical methods, as well as the extensive illustrations and exercises, the book is accessible to a wide audience. It can be used as a graduate text in engineering, operations research, mathematics, computer science, and business. It also serves as a handbook for researchers and practitioners in the field. The authors have strived to produce a text that is pleasant to read, informative, and rigorous - one that reveals both the beautiful nature of the discipline and its practical side.

Numerical Optimization

The Nelder--Mead simplex algorithm, first published in 1965, is an enormously popular direct search method for multidimensional unconstrained minimization. Despite its widespread use, essentially no theoretical results have been proved explicitly for the Nelder--Mead algorithm. This paper presents convergence properties of the Nelder--Mead algorithm applied to strictly convex functions in dimensions 1 and 2. We prove convergence to a minimizer for dimension 1, and various limited convergence results for dimension 2. A counterexample of McKinnon gives a family of strictly convex functions in two dimensions and a set of initial conditions for which the Nelder--Mead algorithm converges to a nonminimizer. It is not yet known whether the Nelder--Mead method can be proved to converge to a minimizer for a more specialized class of convex functions in two dimensions.

/pdf/convergence-properties-of-the-nelder-mead-simplex-method-in-1e36h79lmd.pdf

Convergence Properties of the Nelder--Mead Simplex Method in Low Dimensions

We study the numerical performance of a limited memory quasi-Newton method for large scale optimization, which we call the L-BFGS method. We compare its performance with that of the method developed by Buckley and LeNir (1985), which combines cycles of BFGS steps and conjugate direction steps. Our numerical tests indicate that the L-BFGS method is faster than the method of Buckley and LeNir, and is better able to use additional storage to accelerate convergence. We show that the L-BFGS method can be greatly accelerated by means of a simple scaling. We then compare the L-BFGS method with the partitioned quasi-Newton method of Griewank and Toint (1982a). The results show that, for some problems, the partitioned quasi-Newton method is clearly superior to the L-BFGS method. However we find that for other problems the L-BFGS method is very competitive due to its low iteration cost. We also study the convergence properties of the L-BFGS method, and prove global convergence on uniformly convex problems.

/pdf/on-the-limited-memory-bfgs-method-for-large-scale-5f8hu2qim3.pdf

On the limited memory BFGS method for large scale optimization

In many engineering optimization problems, the number of function evaluations is severely limited by time or cost. These problems pose a special challenge to the field of global optimization, since existing methods often require more function evaluations than can be comfortably afforded. One way to address this challenge is to fit response surfaces to data collected by evaluating the objective and constraint functions at a few points. These surfaces can then be used for visualization, tradeoff analysis, and optimization. In this paper, we introduce the reader to a response surface methodology that is especially good at modeling the nonlinear, multimodal functions that often occur in engineering. We then show how these approximating functions can be used to construct an efficient global optimization algorithm with a credible stopping rule. The key to using response surfaces for global optimization lies in balancing the need to exploit the approximating surface (by sampling where it is minimized) with the need to improve the approximation (by sampling where prediction error may be high). Striking this balance requires solving certain auxiliary problems which have previously been considered intractable, but we show how these computational obstacles can be overcome.

/pdf/efficient-global-optimization-of-expensive-black-box-4m8fv6oubz.pdf

Efficient Global Optimization of Expensive Black-Box Functions

: This report presents a set of test problems for nonlinear equations and nonlinear least-squares algorithms. These problems, sent to the authors by C. V. Nelson of the Maine Medical Center, come from a dipole model of the heart. They are 6x6 or 8x8, easy to code, cheap to evaluate, but not easy to solve. In support of the latter contention, they present test results from MINPACK and NL2SOL.

A New Nonlinear Equations Test Problem

/pdf/on-newton-like-iteration-functions-general-convergence-4lr258lunr.pdf

On Newton-like iteration functions: General convergence theorems and a specific algorithm

A fundamental part of geophysics is to make inferences about the interior of the earth on the basis of data collected at or near the surface of the earth. In almost all cases these measured data are only indirectly related to the properties of the earth that are of interest, so an inverse problem must be solved in order to obtain estimates of the physical properties within the earth. In February of 1999 the U.S. Department of Energy sponsored a workshop that was intended to examine the methods currently being used to solve geophysical inverse problems and to consider what new approaches should be explored in the future. The interdisciplinary area between inverse problems in geophysics and optimization methods in mathematics was specifically targeted as one where an interchange of ideas was likely to be fruitful. Thus about half of the participants were actively involved in solving geophysical inverse problems and about half were actively involved in research on general optimization methods. This report presents some of the topics that were explored at the workshop and the conclusions that were reached. In general, the objective of a geophysical inverse problem is to find an earth model, described by a set of physical parameters, that is consistent with the observational data. It is usually assumed that the forward problem, that of calculating simulated data for an earth model, is well enough understood so that reasonably accurate synthetic data can be generated for an arbitrary model. The inverse problem is then posed as an optimization problem, where the function to be optimized is variously called the objective function, misfit function, or fitness function. The objective function is typically some measure of the difference between observational data and synthetic data calculated for a trial model. However, because of incomplete and inaccurate data, the objective function often incorporates some additional form of regularization, such as a measure of smoothness or distance from a prior model. Various other constraints may also be imposed upon the process. Inverse problems are not restricted to geophysics, but can be found in a wide variety of disciplines where inferences must be made on the basis of indirect measurements. For instance, most imaging problems, whether in the field of medicine or non-destructive evaluation, require the solution of an inverse problem. In this report, however, the examples used for illustration are taken exclusively from the field of geophysics. The generalization of these examples to other disciplines should be straightforward, as all are based on standard second-order partial differential equations of physics. In fact, sometimes the non-geophysical inverse problems are significantly easier to treat (as in medical imaging) because the limitations on data collection, and in particular on multiple views, are not so severe as they generally are in geophysics. This report begins with an introduction to geophysical inverse problems by briefly describing four canonical problems that are typical of those commonly encountered in geophysics. Next the connection with optimization methods is made by presenting a general formulation of geophysical inverse problems. This leads into the main subject of this report, a discussion of methods for solving such problems with an emphasis upon newer approaches that have not yet become prominent in geophysics. A separate section is devoted to a subject that is not encountered in all optimization problems but is particularly important in geophysics, the need for a careful appraisal of the results in terms of their resolution and uncertainty. The impact on geophysical inverse problems of continuously improving computational resources is then discussed. The main results are then brought together in a final summary and conclusions section.

/pdf/optimization-and-geophysical-inverse-problems-ixestcy8j4.pdf

Optimization and geophysical inverse problems

Parallel continuous optimization methods are motivated here by applications in science and engineering The key issues are addressed at di erent computational levels including local and global optimization as well as strategies for large sparse versus small but expensive problems Topics covered include global optimization direct search with and without surrogates optimization of linked subsystems and variable and constraint distribution Finally there is a discussion of future research directions

John E. Dennis

Papers

A New Nonlinear Equations Test Problem

On Newton-like iteration functions: General convergence theorems and a specific algorithm

Optimization and geophysical inverse problems

Parallel continuous optimization

Surrogate Modelling and Space Mapping for Engineering Optimization: A Summary of the Danish Technical University November 2000 Workshop