Numerical Optimization presents a comprehensive and up-to-date description of the most effective methods in continuous optimization. It responds to the growing interest in optimization in engineering, science, and business by focusing on the methods that are best suited to practical problems. For this new edition the book has been thoroughly updated throughout. There are new chapters on nonlinear interior methods and derivative-free methods for optimization, both of which are used widely in practice and the focus of much current research. Because of the emphasis on practical methods, as well as the extensive illustrations and exercises, the book is accessible to a wide audience. It can be used as a graduate text in engineering, operations research, mathematics, computer science, and business. It also serves as a handbook for researchers and practitioners in the field. The authors have strived to produce a text that is pleasant to read, informative, and rigorous - one that reveals both the beautiful nature of the discipline and its practical side.

Numerical Optimization

We consider the class of iterative shrinkage-thresholding algorithms (ISTA) for solving linear inverse problems arising in signal/image processing. This class of methods, which can be viewed as an extension of the classical gradient algorithm, is attractive due to its simplicity and thus is adequate for solving large-scale problems even with dense matrix data. However, such methods are also known to converge quite slowly. In this paper we present a new fast iterative shrinkage-thresholding algorithm (FISTA) which preserves the computational simplicity of ISTA but with a global rate of convergence which is proven to be significantly better, both theoretically and practically. Initial promising numerical results for wavelet-based image deblurring demonstrate the capabilities of FISTA which is shown to be faster than ISTA by several orders of magnitude.

http://people.rennes.inria.fr/Cedric.Herzet/Cedric.Herzet/Sparse_Seminar/Entrees/2012/11/12_A_Fast_Iterative_Shrinkage-Thresholding_Algorithmfor_Linear_Inverse_Problems_(A._Beck,_M._Teboulle)_files/Breck_2009.pdf

A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems

An algorithm for solving large nonlinear optimization problems with simple bounds is described. It is based on the gradient projection method and uses a limited memory BFGS matrix to approximate the Hessian of the objective function. It is shown how to take advantage of the form of the limited memory approximation to implement the algorithm efficiently. The results of numerical tests on a set of large problems are reported.

/pdf/a-limited-memory-algorithm-for-bound-constrained-3pm6n4lpp1.pdf

A limited memory algorithm for bound constrained optimization

* Glossary of notation * Introduction * Preliminaries * Existence and Construction of Generalized Inverses * Linear Systems and Characterization of Generalized Inverses * Minimal Properties of Generalized Inverses * Spectral Generalized Inverses * Generalized Inverses of Partitioned Matrices * A Spectral Theory for Rectangular Matrices * Computational Aspects of Generalized Inverses * Miscellaneous Applications * Generalized Inverses of Linear Operators between Hilbert Spaces * Appendix A: The Moore of the Moore-Penrose Inverse * Bibliography * Subject Index * Author Index

Generalized inverses: theory and applications

Mean shift, a simple interactive procedure that shifts each data point to the average of data points in its neighborhood is generalized and analyzed in the paper. This generalization makes some k-means like clustering algorithms its special cases. It is shown that mean shift is a mode-seeking process on the surface constructed with a "shadow" kernal. For Gaussian kernels, mean shift is a gradient mapping. Convergence is studied for mean shift iterations. Cluster analysis if treated as a deterministic problem of finding a fixed point of mean shift that characterizes the data. Applications in clustering and Hough transform are demonstrated. Mean shift is also considered as an evolutionary strategy that performs multistart global optimization. >

/pdf/mean-shift-mode-seeking-and-clustering-2mzfnll1c8.pdf

Mean shift, mode seeking, and clustering

Preface to the Classics Edition Preface Acknowledgments Glossary of Symbols Introduction Part I. Background Material. 1. Sample Problems 2. Linear Algebra 3. Analysis Part II. Nonconstructive Existence Theorems. 4. Gradient Mappings and Minimization 5. Contractions and the Continuation Property 6. The Degree of a Mapping Part III. Iterative Methods. 7. General Iterative Methods 8. Minimization Methods Part IV. Local Convergence. 9. Rates of Convergence-General 10. One-Step Stationary Methods 11. Multistep Methods and Additional One-Step Methods Part V. Semilocal and Global Convergence. 12. Contractions and Nonlinear Majorants 13. Convergence under Partial Ordering 14. Convergence of Minimization Methods An Annotated List of Basic Reference Books Bibliography Author Index Subject Index.

Iterative Solution of Nonlinear Equations in Several Variables

Monotone iterations for nonlinear elliptic differential equations in boundary-value problems applied to Gauss-Seidel methods

Monotone Iterations for Nonlinear Equations with Application to Gauss-Seidel Methods

Asymptotic rate of convergence of Gauss-Seidel type iterative processes for nonlinear difference equations

/pdf/nonlinear-difference-equations-and-gauss-seidel-type-2fuj7n9y4c.pdf

Nonlinear Difference Equations and Gauss-Seidel Type Iterative Methods

In this report, we review several connections between the theory of convergence of iterative processes and the theory of Lyapunov stability of ordinary difference equations. Among the topics discussed are local convergence and asymptotic stability, global convergence and global asymptotic stability, Lyapunov functions, domain of attraction, total stability and rounding errors, nonautonomous equations and variable operator iterations.

Stability of Difference Equations and Convergence of Iterative Processes

This chapter discusses the processes that may be viewed as generalizations of methods for the solution of one equation in one unknown, such as the n -dimensional counterparts of Newton's method, the secant method, Steffensen's method, and their variations. The chapter discusses generalizations of iterative methods for linear systems of equations, with particular emphasis on successive over-relaxation methods. A number of other processes are also discussed in the chapter, which are attempts to provide a technique for overcoming the problem of finding a suitable initial approximation. In many applications, the system of equations to be solved arises in the attempt to find a minimizer or a critical point of a related nonlinear functional. The Newton and secant methods represent generalizations of one-dimensional iteration methods. The chapter discusses a class of processes that arise from iterative methods used for linear systems of equations. A characteristic of the generalized linear methods is that they all exhibit only a linear rate of convergence except under special circumstances.

J.M. Ortega

Papers

Iterative Solution of Nonlinear Equations in Several Variables

Monotone Iterations for Nonlinear Equations with Application to Gauss-Seidel Methods

Nonlinear Difference Equations and Gauss-Seidel Type Iterative Methods

Stability of Difference Equations and Convergence of Iterative Processes

General iterative methods