Numerical Optimization presents a comprehensive and up-to-date description of the most effective methods in continuous optimization. It responds to the growing interest in optimization in engineering, science, and business by focusing on the methods that are best suited to practical problems. For this new edition the book has been thoroughly updated throughout. There are new chapters on nonlinear interior methods and derivative-free methods for optimization, both of which are used widely in practice and the focus of much current research. Because of the emphasis on practical methods, as well as the extensive illustrations and exercises, the book is accessible to a wide audience. It can be used as a graduate text in engineering, operations research, mathematics, computer science, and business. It also serves as a handbook for researchers and practitioners in the field. The authors have strived to produce a text that is pleasant to read, informative, and rigorous - one that reveals both the beautiful nature of the discipline and its practical side.

Numerical Optimization

Today's Web-enabled deluge of electronic data calls for automated methods of data analysis. Machine learning provides these, developing methods that can automatically detect patterns in data and then use the uncovered patterns to predict future data. This textbook offers a comprehensive and self-contained introduction to the field of machine learning, based on a unified, probabilistic approach. The coverage combines breadth and depth, offering necessary background material on such topics as probability, optimization, and linear algebra as well as discussion of recent developments in the field, including conditional random fields, L1 regularization, and deep learning. The book is written in an informal, accessible style, complete with pseudo-code for the most important algorithms. All topics are copiously illustrated with color images and worked examples drawn from such application domains as biology, text processing, computer vision, and robotics. Rather than providing a cookbook of different heuristic methods, the book stresses a principled model-based approach, often using the language of graphical models to specify models in a concise and intuitive way. Almost all the models described have been implemented in a MATLAB software package--PMTK (probabilistic modeling toolkit)--that is freely available online. The book is suitable for upper-level undergraduates with an introductory-level college math background and beginning graduate students.

Machine Learning : A Probabilistic Perspective

Many problems in signal processing and statistical inference involve finding sparse solutions to under-determined, or ill-conditioned, linear systems of equations. A standard approach consists in minimizing an objective function which includes a quadratic (squared ) error term combined with a sparseness-inducing regularization term. Basis pursuit, the least absolute shrinkage and selection operator (LASSO), wavelet-based deconvolution, and compressed sensing are a few well-known examples of this approach. This paper proposes gradient projection (GP) algorithms for the bound-constrained quadratic programming (BCQP) formulation of these problems. We test variants of this approach that select the line search parameters in different ways, including techniques based on the Barzilai-Borwein method. Computational experiments show that these GP approaches perform well in a wide range of applications, often being significantly faster (in terms of computation time) than competing methods. Although the performance of GP methods tends to degrade as the regularization term is de-emphasized, we show how they can be embedded in a continuation scheme to recover their efficient practical performance.

/pdf/gradient-projection-for-sparse-reconstruction-application-to-jxqvrma0ll.pdf

Gradient Projection for Sparse Reconstruction: Application to Compressed Sensing and Other Inverse Problems

Millimeter wave (mmWave) signals experience orders-of-magnitude more pathloss than the microwave signals currently used in most wireless applications and all cellular systems. MmWave systems must therefore leverage large antenna arrays, made possible by the decrease in wavelength, to combat pathloss with beamforming gain. Beamforming with multiple data streams, known as precoding, can be used to further improve mmWave spectral efficiency. Both beamforming and precoding are done digitally at baseband in traditional multi-antenna systems. The high cost and power consumption of mixed-signal devices in mmWave systems, however, make analog processing in the RF domain more attractive. This hardware limitation restricts the feasible set of precoders and combiners that can be applied by practical mmWave transceivers. In this paper, we consider transmit precoding and receiver combining in mmWave systems with large antenna arrays. We exploit the spatial structure of mmWave channels to formulate the precoding/combining problem as a sparse reconstruction problem. Using the principle of basis pursuit, we develop algorithms that accurately approximate optimal unconstrained precoders and combiners such that they can be implemented in low-cost RF hardware. We present numerical results on the performance of the proposed algorithms and show that they allow mmWave systems to approach their unconstrained performance limits, even when transceiver hardware constraints are considered.

Spatially Sparse Precoding in Millimeter Wave MIMO Systems

Much has been written about theory and practice in the law, and the tension between practitioners and theorists. Judges do not cite theoretical articles often; they rarely "apply" theories to particular cases. These arguments are not revisited. Instead the Essay explores the working and interaction of theory and practice, practitioners and theorists. The Essay starts with a story about solving a legal issue using our intellectual tools - theory, practice, and their progenies: experience and "gut." Next the Essay elaborates on the nature of theory, practice, experience and "gut." The third part of the Essay discusses theories that are helpful to practitioners and those that are less helpful. The Essay concludes that practitioners theorize, and theorists practice. They use these intellectual tools differently because the goals and orientations of theorists and practitioners, and the constraints under which they act, differ. Theory, practice, experience and "gut" help us think, remember, decide and create. They complement each other like the two sides of the same coin: distinct but inseparable.

Of Theory and Practice

Nonmonotone projected gradient techniques are considered for the minimization of differentiable functions on closed convex sets. The classical projected gradient schemes are extended to include a nonmonotone steplength strategy that is based on the Grippo--Lampariello--Lucidi nonmonotone line search. In particular, the nonmonotone strategy is combined with the spectral gradient choice of steplength to accelerate the convergence process. In addition to the classical projected gradient nonlinear path, the feasible spectral projected gradient is used as a search direction to avoid additional trial projections during the one-dimensional search process. Convergence properties and extensive numerical results are presented.

/pdf/nonmonotone-spectral-projected-gradient-methods-on-convex-1tesp83age.pdf

Nonmonotone Spectral Projected Gradient Methods on Convex Sets

The Barzilai and Borwein gradient method for the solution of large scale unconstrained minimization problems is considered. This method requires few storage locations and very inexpensive computations. Furthermore, it does not guarantee descent in the objective function and no line search is required. Recently, the global convergence for the convex quadratic case has been established. However, for the nonquadratic case, the method needs to be incorporated in a globalization scheme. In this work, a nonmonotone line search strategy that guarantees global convergence is combined with the Barzilai and Borwein method. This strategy is based on the nonmonotone line search technique proposed by Grippo, Lampariello, and Lucidi [SIAM J. Numer. Anal., 23 (1986), pp. 707--716]. Numerical results to compare the behavior of this method with recent implementations of the conjugate gradient method are presented. These results indicate that the global Barzilai and Borwein method may allow some significant reduction in the number of line searches and also in the number of gradient evaluations.

The Barzilai and Borwein Gradient Method for the Large Scale Unconstrained Minimization Problem

In a recent paper, Barzilai and Borwein presented a new choice of steplength forthe gradient method. Their choice does not guarantee descent in the objectivefunction and greatly speeds up the convergence of the method. They presented aconvergence analysis of their method only in the two-dimensional quadratic case.We establish the convergence of the Barzilai and Borwein gradient method whenapplied to the minimization of a strictly convex quadratic function of any numberof variables.1. Introduction

/pdf/on-the-barzilai-and-borwein-choice-of-steplength-for-the-5bwvv2kryk.pdf

On the Barzilai and Borwein choice of steplength for the gradient method

A fully derivative-free spectral residual method for solving largescale nonlinear systems of equations is presented. It uses in a systematic way the residual vector as a search direction, a spectral steplength that produces a nonmonotone process and a globalization strategy that allows for this nonmonotone behavior. The global convergence analysis of the combined scheme is presented. An extensive set of numerical experiments that indicate that the new combination is competitive and frequently better than well-known Newton-Krylov methods for largescale problems is also presented.

/pdf/spectral-residual-method-without-gradient-information-for-5cgn62l864.pdf

Spectral residual method without gradient information for solving large-scale nonlinear systems of equations

Fortran 77 software implementing the SPG method is introduced. SPG is a nonmonotone projected gradient algorithm for solving large-scale convex-constrained optimization problems. It combines the classical projected gradient method with the spectral gradient choice of steplength and a nonmonotone line-search strategy. The user provides objective function and gradient values, and projections onto the feasible set. Some recent numerical tests are reported on very large location problems, indicating that SPG is substantially more efficient than existing general-purpose software on problems for which projections can be computed efficiently.

Marcos Raydan

Papers

Nonmonotone Spectral Projected Gradient Methods on Convex Sets

The Barzilai and Borwein Gradient Method for the Large Scale Unconstrained Minimization Problem

On the Barzilai and Borwein choice of steplength for the gradient method

Spectral residual method without gradient information for solving large-scale nonlinear systems of equations

Algorithm 813: SPG—Software for Convex-Constrained Optimization