How bad are the BFGS and DFP methods when the objective function is quadratic

doi:10.1007/BF01582161

Home
/
Papers
/
How bad are the BFGS and DFP methods when the objective function is quadratic

Journal Article•DOI•

How bad are the BFGS and DFP methods when the objective function is quadratic

M. J. D. Powell¹•Institutions (1)

University of Cambridge¹

01 Jan 1986-Mathematical Programming (Springer-Verlag New York, Inc.)-Vol. 34, Iss: 1, pp 34-47

TL;DR: The results help to explain why the DFP method is often less suitable than the BFGS algorithm for general unconstrained optimization calculations, and they show that quadratic functions provide much information about efficiency when the current vector of variables is too far from the solution for an asymptotic convergence analysis.

read less

Abstract: We study the use of the BFGS and DFP algorithms with step-lengths of one for minimizing quadratic functions of only two variables. The updating formulae in this case imply nonlinear three term recurrence relations between the eigenvalues of consecutive second derivative approximations, which are analysed in order to explain some gross inefficiencies that can occur. Specifically, the BFGS algorithm may require more than 10 iterations to achieve the first decimal place of accuracy, while the performance of the DFP method is far worse. The results help to explain why the DFP method is often less suitable than the BFGS algorithm for general unconstrained optimization calculations, and they show that quadratic functions provide much information about efficiency when the current vector of variables is too far from the solution for an asymptotic convergence analysis.

...read moreread less

Citations

PDF

Open Access

More filters

Journal Article•DOI•

Deformable Medical Image Registration: A Survey

[...]

Aristeidis Sotiras¹, Christos Davatzikos¹, Nikos Paragios²•Institutions (2)

University of Pennsylvania¹, École Centrale Paris²

03 Jun 2013-IEEE Transactions on Medical Imaging

TL;DR: This paper attempts to give an overview of deformable registration methods, putting emphasis on the most recent advances in the domain, and provides an extensive account of registration techniques in a systematic manner.

...read moreread less

Abstract: Deformable image registration is a fundamental task in medical image processing. Among its most important applications, one may cite: 1) multi-modality fusion, where information acquired by different imaging devices or protocols is fused to facilitate diagnosis and treatment planning; 2) longitudinal studies, where temporal structural or anatomical changes are investigated; and 3) population modeling and statistical atlases used to study normal anatomical variability. In this paper, we attempt to give an overview of deformable registration methods, putting emphasis on the most recent advances in the domain. Additional emphasis has been given to techniques applied to medical images. In order to study image registration methods in depth, their main components are identified and studied independently. The most recent techniques are presented in a systematic fashion. The contribution of this paper is to provide an extensive account of registration techniques in a systematic manner.

...read moreread less

1,434 citations

Cites methods from "How bad are the BFGS and DFP method..."

...BFGS is considered to be more efficient than DFP [410], [411]....
[...]

Book Chapter•DOI•

Finite element methods for incompressible viscous flow

[...]

Roland Glowinski¹, Roland Glowinski²•Institutions (2)

Pierre-and-Marie-Curie University¹, University of Houston²

01 Jan 2003-Handbook of Numerical Analysis

455 citations

Journal Article•DOI•

Theory of algorithms for unconstrained optimization

[...]

Jorge Nocedal¹•Institutions (1)

Northwestern University¹

01 Jan 1992-Acta Numerica

TL;DR: To make the task more tractable, I decided to consider only algorithms for unconstrained optimization, and select the best optimization methods known to date – those methods that deserve to be in a subroutine library.

...read moreread less

Abstract: A few months ago, while preparing a lecture to an audience that included engineers and numerical analysts, I asked myself the question: from the point of view of a user of nonlinear optimization routines, how interesting and practical is the body of theoretical analysis developed in this field? To make the question a bit more precise, I decided to select the best optimization methods known to date – those methods that deserve to be in a subroutine library – and for each method ask: what do we know about the behaviour of this method, as implemented in practice? To make my task more tractable, I decided to consider only algorithms for unconstrained optimization.

...read moreread less

327 citations

Cites background or methods or result from "How bad are the BFGS and DFP method..."

...A. Griewank and Ph.L. Toint (1982a), 'On the unconstrained optimization of partially separable objective functions', in Nonlinear Optimization 1981 (M.J.D. Powell, ed.)...
[...]
...Therefore it is common to restrict these studies to convex problems (Nemirovsky and Yudin, 1983), or even to strictly convex quadratic objective functions (Powell, 1986)....
[...]
...M.J.D. Powell (1971), 'On the convergence of the variable metric algorithm', J. Inst. Math....
[...]
...Since Powell's example requires that some consecutive search directions become almost contrary, and since this can only be achieved (in the case of exact line searches) when k < 0, (Powell, 1986) suggests modifying the Polak-Ribi ere method by setting k = maxf PR k ; 0g: (4:8) Thus if a negative value of PR k occurs, this strategy will restart the iteration along the steepest descent direction....
[...]
...M.J.D. Powell (1986), 'How bad are the BFGS and DFP methods when the objective function is quadratic?...
[...]

Journal Article•DOI•

Testing a class of methods for solving minimization problems with simple bounds on the variables

[...]

Andrew R. Conn, Nicholas I. M. Gould, Philippe L. Toint

01 Apr 1988-Mathematics of Computation

TL;DR: The authors describe des resultats d'une serie de tests for une classe de nouvelles methodes du type region de confiance, i.e., region of confiance.

...read moreread less

Abstract: Description des resultats d'une serie de tests pour une classe de nouvelles methodes du type region de confiance

...read moreread less

214 citations

Cites background from "How bad are the BFGS and DFP method..."

...) method(s) perform rather poorly but the rank-one formula would prove highly efficient (see Powell [24])....
[...]

Journal Article•DOI•

Convergence properties of algorithms for nonlinear optimization

[...]

M. J. D. Powell

01 Dec 1986-Siam Review

TL;DR: This paper reviews some of the most successful methods for unconstrained, constrained and nondifferentiable optimization calculations and suggests that practical considerations provide the main new ideas, and that subsequent theoretical studies give improvements to algorithms, coherence to the subject, and better understanding.

...read moreread less

Abstract: This paper reviews some of the most successful methods for unconstrained, constrained and nondifferentiable optimization calculations. Particular attention is given to the contribution that theoretical analysis has made to the development of algorithms. It seems that practical considerations provide the main new ideas, and that subsequent theoretical studies give improvements to algorithms, coherence to the subject, and better understanding.

...read moreread less

191 citations

1
2
3
4
…
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

Collapse

References

PDF

Open Access

More filters

Journal Article•DOI•

Practical Methods of Optimization.

[...]

Christoph Witzgall, Roger Fletcher

01 Oct 1989-Mathematics of Computation

9,153 citations

Book•

Practical Methods of Optimization

[...]

Roger Fletcher

01 Jan 2009

TL;DR: The aim of this book is to provide a Discussion of Constrained Optimization and its Applications to Linear Programming and Other Optimization Problems.

...read moreread less

Abstract: Preface Table of Notation Part 1: Unconstrained Optimization Introduction Structure of Methods Newton-like Methods Conjugate Direction Methods Restricted Step Methods Sums of Squares and Nonlinear Equations Part 2: Constrained Optimization Introduction Linear Programming The Theory of Constrained Optimization Quadratic Programming General Linearly Constrained Optimization Nonlinear Programming Other Optimization Problems Non-Smooth Optimization References Subject Index.

...read moreread less

7,278 citations

"How bad are the BFGS and DFP method..." refers background or result in this paper

...creates new difficulties, and many published results, like the ones in [ 2 ], show that it is often very suitable to set ak = 1 on most iterations of a BFGS algorithm....
[...]
...Similar tendencies have been noted already for more elaborate objective functions in several publications, for example see page 56 of [ 2 ], but the advantage of the function (1. l) is that it is straightforward to explain the numerical results theoretically....
[...]