Iterative Methods for Optimization

Home
/
Papers
/
Iterative Methods for Optimization

Book•

Iterative Methods for Optimization

01 Jan 1987-

TL;DR: Iterative Methods for Optimization does more than cover traditional gradient-based optimization: it is the first book to treat sampling methods, including the Hooke& Jeeves, implicit filtering, MDS, and Nelder& Mead schemes in a unified way.

read less

Abstract: This book presents a carefully selected group of methods for unconstrained and bound constrained optimization problems and analyzes them in depth both theoretically and algorithmically. It focuses on clarity in algorithmic description and analysis rather than generality, and while it provides pointers to the literature for the most general theoretical results and robust software, the author thinks it is more important that readers have a complete understanding of special cases that convey essential ideas. A companion to Kelley's book, Iterative Methods for Linear and Nonlinear Equations (SIAM, 1995), this book contains many exercises and examples and can be used as a text, a tutorial for self-study, or a reference. Iterative Methods for Optimization does more than cover traditional gradient-based optimization: it is the first book to treat sampling methods, including the Hooke& Jeeves, implicit filtering, MDS, and Nelder& Mead schemes in a unified way.

...read moreread less

Content maybe subject to copyright Report

Citations

PDF

Open Access

More filters

Journal Article•DOI•

Optimization by Direct Search: New Perspectives on Some Classical and Modern Methods ∗

[...]

Tamara G. Kolda¹, Robert Michael Lewis², Virginia Torczon²•Institutions (2)

Sandia National Laboratories¹, College of William & Mary²

01 Jan 2003-Siam Review

TL;DR: This review begins by briefly summarizing the history of direct search methods and considering the special properties of problems for which they are well suited, then turns to a broad class of methods for which the underlying principles allow general-ization to handle bound constraints and linear constraints.

...read moreread less

Abstract: Direct search methods are best known as unconstrained optimization techniques that do not explicitly use derivatives. Direct search methods were formally proposed and widely applied in the 1960s but fell out of favor with the mathematical optimization community by the early 1970s because they lacked coherent mathematical analysis. Nonetheless, users remained loyal to these methods, most of which were easy to program, some of which were reliable. In the past fifteen years, these methods have seen a revival due, in part, to the appearance of mathematical analysis, as well as to interest in parallel and distributed com- puting. This review begins by briefly summarizing the history of direct search methods and considering the special properties of problems for which they are well suited. Our focus then turns to a broad class of methods for which we provide a unifying framework that lends itself to a variety of convergence results. The underlying principles allow general- ization to handle bound constraints and linear constraints. We also discuss extensions to problems with nonlinear constraints.

...read moreread less

1,652 citations

Journal Article•DOI•

Recent approaches to global optimization problems through Particle Swarm Optimization

[...]

Konstantinos E. Parsopoulos¹, Michael N. Vrahatis¹•Institutions (1)

University of Patras¹

01 Jun 2002-Natural Computing

TL;DR: A Composite PSO, in which the heuristic parameters of PSO are controlled by a Differential Evolution algorithm during the optimization, is described, and results for many well-known and widely used test functions are given.

...read moreread less

Abstract: This paper presents an overview of our most recent results concerning the Particle Swarm Optimization (PSO) method. Techniques for the alleviation of local minima, and for detecting multiple minimizers are described. Moreover, results on the ability of the PSO in tackling Multiobjective, Minimax, Integer Programming and e1 errors-in-variables problems, as well as problems in noisy and continuously changing environments, are reported. Finally, a Composite PSO, in which the heuristic parameters of PSO are controlled by a Differential Evolution algorithm during the optimization, is described, and results for many well-known and widely used test functions are given.

...read moreread less

1,436 citations

Cites methods from "Iterative Methods for Optimization"

...Recently, Arnold in his Ph.D. thesis (Arnold, 2001) extensively tested numerous optimization methods under noise, including: (1) the direct pattern search algorithm of Hooke and Jeeves (Hooke and Jeeves, 1961), (2) the simplex metdod of Nelder and Mead (Nelder and Mead, 1965), (3) the multi-directional search algorithm of Torczon (Torczon, 1989), (4) the implicit filtering algorithm of Gilmore and Kelley (Gilmore and Kelley, 1995; Kelley, 1999) that is based on explicitly approximating the local gradient of the objective functions by means of finite differencing, (5) the simultaneous perturbation stochastic approximation algorithm due to Spall (Spall, 1992; Spall, 1998a; Spall, 1998b), (6) the evolutionary gradient search algorithm of Salomon (Salomon, 1998), (7) the evolution strategy with cumulative mutation strength adaptation mechanism by Hansen and Ostermeier (Hansen, 1998; Hansen and Ostermeier, 2001)....
[...]
...(2) the simplex metdod of Nelder and Mead (Nelder and Mead, 1965), (3) the multi-directional search algorithm of Torczon (Torczon, 1989), (4) the implicit filtering algorithm of Gilmore and Kelley (Gilmore and Kelley, 1995; Kelley, 1999) that is based on explicitly approximating the local gradient of the objective functions by means of finite differencing, (5) the simultaneous perturbation stochastic approximation algorithm due to Spall (Spall, 1992; Spall, 1998a; Spall, 1998b), (6) the evolutionary gradient search algorithm of Salomon (Salomon, 1998), (7) the evolution strategy with cumulative mutation strength adaptation mechanism by Hansen and Ostermeier (Hansen, 1998; Hansen and Ostermeier, 2001)....
[...]
...…multi-directional search algorithm of Torczon (Torczon, 1989), (4) the implicit filtering algorithm of Gilmore and Kelley (Gilmore and Kelley, 1995; Kelley, 1999) that is based on explicitly approximating the local gradient of the objective functions by means of finite differencing, (5) the…...
[...]

Journal Article•DOI•

Derivative-free optimization: a review of algorithms and comparison of software implementations

[...]

Luis Miguel Rios¹, Nikolaos V. Sahinidis¹•Institutions (1)

Carnegie Mellon University¹

01 Jul 2013-Journal of Global Optimization

TL;DR: It is found that the ability of all these solvers to obtain good solutions diminishes with increasing problem size, and TomLAB/MULTIMIN, TOMLAB/GLCCLUSTER, MCS and TOMLab/LGO are better, on average, than other derivative-free solvers in terms of solution quality within 2,500 function evaluations.

...read moreread less

Abstract: This paper addresses the solution of bound-constrained optimization problems using algorithms that require only the availability of objective function values but no derivative information. We refer to these algorithms as derivative-free algorithms. Fueled by a growing number of applications in science and engineering, the development of derivative-free optimization algorithms has long been studied, and it has found renewed interest in recent time. Along with many derivative-free algorithms, many software implementations have also appeared. The paper presents a review of derivative-free algorithms, followed by a systematic comparison of 22 related implementations using a test set of 502 problems. The test bed includes convex and nonconvex problems, smooth as well as nonsmooth problems. The algorithms were tested under the same conditions and ranked under several criteria, including their ability to find near-global solutions for nonconvex problems, improve a given starting point, and refine a near-optimal solution. A total of 112,448 problem instances were solved. We find that the ability of all these solvers to obtain good solutions diminishes with increasing problem size. For the problems used in this study, TOMLAB/MULTIMIN, TOMLAB/GLCCLUSTER, MCS and TOMLAB/LGO are better, on average, than other derivative-free solvers in terms of solution quality within 2,500 function evaluations. These global solvers outperform local solvers even for convex problems. Finally, TOMLAB/OQNLP, NEWUOA, and TOMLAB/MULTIMIN show superior performance in terms of refining a near-optimal solution.

...read moreread less

1,183 citations

Cites background from "Iterative Methods for Optimization"

...Recent works on the subject have led to significant progress by providing convergence proofs [5,9,31,34,76,80, 85,88,134], incorporating the use of surrogate models [22,24,127,131], and offering the first textbook that is exclusively devoted to this topic [35]....
[...]

Journal Article•DOI•

SBA: A software package for generic sparse bundle adjustment

[...]

Manolis I. A. Lourakis¹, Antonis A. Argyros¹•Institutions (1)

Foundation for Research & Technology – Hellas¹

16 Mar 2009-ACM Transactions on Mathematical Software

TL;DR: Sba as mentioned in this paper is a C/C++ software package for generic bundle adjustment with high efficiency and flexibility regarding parameterization, which can be used to achieve considerable computational savings when applied to bundle adjustment.

...read moreread less

Abstract: Bundle adjustment constitutes a large, nonlinear least-squares problem that is often solved as the last step of feature-based structure and motion estimation computer vision algorithms to obtain optimal estimates. Due to the very large number of parameters involved, a general purpose least-squares algorithm incurs high computational and memory storage costs when applied to bundle adjustment. Fortunately, the lack of interaction among certain subgroups of parameters results in the corresponding Jacobian being sparse, a fact that can be exploited to achieve considerable computational savings. This article presents sba, a publicly available C/C++ software package for realizing generic bundle adjustment with high efficiency and flexibility regarding parameterization.

...read moreread less

901 citations

Journal Article•DOI•

A coordinate gradient descent method for nonsmooth separable minimization

[...]

Paul Tseng¹, Sangwoon Yun¹•Institutions (1)

University of Washington¹

02 Jul 2008-Mathematical Programming

TL;DR: A (block) coordinate gradient descent method for solving this class of nonsmooth separable problems and establishes global convergence and, under a local Lipschitzian error bound assumption, linear convergence for this method.

...read moreread less

Abstract: We consider the problem of minimizing the sum of a smooth function and a separable convex function This problem includes as special cases bound-constrained optimization and smooth optimization with l1-regularization We propose a (block) coordinate gradient descent method for solving this class of nonsmooth separable problems We establish global convergence and, under a local Lipschitzian error bound assumption, linear convergence for this method The local Lipschitzian error bound holds under assumptions analogous to those for constrained smooth optimization, eg, the convex function is polyhedral and the smooth function is (nonconvex) quadratic or is the composition of a strongly convex function with a linear mapping We report numerical experience with solving the l1-regularization of unconstrained optimization problems from More et al in ACM Trans Math Softw 7, 17–41, 1981 and from the CUTEr set (Gould and Orban in ACM Trans Math Softw 29, 373–394, 2003) Comparison with L-BFGS-B and MINOS, applied to a reformulation of the l1-regularized problem as a bound-constrained optimization problem, is also reported

...read moreread less

853 citations

Cites background or methods from "Iterative Methods for Optimization"

...be judged. In the special case of bound-constrained optimization, gradient-projection methods [3,4, 24 ,32,38] or coordinate descent methods [8,23,30,33,42] can be effective....
[...]
...–i fJ = N and P is given by (2), then d is a scaled gradient-projection direction for bound-constrained minimization [4, 24 ,38,41]; –i ff isquadraticandwechoose H =∇ 2 f (x),then d isa(block)coordinatedescent direction [4,41,49,54,55]....
[...]

1
2
3
4
…
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200

Collapse

References

PDF

Open Access

More filters

Journal Article•DOI•

Optimization by Simulated Annealing

[...]

Scott Kirkpatrick¹, C. D. Gelatt¹, Mario P. Vecchi²•Institutions (2)

IBM¹, Venezuelan Institute for Scientific Research²

13 May 1983-Science

TL;DR: There is a deep and useful connection between statistical mechanics and multivariate or combinatorial optimization (finding the minimum of a given function depending on many parameters), and a detailed analogy with annealing in solids provides a framework for optimization of very large and complex systems.

...read moreread less

Abstract: There is a deep and useful connection between statistical mechanics (the behavior of systems with many degrees of freedom in thermal equilibrium at a finite temperature) and multivariate or combinatorial optimization (finding the minimum of a given function depending on many parameters). A detailed analogy with annealing in solids provides a framework for optimization of the properties of very large and complex systems. This connection to statistical mechanics exposes new information and provides an unfamiliar perspective on traditional optimization problems and methods.

...read moreread less

41,772 citations

Book•

Matrix computations

[...]

Gene H. Golub

01 Jan 1983

34,729 citations

"Iterative Methods for Optimization" refers background or methods in this paper

...32), [115], [116], [127]....
[...]
...The minimum norm solution can be expressed in terms of the singular value decomposition [127], [249] of A, A = UΣV T ....
[...]
...11) by computing the Cholesky factorization [249], [127] of H H = LL ,...
[...]
...We refer the reader to [11], [15], [127], and [154] for more discussion of preconditioners and their construction....
[...]
...The material in texts such as [127] and [264] is sufficient....
[...]

Journal Article•DOI•

An Algorithm for Least-Squares Estimation of Nonlinear Parameters

[...]

Donald W. Marquardt

01 Jun 1963-Journal of The Society for Industrial and Applied Mathematics

28,888 citations

"Iterative Methods for Optimization" refers methods in this paper

...The Levenberg–Marquardt method [172], [183] addresses these issues by adding a regularization parameter ν > 0 to R′(xc)TR′(xc) to obtain x+ = xc + s where...
[...]

Journal Article•DOI•

A simplex method for function minimization

[...]

John A. Nelder, R. Mead¹•Institutions (1)

University of Warwick¹

01 Jan 1965-The Computer Journal

TL;DR: A method is described for the minimization of a function of n variables, which depends on the comparison of function values at the (n 41) vertices of a general simplex, followed by the replacement of the vertex with the highest value by another point.

...read moreread less

Abstract: A method is described for the minimization of a function of n variables, which depends on the comparison of function values at the (n 41) vertices of a general simplex, followed by the replacement of the vertex with the highest value by another point. The simplex adapts itself to the local landscape, and contracts on to the final minimum. The method is shown to be effective and computationally compact. A procedure is given for the estimation of the Hessian matrix in the neighbourhood of the minimum, needed in statistical estimation problems.

...read moreread less

27,271 citations

"Iterative Methods for Optimization" refers background or methods in this paper

...1 Description and Implementation The Nelder–Mead [204] simplex algorithm maintains a simplex S of approximations to an optimal point....
[...]
...We will focus on three such methods: the Nelder–Mead simplex algorithm [204], the multidirectional search method [85], [261], [262], and the Hooke–Jeeves algorithm [145]....
[...]
...The ideas in this section were originally used in [155] to analyze the Nelder–Mead [204] algorithm, which we discuss in §8....
[...]
...However [204], the ability of the Nelder–Mead simplices to drastically vary their shape is an important feature of the algorithm and looking at the simplex condition alone would lead to poor results....
[...]
...Even though some of the methods, such as the Nelder–Mead [204] and Hooke–Jeeves [145] algorithms are classic, most of the convergence analysis in this part of the book was done after 1990....
[...]