scispace - formally typeset
Search or ask a question
Journal ArticleDOI

A trust region method based on interior point techniques for nonlinear programming

TL;DR: This paper focuses on the primal version of the new algorithm, an algorithm for minimizing a nonlinear function subject to nonlinear inequality constraints, which applies sequential quadratic programming techniques to a sequence of barrier problems.
Abstract: An algorithm for minimizing a nonlinear function subject to nonlinear inequality constraints is described. It applies sequential quadratic programming techniques to a sequence of barrier problems, and uses trust regions to ensure the robustness of the iteration and to allow the direct use of second order derivatives. This framework permits primal and primal-dual steps, but the paper focuses on the primal version of the new algorithm. An analysis of the convergence properties of this method is presented.

Summary (1 min read)

Algorithm Outline

  • Compute and approximate solution of the barrier problem (1:10), as follows.
  • Since the inequality constraints are already being handled as equalities, this algorithm can be easily extended to handle equality constraints.
  • The authors now digress to discuss the relationship between their approach and other interior point methods.
  • This discussion makes use of the well-known fact that Sequential Quadratic Programming, in at least one formulation, is equivalent to Newton's method applied to the optimality conditions of a nonlinear program 11].

2 Algorithm for the Barrier Problem

  • Next the authors consider conditions that determine when approximate solutions to the normal and tangential subproblems are acceptable.
  • Since these conditions require detailed justi cation, the authors consider these subproblems separately.

To d o t h i s w e i n troduce the change of variables

  • A condition of this type is necessary since, if u is unnecessarily long, the objective function value could get worse, making the job of the tangential step more di cult.
  • For the analysis in this paper it su ces to impose the following weaker condition.
  • The second condition on the normal step requires that the reduction in the objective o f (2:6) be comparable to that obtained by minimizing along the steepest descent direction in u. Normal Cauchy Decrease Condition.
  • Both conditions are also satis ed if the step is chosen by truncated conjugate gradient iterations in the variable u on the objective o f ( 2 :9) (see Steihaug 25] ), and the results are transformed back i n to the original variables.

Did you find this useful? Give us your feedback

Content maybe subject to copyright    Report

A Trust Region Metho d Based on Interior Point Techniques
for Nonlinear Programming
Richard H. Byrd
Jean Charles Gilb ert
y
Jorge No cedal
z
August 10, 1998
Abstract
An algorithm for minimizing a nonlinear function sub ject to nonlinear inequality
constraints is described. It applies sequential quadratic programming techniques to
a sequence of barrier problems, and uses trust regions to ensure the robustness of
the iteration and to allow the direct use of second order derivatives. This framework
permits primal and primal-dual steps, but the pap er fo cuses on the primal version
of the new algorithm. An analysis of the convergence properties of this metho d is
presented.
Key words:
constrained optimization, interior point metho d, large-scale optimization, non-
linear programming, primal metho d, primal-dual metho d, SQP iteration, barrier metho d,
trust region metho d.
Computer Science Department, University of Colorado, Boulder CO 80309. This author was supp orted
by NSF grant CCR-9101795, AROgrantDAAH04-94-0228, and AFOSR grant F49620-94-1-0101.
y
INRIA Rocquencourt, B.P. 105, 78153 Le Chesnay Cedex (France).
z
ECE Department, Northwestern University,Evanston Il 60208. This author was supp orted by National
Science Foundation Grants CCR-9400881 and ASC-9213149, and by DepartmentofEnergy Grant DE-
FG02-87ER25047-A004.
1

1 Intro duction
Sequential Quadratic Programming (SQP) methods have proved to be very ecient for
solving medium-size nonlinear programming problems 12, 11]. They require few iterations
and function evaluations, but since they need to solve a quadratic subproblem at every
step, the cost of their iteration is p otentially high for problems with large numb ers of vari-
ables and constraints. On the other hand, interior-point metho ds have proved to b e very
successful in solving large linear programming problems, and it is natural to ask whether
they can be extended to nonlinear problems. Preliminary computational experience with
simple adaptations of primal-dual interior point metho ds have given encouraging results
on some classes on nonlinear problems (see for example 27 , 14 , 29, 1]).
In this pap er we describ e and analyze an algorithm for large-scale nonlinear program-
ming that uses ideas from interior point metho ds
and
sequential quadratic programming.
One of its unique features is the use of a trust region framework that allows for the direct
use of second derivatives and the inaccurate solution of subproblems. The algorithm is
well suited for handling equality constraints (see 4]), but for simplicityofexp osition we
will only consider here inequality constrained problems of the form
min
x
f
(
x
)
sub ject to
g
(
x
)
0
(1.1)
where
f
:
R
n
!
R
and
g
:
R
n
!
R
m
are smooth functions.
Following the strategy of interior p oint metho ds (see for example 13 , 28, 19]) we
asso ciate with (1.1) the following barrier problem in the variables
x
and
s
min
xs
f
(
x
)
;
m
X
i
=1
ln
s
(
i
)
sub ject to
g
(
x
)+
s
=0
(1.2)
where
>
0 and where the vector of slack variables
s
= (
s
(1)
:::s
(
m
)
)
>
is implicitly
assumed to b e p ositive.
The main goal of this pap er is to prop ose and analyze an algorithm for nding an
approximate solution to (1.2), for xed
, that can eectively enforce the p ositivitycon-
dition
s>
0 on the slack variables without incurring in a high cost. This algorithm can
be applied repeatedly to problem (1.2), for decreasing values of
, to approximate the
solution of the original problem (1.1). The key to our approach is to view interior point
metho ds from the persp ective of sequential quadratic programming and formulate the
quadratic subproblem so that the steps are discouraged from violating the b ounds
s>
0.
This framework suggests how to generate steps with primal or primal-dual characteristics,
and is well suited for large problems. Numerical exp eriments with an implementation of
the new method have been performed by Byrd, Hribar and No cedal 4], and show that
this approach holds much promise. We should note that in this pap er we do not address
the imp ortant issue of how fast to decrease the barrier parameter, which is currently an
active area of research.
1

We begin byintro ducing some notation and by stating the rst-order optimalitycon-
ditions for the barrier problem. The Lagrangian of (1.2) is
L
(
x s
)=
f
(
x
)
;
m
X
i
=1
ln
s
(
i
)
+
>
(
g
(
x
)+
s
)
(1.3)
where
2
R
m
are the Lagrange multipliers. At an optimal solution (
x s
) of (1.2) wehave
r
x
L
(
x s
)=
r
f
(
x
)+
A
(
x
)
=0 (1.4)
r
s
L
(
x s
)=
;
S
;
1
e
+
=0
(1.5)
where
A
(
x
)=
r
g
(1)
(
x
)
:::
r
g
(
m
)
(
x
)
(1.6)
is the matrix of constraint gradients, and where
e
=
0
B
@
1
.
.
.
1
1
C
A
S
=
0
B
B
@
s
(1)
.
.
.
s
(
m
)
1
C
C
A
:
(1.7)
To facilitate the derivation of the new algorithm wedene
z
=
x
s
!
'
(
z
)=
f
(
x
)
;
m
X
i
=1
ln
s
(
i
)
(1.8)
c
(
z
)=
g
(
x
)+
s
(1.9)
and rewrite the barrier problem (1.2) as
min
z
'
(
z
)
sub ject to
c
(
z
)=0
:
(1.10)
Wenow apply the sequential quadratic programming metho d (see for example 12, 11]) to
this problem. At an iterate
z
,we generate a displacement
d
=
d
x
d
s
!
by solving the quadratic program
min
d
r
'
(
z
)
>
d
+
1
2
d
>
Wd
sub ject to
^
A
(
z
)
>
d
+
c
(
z
)=0
(1.11)
where
W
is the Hessian of the Lagrangian of the barrier problem (1.10) with resp ect to
z
,
and where
^
A
>
is the Jacobian of
c
and is given by
^
A
(
z
)
>
=
A
(
x
)
>
I
:
(1.12)
2

Note that (1.10) is just a restatement of (1.2), and thus from (1.4){(1.5) wehavethat
W
r
2
zz
L
(
x s
)=
r
2
xx
L
(
x s
) 0
0
S
;
2
!
:
(1.13)
To obtain convergence from remote starting p oints, and to allow for the case when
W
is not p ositive denite in the null space of
^
A
>
, we introduce a trust region constraint in
(1.11) of the form
d
x
S
;
1
d
s
!
(1.14)
where the trust region radius
>
0 is up dated at every iteration. The step in the
slack variables is scaled by
S
;
1
due to the form
S
;
2
of the p ortion of the Hessian
W
corresp onding to the slackvariables. Since this submatrix is p ositive denite and diagonal,
it seems to b e the b est scale at the current p oint see also 4] for a discussion of howthis
scaling is b enecial when using a conjugate gradient iteration to compute the step.
From nowon we simplify the notation by writing a vector suchas
z
, whichhas
x
and
s
-comp onents, as
z
=(
x s
) instead of
z
=(
x
>
s
>
)
>
. In this way an expression like that
in (1.14) is simply written as
d
x
S
;
1
d
s
!
(
d
x
S
;
1
d
s
)
:
(1.15)
The trust region constraint(1
:
14) do es not prevent the new slackvariable values
s
+
d
s
from becoming negative unless is suciently small. Since it is not desirable to imp ede
progress of the iteration by employing small trust regions, we explicitly b ound the slack
variables away from zero by imp osing the well-known fraction to the boundary rule 28]
s
+
d
s
(1
;
)
s
where the parameter
2
(0
1) is chosen close to 1. This results in the subproblem
min
d
r
'
(
z
)
>
d
+
1
2
d
>
Wd
sub ject to
^
A
(
z
)
>
d
+
c
(
z
)=0
(
d
x
S
;
1
d
s
)
d
s
;
s:
(1.16)
We will assume for simplicity that the trust region is dened using the Euclidean norm,
although our analysis would be essentially the same for any other xed norm. It is true
that problem (1
:
16) could b e quite dicult to solve exactly,butweintend to only compute
approximate solutions using techniques such as a dogleg metho d or the conjugate gradient
algorithm. Due to the formulation of our subproblem these techniques will tend to avoid
the boundaries of the constraints
s >
0 and will lo cate an approximate solution with
mo derate cost. To see that our subproblem (1
:
16) is appropriate, note that if the slack
variables are scaled by
S
;
1
, the feasible region of the transformed problem has the essential
3

characteristics of a trust region: it is b ounded and contains a ball centered at
z
whose
radius is bounded belowbyavalue that depends on and not on
z
.
It is well known 26 ] that the constraints in (1
:
16) can b e incompatible since the steps
d
satisfying the linear constraints may not lie within the trust region. Several strategies
have been prop osed to make the constraints consistent 7, 6, 24], and in this paper we
follow the approach of Byrd 3] and Omo jokun 20], which we have found suitable for
solving large problems 18 ].
The strategy of Byrd and Omo jokun consists of rst taking a normal (or transver-
sal) step
v
that lies well inside the trust region and that attempts to satisfy the linear
constraints in (1.16) as well as p ossible. To compute the normal step
v
, we choose a
contraction parameter 0
< <
1 (say
= 0
:
8) that determines a tighter version of the
constraints (1
:
16), i.e., a smaller trust region radius
and tighter lower b ounds
;

.
Then we approximately solve the problem
min
v
k
^
A
(
z
)
>
v
+
c
(
z
)
k
sub ject to
(
v
x
S
;
1
v
s
)
v
s
;
 s
(1.17)
where here, and for the rest of the pap er,
kk
denotes the Euclidean (or
`
2
) norm. The
normal step
v
determines how well the linear constraints in (1.16) will be satised. We
now compute the total step
d
by approximately solving the following mo dication of (1.16)
min
d
r
'
(
z
)
>
d
+
1
2
d
>
Wd
sub ject to
^
A
(
z
)
>
d
=
^
A
(
z
)
>
v
(
d
x
S
;
1
d
s
)
d
s
;
s:
(1.18)
The constraints for this subproblem are always consistent for example
d
=
v
is feasible.
Lalee, No cedal and Plantenga 18] describ e direct and iterative metho ds for approximately
solving (1
:
18) when the number of variables is large.
Wenow need to decide if the trial step
d
obtained from (1.18) should b e accepted, and
for this purp ose weintro duce a merit function for the barrier problem (1.10). (Recall that
our ob jective at this stage is to solve the barrier problem for a xed value of the barrier
parameter
.) We follow Byrd and Omo jokun and dene the merit function to b e
(
z
)=
'
(
z
)+
k
c
(
z
)
k
(1.19)
where
>
0 is a
penalty parameter
. Since the Euclidean norm in the second term is not
squared, this merit function is non-dierentiable. It is also exact in the sense that if
is
greater than a certain threshold value, then a Karush-Kuhn-Tucker point of the barrier
problem (1.2) is a stationary point of the merit function
. The step
d
is accepted if it
gives sucient reduction in the merit function otherwise it is rejected.
We complete the iteration by up dating the trust region radius according to standard
trust region techniques that will b e discussed later on.
We summarize the discussion given so far by presenting a broad outline of the new
algorithm for solving the nonlinear programming problem (1.1).
4

Citations
More filters
Book
01 Nov 2008
TL;DR: Numerical Optimization presents a comprehensive and up-to-date description of the most effective methods in continuous optimization, responding to the growing interest in optimization in engineering, science, and business by focusing on the methods that are best suited to practical problems.
Abstract: Numerical Optimization presents a comprehensive and up-to-date description of the most effective methods in continuous optimization. It responds to the growing interest in optimization in engineering, science, and business by focusing on the methods that are best suited to practical problems. For this new edition the book has been thoroughly updated throughout. There are new chapters on nonlinear interior methods and derivative-free methods for optimization, both of which are used widely in practice and the focus of much current research. Because of the emphasis on practical methods, as well as the extensive illustrations and exercises, the book is accessible to a wide audience. It can be used as a graduate text in engineering, operations research, mathematics, computer science, and business. It also serves as a handbook for researchers and practitioners in the field. The authors have strived to produce a text that is pleasant to read, informative, and rigorous - one that reveals both the beautiful nature of the discipline and its practical side.

17,420 citations

Journal ArticleDOI
TL;DR: A comprehensive description of the primal-dual interior-point algorithm with a filter line-search method for nonlinear programming is provided, including the feasibility restoration phase for the filter method, second-order corrections, and inertia correction of the KKT matrix.
Abstract: We present a primal-dual interior-point algorithm with a filter line-search method for nonlinear programming. Local and global convergence properties of this method were analyzed in previous work. Here we provide a comprehensive description of the algorithm, including the feasibility restoration phase for the filter method, second-order corrections, and inertia correction of the KKT matrix. Heuristics are also considered that allow faster performance. This method has been implemented in the IPOPT code, which we demonstrate in a detailed numerical study based on 954 problems from the CUTEr test set. An evaluation is made of several line-search options, and a comparison is provided with two state-of-the-art interior-point codes for nonlinear programming.

7,966 citations


Cites methods from "A trust region method based on inte..."

  • ...As a barrier method, like the methods discussed in [2, 8, 11, 29], the proposed algorithm computes (approximate) solutions for a sequence of barrier problems...

    [...]

  • ...To allow convergence from poor starting points, interior-point methods, in both trust region and line-search frameworks, have been developed that use exact penalty merit functions to enforce progress toward the solution [2, 21, 29]....

    [...]

  • ...4 The problems with altered problem size are CATENARY, CHARDIS1, CONT5-QP, CONT6-QQ, CVXQP1, CVXQP2, CVXQP3, DRCAV1LQ, DRCAV2LQ, DRCAV3LQ, DRCAVTY3, EG3, EIGENA, EIGENALS, EIGENB, EIGENB2, EIGENBCO, EIGENBLS, EIGENC, EIGENC2, EIGENCCO, EIGENCLS, FLOSP2HH, FLOSP2HL, FLOSP2HM, FMINSURF, GAUSSELM, HARKERP2, LUBRIF, LUBRIFC, NCVXQP[1-9], NONCVXU2, NONMSQRT, POWER, SCURLY30, SPARSINE, SPARSQUR....

    [...]

  • ...Furthermore, IPOPT aborted in 3 problems because it reverted to the restoration phase when the constraint violation was already below the termina- 4 The problems with altered problem size are CATENARY, CHARDIS1, CONT5-QP, CONT6-QQ, CVXQP1, CVXQP2, CVXQP3, DRCAV1LQ, DRCAV2LQ, DRCAV3LQ, DRCAVTY3, EG3, EIGENA, EIGENALS, EIGENB, EIGENB2, EIGENBCO, EIGENBLS, EIGENC, EIGENC2, EIGENCCO, EIGENCLS, FLOSP2HH, FLOSP2HL, FLOSP2HM, FMINSURF, GAUSSELM, HARKERP2, LUBRIF, LUBRIFC, NCVXQP[1-9], NONCVXU2, NONMSQRT, POWER, SCURLY30, SPARSINE, SPARSQUR....

    [...]

Journal ArticleDOI
TL;DR: An SQP algorithm that uses a smooth augmented Lagrangian merit function and makes explicit provision for infeasibility in the original problem and the QP subproblems is discussed.
Abstract: Sequential quadratic programming (SQP) methods have proved highly effective for solving constrained optimization problems with smooth nonlinear functions in the objective and constraints. Here we consider problems with general inequality constraints (linear and nonlinear). We assume that first derivatives are available and that the constraint gradients are sparse. We discuss an SQP algorithm that uses a smooth augmented Lagrangian merit function and makes explicit provision for infeasibility in the original problem and the QP subproblems. SNOPT is a particular implementation that makes use of a semidefinite QP solver. It is based on a limited-memory quasi-Newton approximation to the Hessian of the Lagrangian and uses a reduced-Hessian algorithm (SQOPT) for solving the QP subproblems. It is designed for problems with many thousands of constraints and variables but a moderate number of degrees of freedom (say, up to 2000). An important application is to trajectory optimization in the aerospace industry. Numerical results are given for most problems in the CUTE and COPS test collections (about 900 examples).

2,831 citations


Cites background from "A trust region method based on inte..."

  • ...LOQO [78] and KNITRO [17, 16] are examples of large-scale optimization packages that treat inequality constraints by a primal-dual interior method....

    [...]

  • ...LOQO [100], KNITRO [21, 20], and IPOPT [101] are examples of large-scale optimization packages that treat inequality constraints by a primal-dual interior method....

    [...]

Journal ArticleDOI
TL;DR: The design and implementation of a new algorithm for solving large nonlinear programming problems follows a barrier approach that employs sequential quadratic programming and trust regions to solve the subproblems occurring in the iteration.
Abstract: The design and implementation of a new algorithm for solving large nonlinear programming problems is described. It follows a barrier approach that employs sequential quadratic programming and trust regions to solve the subproblems occurring in the iteration. Both primal and primal-dual versions of the algorithm are developed, and their performance is illustrated in a set of numerical tests.

1,605 citations


Cites background or methods from "A trust region method based on inte..."

  • ...…author was supported by ARO grant DAAH and AFOSR grant F yCAAM Department Rice University Houston TX This author was supported by Department of Energy grant DE FG ER A zECE Department Northwestern University Evanston Il This author was supported by National Science Foundation grant CCR and by…...

    [...]

  • ...…gradients Of crucial importance in the new algorithm is the formulation and solution of the equal ity constrained quadratic subproblems that determine the steps of the algorithm The formulation of the subproblems gives the iteration primal or primal dual characteristics and ensures that the…...

    [...]

  • ...…step that lies on the tangent space of the constraints and that tries to achieve optimality The e ciency of the new algorithm depends to a great extent on how these two components of the step are computed Throughout this section we omit the iteration subscript and write sk as s Ah xk as Ah etc...

    [...]

  • ...The design and implementation of a new algorithm for solving large nonlinear pro gramming problems is described It follows a barrier approach that employs sequential quadratic programming and trust regions to solve the subproblems occurring in the iteration Both primal and primal dual versions of the algorithm are developed and their performance is illustrated in a set of numerical tests Key words constrained optimization interior point method large scale optimization non linear programming primal method primal dual method successive quadratic program ming trust region method Computer Science Department University of Colorado Boulder CO This author was supported by ARO grant DAAH and AFOSR grant F yCAAM Department Rice University Houston TX This author was supported by Department of Energy grant DE FG ER A zECE Department Northwestern University Evanston Il This author was supported by National Science Foundation grant CCR and by Department of Energy grant DE FG ER A Introduction In this paper we discuss the design implementation and performance of an interior point method for solving the nonlinearly constrained optimization problem min f x subject to h x g x where f Rn R h Rn Rt and g Rn Rm are smooth functions We are particularly interested in the case when is not a convex program and when the number of variables n is large We assume in this paper that rst and second derivatives of the objective function and constraints are available but our strategy can be extended so as to make use of quasi Newton approximations Interior point methods provide an alternative to active set methods for the treatment of inequality constraints Our algorithm which is based on the framework proposed by Byrd Gilbert and Nocedal incorporates within the interior point method two powerful tools for solving nonlinear problems sequential quadratic programming and trust region techniques Sequential quadratic programming SQP ideas are used to e ciently handle nonlinearities in the constraints Trust region strategies allow the algorithm to treat convex and non convex problems uniformly permit the direct use of second derivative information and provide a safeguard in the presence of nearly dependent constraint gradients Of crucial importance in the new algorithm is the formulation and solution of the equal ity constrained quadratic subproblems that determine the steps of the algorithm The formulation of the subproblems gives the iteration primal or primal dual characteristics and ensures that the slack variables remain safely positive The technique used to solve the subproblems has a great impact on the e ciency and robustness of the algorithm we use an adaptation of the trust region method of Byrd and Omojokun which has proved to be e ective for solving large equality constrained problems Our numerical results suggest that the new algorithm holds much promise it appears to be robust and e cient in terms of function evaluations and can make e ective use of second derivative information The test results also indicate that the primal dual version of the algorithm is superior to the primal version The new algorithm has a solid theo retical foundation since it follows the principles of the globally convergent primal method developed in There has been much research in using interior point methods for nonlinear program ming most of it concerns line search methods The special case when the problem is a convex program can be handled by line search methods that are in a sense direct exten sions of interior point methods for linear programming see e g In the convex case the step generated by the solution of the primal dual equations can be shown to be a descent direction for several merit functions and this allows one to establish fairly satisfactory con vergence results Other research has focused on the local behavior of interior point line search methods for nonlinear programming Conditions have been given that guarantee superlinear and quadratic rates of convergence These algorithms can also be viewed as a direct extension of linear programming methods in that they do not make provisions for the case when the problems is non convex Several line search algorithms designed for non convex problems have recently been proposed An important feature of many of these methods is a strategy for modifying the KKT system used in the computation of the search direction This modi cation which is usually based on a matrix factorization algorithm ensures that the search direction is a descent direction for the merit function Since these algorithms are quite recent it is di cult to assess at this point whether they will lead to robust general purpose codes The use of trust region strategies in interior point methods for linear and nonlinear problems is not new Coleman and Li proposed a primal method for bound constrained nonlinear optimization see also Plantenga developed an algorithm for general nonlinear programming that has some features in common with our algorithm the main di erences lie in his treatment of the trust region in the purely primal nature of this step and in the fact that his algorithm reverts to an active set method near the solution The algorithm proposed in this paper makes use of successive quadratic programming techniques and in this sense is related to the line search algorithm of Yamashita But the way in which our algorithm combines trust region strategies interior point approaches and successive quadratic programming techniques leads to an iteration that is di erent from those proposed in the literature The New Algorithm The algorithm is essentially a barrier method in which the subproblems are solved approximately by an SQP iteration with trust regions Each barrier subproblem is of the form min x s f x mX i ln si subject to h x g x s where is the barrier parameter and where the slack variable s is assumed to be positive By letting converge to zero the sequence of approximate solutions to will normally converge to a minimizer of the original nonlinear program As in some interior point methods for linear programming and in contrast with the barrier methods of Fiacco and McCormick our algorithm does not require feasibility of the iterates with respect to the inequality constraints but only forces the slack variables to remain positive To characterize the solution of the barrier problem we introduce its Lagrangian L x s h g f x mX i ln si T hh x T g g x s where h and g are the multipliers associated with the equality and inequality constraints respectively Rather than solving each barrier subproblem accurately we will be content with an approximate solution x s satisfying E x s where E measures the optimality conditions of the barrier problem and is de ned by E x s max krf x Ah x h Ag x gk kS g ek kh x k kg x sk Here e T S diag s sm with superscripts indicating components of a vector and Ah x rh x rht x Ag x rg x rgm x are the matrices of constraint gradients In the de nition of the optimality measure E the vectors h g are least squares multiplier estimates and thus are functions of x s and We will show later see that the terms in correspond to each of the equations of the so called perturbed KKT system upon which our primal dual algorithm is based The tolerance which determines the accuracy in the solution of the barrier problems is decreased from one barrier problem to the next and must converge to zero In this paper we will use the simple strategy of reducing both and by a constant factor We test for optimality for the nonlinear program by means of E x s Algorithm I Barrier Algorithm for Solving the Nonlinear Problem Choose an initial value for the barrier parameter and select the parame ters and the nal stop tolerance TOL Choose the starting point x and s and evaluate the objective function constraints and their derivatives at x Repeat until E x s TOL Apply an SQP method with trust regions starting from x s to nd an approximate solution x s of the barrier problem satisfying E x s Set x x s s end To obtain a rapidly convergent algorithm it is necessary to carefully control the rate at which the barrier parameter and the convergence tolerance are decreased We will however not consider this question here and defer its study in the context of our algorithm to a future article Most of the work of Algorithm I lies clearly in step in the approximate solution of an equality constrained problem with an implicit lower bound on the slack variables The challenge is to perform this step e ciently even when is small while forcing the slack variables to remain positive To do this we apply an adaptation of the equality constrained SQP iteration with trust regions proposed by Byrd and Omojokun and developed by Lalee Nocedal and Plantenga for large scale equality constrained optimization We follow an SQP approach because it is known to be e ective for solving equality constrained problems even when the problem is ill conditioned and the constraints are highly nonlinear and choose to use trust region strategies to globalize the SQP iteration because they facilitate the use of second derivative information when the problem is non convex However a straightforward application of this SQP method to the barrier problem leads to ine cient primal steps that tend to violate the positivity of the slack variables and that are thus frequently cut short by the trust region constraint The novelty of our approach lies in the formulation of the quadratic model in the SQP iteration and in the de nition of the scaled trust region These are designed so as to produce steps that have some of the properties of primal dual iterations and that avoid approaching the boundary of the feasible region too soon In order to describe our approach more precisely it is instructive to brie y review the basic principles of Sequential Quadratic Programming Every iteration of an SQP method with trust regions begins by constructing a quadratic model of the Lagrangian function A step d of the algorithm is computed by minimizing the quadratic model subject to satisfying a linear approximation to the constraints and subject to a trust region bound on this step If the step d gives a su cient reduction in the merit function then it is accepted otherwise the step is rejected the trust region is reduced and a new step is computed Let us apply these ideas to the barrier problem in order to compute a step d dx ds from the current iterate xk sk To economize space we will often write vectors with x and s components as dx ds dx ds After computing Lagrange multiplier estimates h g we formulate the quadratic sub problem min dx ds rf xk Tdx dTxr xxL xk sk h g dx eTS k ds dTs kds subject to Ah xk Tdx h xk rh Ag xk Tdx ds g xk sk rg dx ds Tk Here k is an m m positive de nite diagonal matrix that represents either the Hessian of the Lagrangian with respect to s or an approximation to it As we will see in the next section the choice of k is of crucial importance because it determines whether the iteration has primal or primal dual characteristics The residual vector r rh rg in which is in essence chosen to be the vector of minimum Euclidean norm such that are consistent will be de ned in the next section The closed and bounded set Tk de nes the region around xk where the quadratic model and the linearized constraints can be trusted to be good approximations to the problem and also ensures the feasibility of the slack variables This trust region also guarantees that has a nite solution even when r xxL xk sk h g is not positive de nite The precise form of the trust region Tk requires careful consideration and will be described in the next section We compute a step d dx ds by approximately minimizing the quadratic model subject to the constraints as will be described in x We then determine if the step is acceptable according to the reduction obtained in the following merit function x s f x mX i ln si h x g x s where is a penalty parameter This non di erentiable merit function has been success fully used in the SQP algorithm of Byrd and Omojokun and has been analyzed in the context of interior point methods in We summarize this SQP trust region approach as follows Algorithm II SQP Trust Region Algorithm for the Barrier Problem Input parameters and and values k xk and sk set trust region Tk compute Lagrange multipliers h and g Repeat until E xk sk Compute d dx ds by approximately solving If the step d provides su cient decrease in then set xk xk dx sk sk ds compute new Lagrange multiplier estimates h and g and possibly enlarge the trust region else set xk xk sk sk and shrink the trust region Set k k end Algorithm II is called at each execution of step of Algorithm I The iterates of Algo rithm II are indexed by xk sk where the index k runs continuously during Algorithm I In the next section we present a full description of Algorithm II which forms the core of the new interior point algorithm Numerical results are then reported in x Algorithm for Solving the Barrier Problem Many details of the SQP trust region method outlined in Algorithm II need to be developed We rst give a precise description of the quadratic subproblem including the choice of the diagonal matrix k which gives rise to primal or primal dual iterations Further we de ne the right hand side vectors rh rg the form of the trust region constraint Tk and the choice of Lagrange multiplier estimates Once a complete description of the subproblem has been given we will present our procedure for nding an approximate solution of it We will conclude this section with a discussion of various other details of implementation of the new algorithm Formulation of the Subproblem Let us begin by considering the quadratic model We have mentioned that SQP methods choose the Hessian of this model to be the Hessian of the Lagrangian of the problem under consideration or an approximation to it Since the problem being solved by Algorithm II is the barrier problem which has a separable objective function in the variables x and s its Hessian consists of two blocks As indicated in we choose the Hessian of the quadratic model with respect to dx to be r xxL xk sk h g which we abbreviate as r xxLk but consider several choices for the Hessian k of the model with respect to ds The rst choice is to de ne k r ssLk which gives k S k The general algorithm studied in Byrd Gilbert and Nocedal de nes k in this manner To study the e ect of k in the step computation let us analyze the simple case when the matrix r xxLk is positive de nite on the null space of the constraints when the residual rh rg is zero and when the step generated by lies strictly inside the trust region In this case the quadratic subproblem has a unique solution d dx ds which satis es the linear system r xxLk Ah xk Ag xk k I ATh xk ATg xk I dx ds h g rf xk S k e h xk g xk sk It is well known see e g and easy to verify that if k is de ned by the system is equivalent to a Newton iteration on the KKT conditions of the barrier problem which are given by rf x Ah x h Ag x g S e g h x g x s This approach is usually referred to as a primal method Several authors including Jarre and S Wright M Wright and Conn Gould and Toint have given arguments suggesting that the primal search direction will often cause the slack variables to become negative and can be ine cient Research in linear programming has shown that a more e ective interior point method is obtained by considering the perturbed KKT system rf x Ah x h Ag x g S g e h x g x s which is obtained by multiplying by S It is well known and also easy to verify that a Newton step on this system is given by the solution to with k S k g Here g diag g m g contains the Lagrange multiplier estimates corresponding to the inequality constraints The system with k de ned by is called the primal dual system This choice of k may be viewed as an approximation to r ssLk since by at the solution of the barrier problem the equation S g is satis ed Substituting this equation in gives The system has the advantage that the second derivatives of are bounded as any slack variables approach zero which is not the case with In fact analysis of the primal dual step as well as computational experience with linear programs has shown that it overcomes the drawbacks of the primal step it does not tend to violate the constraints on the slacks and usually makes excellent progress towards the solution see e g These observations suggest that the primal dual model in which k is given by is likely to perform better than the primal choice Of course these ar guments do not apply directly to our algorithm which solves the SQP subproblem inexactly and whose trust region constraint may be active Nevertheless as the iterates approach a solution point the algorithm will resemble more and more an interior point method in which a Newton step on some form of the KKT conditions of the barrier problem is taken at each step Lagrange multiplier estimates are needed both in the primal dual choice of k and in the Hessian r Lxx xk sk h g To complete our description of the quadratic model we must discuss how these multipliers are computed...

    [...]

Book ChapterDOI
01 Jan 2006
TL;DR: The package provides crossover techniques between algorithmic options as well as automatic selection of options and settings, and it is effective for the following special cases: unconstrained optimization, nonlinear systems of equations, least squares, and linear and quadratic programming.
Abstract: This paper describes Knitro 5.0, a C-package for nonlinear optimization that combines complementary approaches to nonlinear optimization to achieve robust performance over a wide range of application requirements. The package is designed for solving large-scale, smooth nonlinear programming problems, and it is also effective for the following special cases: unconstrained optimization, nonlinear systems of equations, least squares, and linear and quadratic programming. Various algorithmic options are available, including two interior methods and an active-set method. The package provides crossover techniques between algorithmic options as well as automatic selection of options and settings.

1,022 citations


Cites methods from "A trust region method based on inte..."

  • ...In the interior point algorithms, our heuristics are based on the theory developed in [3]....

    [...]

  • ...Interior/CG is an implementation of the algorithm described in [6], which is based on the approach described and analyzed in [3]....

    [...]

References
More filters
Journal ArticleDOI
TL;DR: In this paper, a trust region approach for minimizing nonlinear functions subject to simple bounds is proposed, where the trust region is defined by minimizing a quadratic function subject only to an ellipsoidal constraint and the iterates generated by these methods are always strictly feasible.
Abstract: We propose a new trust region approach for minimizing nonlinear functions subject to simple bounds. By choosing an appropriate quadratic model and scaling matrix at each iteration, we show that it is not necessary to solve a quadratic programming subproblem, with linear inequalities, to obtain an improved step using the trust region idea. Instead, a solution to a trust region subproblem is defined by minimizing a quadratic function subject only to an ellipsoidal constraint. The iterates generated by these methods are always strictly feasible. Our proposed methods reduce to a standard trust region approach for the unconstrained problem when there are no upper or lower bounds on the variables. Global and quadratic convergence of the methods is established; preliminary numerical experiments are reported.

3,026 citations

Journal ArticleDOI
TL;DR: The design and implementation of a new algorithm for solving large nonlinear programming problems follows a barrier approach that employs sequential quadratic programming and trust regions to solve the subproblems occurring in the iteration.
Abstract: The design and implementation of a new algorithm for solving large nonlinear programming problems is described. It follows a barrier approach that employs sequential quadratic programming and trust regions to solve the subproblems occurring in the iteration. Both primal and primal-dual versions of the algorithm are developed, and their performance is illustrated in a set of numerical tests.

1,605 citations

Journal ArticleDOI
TL;DR: An implementation of an interior point method to the optimal reactive dispatch problem is described in this article, which is based on the primal-dual algorithm and the numerical results in large scale networks (1832 and 3467 bus systems) have shown that this technique can be very effective to some optimal power flow applications.
Abstract: An implementation of an interior point method to the optimal reactive dispatch problem is described. The interior point method used is based on the primal-dual algorithm and the numerical results in large scale networks (1832 and 3467 bus systems) have shown that this technique can be very effective to some optimal power flow applications. >

842 citations

Journal ArticleDOI
TL;DR: It is shown in this paper that an approximate solution of the trust region problem may be found by the preconditioned conjugate gradient method, and it is shown that the method has the same convergence properties as existing methods based on the dogleg strategy using an approximate Hessian.
Abstract: Algorithms based on trust regions have been shown to be robust methods for unconstrained optimization problems. All existing methods, either based on the dogleg strategy or Hebden-More iterations, require solution of system of linear equations. In large scale optimization this may be prohibitively expensive. It is shown in this paper that an approximate solution of the trust region problem may be found by the preconditioned conjugate gradient method. This may be regarded as a generalized dogleg technique where we asymptotically take the inexact quasi-Newton step. We also show that we have the same convergence properties as existing methods based on the dogleg strategy using an approximate Hessian.

829 citations


"A trust region method based on inte..." refers background in this paper

  • ...It is also satisfied if the step is chosen by truncated conjugate gradient iterations in the variable p on the objective of (2.28) (see Steihaug [25])....

    [...]

  • ...9) (see Steihaug [25]), and the results are transformed back into the original variables....

    [...]

  • ...Both conditions are also satisfied if the step is computed by truncated conjugate gradient iterations in the variable u on the objective of (2.9) (see Steihaug [25]), and the results are transformed back into the original variables....

    [...]

Journal ArticleDOI
TL;DR: In this paper, a new algorithm using the primal-dual interior point method with the predictor-corrector for solving nonlinear optimal power flow (OPF) problems is presented.
Abstract: A new algorithm using the primal-dual interior point method with the predictor-corrector for solving nonlinear optimal power flow (OPF) problems is presented. The formulation and the solution technique are new. Both equalities and inequalities in the OPF are considered and simultaneously solved in a nonlinear manner based on the Karush-Kuhn-Tucker (KKT) conditions. The major computational effort of the algorithm is solving a symmetrical system of equations, whose sparsity structure is fixed. Therefore only one optimal ordering and one symbolic factorization are involved. Numerical results of several test systems ranging in size from 9 to 2423 buses are presented and comparisons are made with the pure primal-dual interior point algorithm. The results show that the predictor-corrector primal-dual interior point algorithm for OPF is computationally more attractive than the pure primal-dual interior point algorithm in terms of speed and iteration count. >

422 citations

Frequently Asked Questions (11)
Q1. What contributions have the authors mentioned in the paper "A trust region method based on interior point techniques for nonlinear programming" ?

This framework permits primal and primal dual steps but the paper focuses on the primal version of the new algorithm An analysis of the convergence properties of this method is presented 

The KKT conditions for the equality constrained barrier problem give rise to the following system of nonlinear equations in x s seeB rf x A x S eg x sCAApplying Newton s method to this system the authors obtain the iteration B rxxL A x S IA x ICA B dxds CA B rf x S e g x s CAwhere d and where the authors have omitted the argument of r xxL x s for brevity Note that the current values of the multipliers only enter in through r xxL 

The steepest descent direction for the objective function of at p is given bypck Z x rfk Bkvx Z s S k e S k vsThe authors are now ready to state the condition the authors impose on the tangential stepTangential Cauchy Decrease Condition 

This is because near the solution point the quadratic subproblem will be convex and the tolerances of the procedure for solving subject to the trust region constraint will be set so that asymptotically it is solved exactly Moreover as the iterates converge to the solution the authors expect the trust region constraint to become inactive provided a second order correction is incorporated in the algorithm 

Sk us lies in the range of A k S k and gives a value of zero for the objective of By the above argument if kgk skk is su ciently small v is feasible for problem and is therefore a solution to Since v is a solution to lying in the range of A k S k the range space condition implies that the normal step vk must also lie in the range of A k Sk This implies that since vpredk vk vk satis es so that vx S k vs kgk skkNow recall that by andvpredk v minkgk skkwhich together with impliesThe authors should note that if the Lagrange multipliers k are de ned as the least squares solution torfk Ak Sk ethen the boundedness of frfkg fAkg fskg and imply that the sequence f kg is bounded 

The authors will divide the analysis in two cases depending on whether the matrices A k Sk lose rank or not The authors use the notation min M to denote the smallest singular value of a matrix M and recall that in De nitions the authors describe their notion of linear independence constraint quali cationLemma Suppose that the sequences fgkg and fAkg are bounded that ffkg is bounded below and that gk sk Then either there is some bound such thatmin A k Skfor all k or the sequence f gk Ak g has a limit point g A failing the linear independence constraint quali cation 

Among these are the fact that his trust region does not include a scaling that his iteration produces a ne scaling steps near the solution and that his approach reverts to an active set method when progress is slowThe authors emphasize that the equivalence between SQP and Newton s method applied to the KKT system holds only if the subproblem is strictly convex if this subproblem is solved exactly and if the trust region constraint is inactive 

All limit points x of fxklg are feasible Furthermore if any limit point x of fxklg satis es the linear independence constraint quali cation then the rst order optimality conditions of the problemmin x f xs t g xhold at x there exists 

Writing d vk h where vk is the approximate solution of the normal subproblem the authors can write the equality constraints in asA khx hsIt follows from this equation that if the normal step satis es then the vector hx Sk hs is orthogonal to vx S k vs The authors can therefore write the trust region constraintas k hx S k hs kkwherek k k vx S k vs kIf orthogonality does not hold the authors can still write the trust region constraint as where nowk k k vx S k vs kThis ensures that d is within the trust region of although it restricts h more than necessary in some cases 

The vector of slack variables at the k th iteration is written as sk and its i th component is s i kAlgorithm for the Barrier ProblemThe authors now give a detailed description of the algorithm for solving the barrier problem that was loosely described in step of the Algorithm Outline in xFrom now on the authors will let Bk stand for r xxL xk sk k or for a symmetric matrix approximating this Hessian 

The authors refer to as the primal dual iterationConsider now the SQP subproblem with the Hessian of the Lagrangian W replaced byWr xxL x sSIt is easy to see that if the quadratic program is strictly convex the step generated by the SQP approach coincides with the solution of Comparing and the authors see that the only di erence between the primal and primal dual SQP formulations is that the matrix S has been replaced by SThis degree of generality justi es the investigation of SQP as a framework for designing interior point methods for nonlinear programming Several choices for the Hessian matrix W could be considered but in this study the authors focus on the primal exact Hessian version because of its simplicity