For nonlinear programming problems with equality constraints, Hestenes and Powell have independently proposed a dual method of solution in which squares of the constraint functions are added as penalties to the Lagrangian, and a certain simple rule is used for updating the Lagrange multipliers after each cycle. Powell has essentially shown that the rate of convergence is linear if one starts with a sufficiently high penalty factor and sufficiently near to a local solution satisfying the usual second-order sufficient conditions for optimality. This paper furnishes the corresponding method for inequality-constrained problems. Global convergence to an optimal solution is established in the convex case for an arbitrary penalty factor and without the requirement that an exact minimum be calculated at each cycle. Furthermore, the Lagrange multipliers are shown to converge, even though the optimal multipliers may not be unique.

The multiplier method of Hestenes and Powell applied to convex programming

Lagrange multipliers used to be viewed as auxiliary variables introduced in a problem of constrained minimization in order to write first-order optimality conditions formally as a system of equations. Modern applications, with their emphasis on numerical methods and more complicated side conditions than equations, have demanded deeper understanding of the concept and how it fits into a larger theoretical picture. A major line of research has been the nonsmooth geometry of one-sided tangent and normal vectors to the set of points satisfying the given constraints. Another has been the game-theoretic role of multiplier vectors as solutions to a dual problem. Interpretations as generalized derivatives of the optimal value with respect to problem parameters have also been explored. Lagrange multipliers are now being seen as arising from a general rule for the subdifferentiation of a nonsmooth objective function which allows black-and-white constraints to be replaced by penalty expressions. This paper traces su...

Lagrange multipliers and optimality

In this paper sufficient conditions for local and superlinear convergence to a Kuhn—Tucker point are established for a class of algorithms which may be broadly defined and comprise a quadratic programming algorithm for repeated solution of a subproblem and a variable metric update to develop the Hessian in the subproblem. In particular the DFP update and an update attributed to Powell are shown to provide a superlinear convergent subclass of algorithms provided a start is made sufficiently close to the solution and the initial Hessian in the subproblem is sufficiently close to the Hessian of the Lagrangian at this point.

/pdf/superlinearly-convergent-variable-metric-algorithms-for-3omhmvevjd.pdf

Superlinearly convergent variable metric algorithms for general nonlinear programming problems.

Several recent algorithms for solving nonlinear programming problems with equality constraints have made use of an augmented “penalty” Lagrangian function, where terms involving squares of the constraint functions are added to the ordinary Lagrangian. In this paper, the corresponding penalty Lagrangian for problems with inequality constraints is described, and its relationship with the theory of duality is examined. In the convex case, the modified dual problem consists of maximizing a differentiable concave function (indirectly defined) subject to no constraints at all. It is shown that any maximizing sequence for the dual can be made to yield, in a general way, an asymptotically minimizing sequence for the primal which typically converges at least as rapidly.

A dual approach to solving nonlinear programming problems by unconstrained optimization

https://www.mit.edu/~dimitrib/Multiplier_Methods_Paper.pdf

Expository & survey paper: Multiplier methods: A survey

A saddle point theory in terms of extended Lagrangian functions is presented for nonconvex programs. The results parallel those for convex programs conjoined with the usual Lagrangian formulation.

S. M. Howe

Papers

A general saddle point result for constrained optimization