The paper proposes and justifies a new algorithm of the proximal Newton type to solve a broad class of nonsmooth composite convex optimization problems without strong convexity assumptions. Based on advanced notions and techniques of variational analysis, we establish implementable results on the global convergence of the proposed algorithm as well as its local convergence with superlinear and quadratic rates. For certain structural problems, the obtained local convergence conditions do not require the local Lipschitz continuity of the corresponding Hessian mappings that is a crucial assumption used in the literature to ensure a superlinear convergence of other algorithms of the proximal Newton type. The conducted numerical experiments of solving the $l_1$ regularized logistic regression model illustrate the possibility of applying the proposed algorithm to deal with practically important problems.

A Globally Convergent Proximal Newton-Type Method in Nonsmooth Convex Optimization

In this paper, we study continuity and Lipschitzian properties of set-valued mappings, focusing on inner-type conditions. We introduce new notions of inner calmness* and, its relaxation, fuzzy inner calmness*. We show that polyhedral maps enjoy inner calmness* and examine (fuzzy) inner calmness* of a multiplier mapping associated with constraint systems in depth. Then we utilize these notions to develop some new rules of generalized differential calculus, mainly for the primal objects (e.g. tangent cones). In particular, we propose an exact chain rule for graphical derivatives. We apply these results to compute the derivatives of the normal cone mapping, essential e.g. for sensitivity analysis of variational inequalities.

On inner calmness*, generalized calculus, and derivatives of the normal cone mapping

The optimal value function is one of the basic objects in the field of mathematical optimization, as it allows the evaluation of the variations in the cost/revenue generated while minimizing/maximizing a given function under some constraints. In the context of stability/sensitivity analysis, a large number of publications have been dedicated to the study of continuity and differentiability properties of the optimal value function. The differentiability aspect of works in the current literature has mostly been limited to first order analysis, with focus on estimates of its directional derivatives and subdifferentials, given that the function is typically nonsmooth. With the progress made in the last two to three decades in major subfields of optimization such as robust, minmax, semi-infinite and bilevel optimization, and their connection to the optimal value function, there is a crucial need for a second order analysis of the generalized differentiability properties of this function. This type of analysis will enable the development of robust solution algorithms, such as the Newton method. The main goal of this paper is to provide results in this direction. In fact, we derive estimates of the generalized Hessian for the optimal value function. Our results are based on two handy tools from parametric optimization, namely the optimal solution and Lagrange multiplier mappings, for which completely detailed estimates of their generalized derivatives are either well-known or can easily be obtained.

Estimates of generalized Hessians for optimal value functions in mathematical programming

This paper provides a variational analysis of the unconstrained formulation of the LASSO problem, ubiquitous in statistical learning, signal processing, and inverse problems. In particular, we establish smoothness results for the optimal value as well as Lipschitz properties of the optimal solution as functions of the right-hand side (or measurement vector) and the regularization parameter. Moreover, we show how to apply the proposed variational analysis to study the sensitivity of the optimal solution to the tuning parameter in the context of compressed sensing with subgaussian measurements. Our theoretical findings are validated by numerical experiments.

LASSO reloaded: a variational analysis perspective with applications to compressed sensing

The projection onto the epigraph or a level set of a closed proper convex function can be achieved by finding a root of a scalar equation that involves the proximal operator as a function of the proximal parameter. This paper develops the variational analysis of this scalar equation. The approach is based on a study of the variational-analytic properties of general convex optimization problems that are (partial) infimal projections of the the sum of the function in question and the perspective map of a convex kernel. When the kernel is the Euclidean norm squared, the solution map corresponds to the proximal map, and thus the variational properties derived for the general case apply to the proximal case. Properties of the value function and the corresponding solution map -- including local Lipschitz continuity, directional differentiability, and semismoothness -- are derived. An SC$^1$ optimization framework for computing epigraphical and level-set projections is thus established. Numerical experiments on 1-norm projection illustrate the effectiveness of the approach as compared with specialized algorithms

From perspective maps to epigraphical projections.

This paper proposes and develops a new Newton-type algorithm to solve subdifferential inclusions defined by subgradients of extended-real-valued prox-regular functions. The proposed algorithm is formulated in terms of the second-order subdifferential of such functions that enjoys extensive calculus rules and can be efficiently computed for broad classes of extended-real-valued functions. Based on this and on metric regularity and subregularity properties of subgradient mappings, we establish verifiable conditions ensuring well-posedness of the proposed algorithm and its local superlinear convergence. The obtained results are also new for the class of equations defined by continuously differentiable functions with Lipschitzian derivatives ($\mathcal{C}^{1,1}$ functions), which is the underlying case of our consideration. The developed algorithm for prox-regular functions is formulated in terms of proximal mappings related to and reduces to Moreau envelopes. Besides numerous illustrative examples and comparison with known algorithms for $\mathcal{C}^{1,1}$ functions and generalized equations, the paper presents applications of the proposed algorithm to the practically important class of Lasso problems arising in statistics and machine learning.

A Generalized Newton Method for Subgradient Systems

For a 
$$\mathcal {C}^2$$

-smooth function on a finite-dimensional space, a necessary condition for its quasiconvexity is the positive semidefiniteness of its Hessian matrix on the subspace orthogonal to its gradient, whereas a sufficient condition for its strict pseudoconvexity is the positive definiteness of its Hessian matrix on the subspace orthogonal to its gradient. Our aim in this paper is to extend those conditions for 
$$\mathcal {C}^{1,1}$$

-smooth functions by using the Frechet and Mordukhovich second-order subdifferentials.

Second-order characterizations of quasiconvexity and pseudoconvexity for differentiable functions with Lipschitzian derivatives

The paper is devoted to the study, characterizations, and applications of variational convexity of functions, the property that has been recently introduced by Rockafellar together with its strong counterpart. First we show that these variational properties of an extended-real-valued function are equivalent to, respectively, the conventional (local) convexity and strong convexity of its Moreau envelope. Then we derive new characterizations of both variational convexity and variational strong convexity of general functions via their second-order subdifferentials (generalized Hessians), which are coderivatives of subgradient mappings. We also study relationships of these notions with local minimizers and tilt-stable local minimizers. The obtained results are used for characterizing related notions of variational and strong variational sufficiency in composite optimization with applications to nonlinear programming.

Variational Convexity of Functions and Variational Sufficiency in Optimization

For a $\mathcal{C}^2$-smooth function on a finite-dimensional space, a necessary condition for its quasiconvexity is the positive semidefiniteness of its Hessian matrix on the subspace orthogonal to its gradient, whereas a sufficient condition for its strict pseudoconvexity is the positive definiteness of its Hessian matrix on the subspace orthogonal to its gradient. Our aim in this paper is to extend those conditions for $\mathcal{C}^{1,1}$-smooth functions by using the Fr\'echet and Mordukhovich second-order subdifferentials.

On Second-order Conditions for Quasiconvexity and Pseudoconvexity of $\mathcal{C}^{1,1}$-smooth Functions

This paper proposes and justifies two new globally convergent Newton-type methods to solve unconstrained and constrained problems of nonsmooth optimization by using tools of variational analysis and generalized differentiation. Both methods are coderivative-based and employ generalized Hessians (coderivatives of subgradient mappings) associated with objective functions, which are either of class $\mathcal{C}^{1,1}$, or are represented in the form of convex composite optimization, where one of the terms may be extended-real-valued. The proposed globally convergent algorithms are of two types. The first one extends the damped Newton method and requires positive-definiteness of the generalized Hessians for its well-posedness and efficient performance, while the other algorithm is of the Levenberg-Marquardt type being well-defined when the generalized Hessians are merely positive-semidefinite. The obtained convergence rates for both methods are at least linear, but becomes superlinear under the so-called semismooth$^*$ property of subgradient mappings. Problems of convex composite optimization are investigated with and without the strong convexity assumption on of smooth parts of objective functions by implementing the machinery of forward-backward envelopes. Numerical experiments are conducted for a basic class of Lasso problems by providing performance comparisons of the new algorithms with some other first-order and second-order methods that are highly recognized in nonsmooth optimization.

Vo Thanh Phat

Papers

A Generalized Newton Method for Subgradient Systems

Second-order characterizations of quasiconvexity and pseudoconvexity for differentiable functions with Lipschitzian derivatives

Variational Convexity of Functions and Variational Sufficiency in Optimization

On Second-order Conditions for Quasiconvexity and Pseudoconvexity of $\mathcal{C}^{1,1}$-smooth Functions

Globally Convergent Coderivative-Based Generalized Newton Methods in Nonsmooth Optimization.