/pdf/convex-analysisnoer-san-nojin-zhan-nituite-5dl2rbmoyr.pdf

Convex Analysisの二,三の進展について

This monograph is about a class of optimization algorithms called proximal algorithms. Much like Newton's method is a standard tool for solving unconstrained smooth optimization problems of modest size, proximal algorithms can be viewed as an analogous tool for nonsmooth, constrained, large-scale, or distributed versions of these problems. They are very generally applicable, but are especially well-suited to problems of substantial recent interest involving large or high-dimensional datasets. Proximal methods sit at a higher level of abstraction than classical algorithms like Newton's method: the base operation is evaluating the proximal operator of a function, which itself involves solving a small convex optimization problem. These subproblems, which generalize the problem of projecting a point onto a convex set, often admit closed-form solutions or can be solved very quickly with standard or simple specialized methods. Here, we discuss the many different interpretations of proximal operators and algorithms, describe their connections to many other topics in optimization and applied mathematics, survey some popular algorithms, and provide a large number of examples of proximal operators that commonly arise in practice.

Proximal Algorithms

General Topology-I

Convex Analysis: (pms-28)

One of the mysteries in the success of neural networks is randomly initialized first order methods like gradient descent can achieve zero training loss even though the objective function is non-convex and non-smooth. This paper demystifies this surprising phenomenon for two-layer fully connected ReLU activated neural networks. For an $m$ hidden node shallow neural network with ReLU activation and $n$ training data, we show as long as $m$ is large enough and no two inputs are parallel, randomly initialized gradient descent converges to a globally optimal solution at a linear convergence rate for the quadratic loss function. 
Our analysis relies on the following observation: over-parameterization and random initialization jointly restrict every weight vector to be close to its initialization for all iterations, which allows us to exploit a strong convexity-like property to show that gradient descent converges at a global linear rate to the global optimum. We believe these insights are also useful in analyzing deep models and other first order methods.

Gradient Descent Provably Optimizes Over-parameterized Neural Networks

Many basic functions of eigenvalue optimization can be written in the composite form $f\circ\lambda$, where $f$ is a permutation-invariant function on $\R^n$ and $\lambda$ is the mapping assigning to each symmetric matrix its ordered vector of eigenvalues. We prove that for such functions, partly smooth structure of $f$ corresponds exactly to the partly smooth structure of $f\circ\lambda$. Many eigenvalue functions admit the composite form with $f$ in addition convex polyhedral. Since convex polyhedral functions always have partly smooth structure, our results imply that partial smoothness is a persistent feature of problems involving eigenvalues. Moreover, for such functions, a duality theory of partly smooth manifolds becomes available.

Spectral lifts of identifiable sets and partly smooth manifolds

Minimizing the Euclidean distance to a set arises frequently in applications. When the set is algebraic, a measure of complexity of this optimization problem is its number of critical points. In this paper we provide a general framework to compute and count the real smooth critical points of a data matrix on an orthogonally invariant (spectral) set of matrices. The technique relies on “transfer principles” that allow calculations to be done in the space of singular values of the matrices in the spectral set. The calculations often simplify greatly and yield transparent formulas. We illustrate the method on several examples, and compare our results to the recently introduced notion of Euclidean distance degree of an algebraic variety.

Counting real critical points of the distance to spectral matrix sets

We introduce a class of stochastic algorithms for minimizing weakly convex functions over proximally smooth sets. As their main building blocks, the algorithms use simplified models of the objective function and the constraint set, along with a retraction operation to restore feasibility. All the proposed methods come equipped with a finite time efficiency guarantee in terms of a natural stationarity measure. We discuss consequences for nonsmooth optimization over smooth manifolds and over sets cut out by weakly-convex inequalities.

Stochastic optimization over proximally smooth sets

We prove that quasiconvex functions always admit descent trajectories bypassing all non-minimizing critical points.

Orbits of geometric descent

We consider the problem of constructing quantum operations or channels, if they exist, that transform a given set of quantum states $\{\rho_1, \dots, \rho_k\}$ to another such set $\{\hat\rho_1, \dots, \hat\rho_k\}$. In other words, we must find a {\em completely positive linear map}, if it exists, that maps a given set of density matrices to another given set of density matrices. This problem, in turn, is an instance of a positive semi-definite feasibility problem, but with highly structured constraints. The nature of the constraints makes projection based algorithms very appealing when the number of variables is huge and standard interior point-methods for semi-definite programming are not applicable. We provide emperical evidence to this effect. We moreover present heuristics for finding both high rank and low rank solutions. Our experiments are based on the \emph{method of alternating projections} and the \emph{Douglas-Rachford} reflection method.

Dmitriy Drusvyatskiy

Papers

Spectral lifts of identifiable sets and partly smooth manifolds

Counting real critical points of the distance to spectral matrix sets

Stochastic optimization over proximally smooth sets

Orbits of geometric descent

Projection methods in quantum information science