The Frank-Wolfe (FW) optimization algorithm has lately re-gained popularity thanks in particular to its ability to nicely handle the structured constraints appearing in machine learning applications. However, its convergence rate is known to be slow (sublinear) when the solution lies at the boundary. A simple less-known fix is to add the possibility to take 'away steps' during optimization, an operation that importantly does not require a feasibility oracle. In this paper, we highlight and clarify several variants of the Frank-Wolfe optimization algorithm that have been successfully applied in practice: away-steps FW, pairwise FW, fully-corrective FW and Wolfe's minimum norm point algorithm, and prove for the first time that they all enjoy global linear convergence, under a weaker condition than strong convexity of the objective. The constant in the convergence rate has an elegant interpretation as the product of the (classical) condition number of the function with a novel geometric quantity that plays the role of a 'condition number' of the constraint set. We provide pointers to where these algorithms have made a difference in practice, in particular with the flow polytope, the marginal polytope and the base polytope for submodular optimization.

/pdf/on-the-global-linear-convergence-of-frank-wolfe-optimization-3vnbzuwtus.pdf

On the global linear convergence of Frank-Wolfe optimization variants

Modern machine learning focuses on highly expressive models that are able to fit or interpolate the data completely, resulting in zero training loss. For such models, we show that the stochastic gradients of common loss functions satisfy a strong growth condition. Under this condition, we prove that constant step-size stochastic gradient descent (SGD) with Nesterov acceleration matches the convergence rate of the deterministic accelerated method for both convex and strongly-convex functions. We also show that this condition implies that SGD can find a first-order stationary point as efficiently as full gradient descent in non-convex settings. Under interpolation, we further show that all smooth loss functions with a finite-sum structure satisfy a weaker growth condition. Given this weaker condition, we prove that SGD with a constant step-size attains the deterministic convergence rate in both the strongly-convex and convex settings. Under additional assumptions, the above results enable us to prove an O(1/k^2) mistake bound for k iterations of a stochastic perceptron algorithm using the squared-hinge loss. Finally, we validate our theoretical findings with experiments on synthetic and real datasets.

Fast and Faster Convergence of SGD for Over-Parameterized Models and an Accelerated Perceptron

Motivated principally by the low-rank matrix completion problem, we present an extension of the Frank--Wolfe method that is designed to induce near-optimal solutions on low-dimensional faces of the feasible region. This is accomplished by a new approach to generating “in-face” directions at each iteration, as well as through new choice rules for selecting between in-face and “regular” Frank--Wolfe steps. Our framework for generating in-face directions generalizes the notion of away steps introduced by Wolfe. In particular, the in-face directions always keep the next iterate within the minimal face containing the current iterate. We present computational guarantees for the new method that trade off efficiency in computing near-optimal solutions with upper bounds on the dimension of minimal faces of iterates. We apply the new method to the matrix completion problem, where low-dimensional faces correspond to low-rank matrices. We present computational results that demonstrate the effectiveness of our methodo...

http://www.optimization-online.org/DB_FILE/2015/11/5191.pdf

An Extended Frank--Wolfe Method with “In-Face” Directions, and Its Application to Low-Rank Matrix Completion

In this paper we propose a primal-dual algorithm with a momentum term that can be viewed as a generalization of the method proposed by Chambolle and Pock in 2016 to solve saddle point problems defined by a convex-concave function $\mathcal{L}(x,y)=f(x)+\Phi(x,y)-h(y)$ with a general coupling term $\Phi(x,y)$ that is not assumed to be bilinear. Given a saddle point $(x^*,y^*)$, assuming $\nabla_y\Phi(\cdot,\cdot)$ is Lipschitz and $\nabla_x\Phi(\cdot,y)$ is Lipschitz in $x$ for any fixed $y$, we derive error bounds in terms of $\mathcal{L}(\bar{x}_k,x^*)-\mathcal{L}(y^*,\bar{y}_k)$ for the ergodic sequence $\{\bar{x}_k,\bar{y}_k\}$; in particular, we show $\mathcal{O}(1/k)$ rate that when the problem is merely convex in $x$. Furthermore, assuming $\Phi(x,\cdot)$ is linear in $y$ for each fixed $x$ and $f$ is strongly convex, we can obtain the ergodic convergence rate of $\mathcal{O}(1/k^2)$ - we are not aware of any other work in the related literature showing $\mathcal{O}(1/k^2)$ rate when $\Phi$ is not bilinear. We tested our method for solving kernel matrix learning problem, and compare it against the Mirror-prox algorithm and interior point methods.

A Primal-Dual Algorithm for General Convex-Concave Saddle Point Problems

Springer Proceedings in Mathematics & Statistics  This book series features volumes composed of selected contributions from workshops and conferences in all areas of current research in mathematics and statistics, including operation research and optimization. In addition to an overall evaluation of the interest, scientific quality, and timeliness of each proposal at the hands of the publisher, individual contributions are all refereed to the high quality standards of leading journals in the field. Thus, this series provides the research community with well-edited, authoritative reports on developments in the most exciting areas of mathematical and statistical research today.

Springer Proceedings in Mathematics & Statistics

Large-scale constrained convex optimization problems arise in several application domains. First-order methods are good candidates to tackle such problems due to their low iteration complexity and ...

A Level-Set Method for Convex Optimization with a Feasible Solution Path

The perceptron algorithm, introduced in the late 1950s in the machine learning community, is a simple greedy algorithm for finding a solution to a finite set of linear inequalities. The algorithm's main advantages are its simplicity and noise tolerance. The algorithm's main disadvantage is its slow convergence rate. We propose a modified version of the perceptron algorithm that retains the algorithm's original simplicity but has a substantially improved convergence rate.

/pdf/a-smooth-perceptron-algorithm-1dtavoz4c3.pdf

A Smooth Perceptron Algorithm

The perceptron algorithm is a simple iterative procedure for finding a point in a convex cone $$F\subseteq \mathbb {R}^m$$F⊆Rm. At each iteration, the algorithm only involves a query to a separation oracle for $$F$$F and a simple update on a trial solution. The perceptron algorithm is guaranteed to find a point in $$F$$F after $$\mathcal O(1/\tau _F^2)$$O(1/?F2) iterations, where $$\tau _F$$?F is the width of the cone $$F$$F. We propose a version of the perceptron algorithm that includes a periodic rescaling of the ambient space. In contrast to the classical version, our rescaled version finds a point in $$F$$F in $$\mathcal O(m^5 \log (1/\tau _F))$$O(m5log(1/?F)) perceptron updates. This result is inspired by and strengthens the previous work on randomized rescaling of the perceptron algorithm by Dunagan and Vempala (Math Program 114:101---114, 2006) and by Belloni et al. (Math Oper Res 34:621---641, 2009). In particular, our algorithm and its complexity analysis are simpler and shorter. Furthermore, our algorithm does not require randomization or deep separation oracles.

http://www.optimization-online.org/DB_FILE//2013/06/3928.pdf

A deterministic rescaled perceptron algorithm

We propose a simple projection and rescaling algorithm to solve the feasibility problem 

$$\begin{aligned} \text { find } x \in L \cap \Omega , \end{aligned}$$

where L and $\Omega $ are respectively a linear subspace and the interior of a symmetric cone in a finite-dimensional vector space V. This projection and rescaling algorithm is inspired by previous work on rescaled versions of the perceptron algorithm and by Chubanov’s projection-based method for linear feasibility problems. As in these predecessors, each main iteration of our algorithm contains two steps: a basic procedure and a rescaling step. When $L \cap \Omega \ne \emptyset $, the projection and rescaling algorithm finds a point $x \in L \cap \Omega $ in at most $\mathcal {O}(\log (1/\delta (L \cap \Omega )))$ iterations, where $\delta (L \cap \Omega ) \in (0,1]$ is a measure of the most interior point in $L \cap \Omega $. The ideal value $\delta (L\cap \Omega ) = 1$ is attained when $L \cap \Omega $ contains the center of the symmetric cone $\Omega $. We describe several possible implementations for the basic procedure including a perceptron scheme and a smooth perceptron scheme. The perceptron scheme requires $\mathcal {O}(r^4)$ perceptron updates and the smooth perceptron scheme requires $\mathcal {O}(r^2)$ smooth perceptron updates, where r stands for the Jordan algebra rank of V.

http://www.optimization-online.org/DB_FILE/2015/12/5257.pdf

Solving Conic Systems via Projection and Rescaling

We propose an elementary algorithm for solving a system of linear inequalities A T y>0 or its alternative Ax=0,x≥0,x≠0. Our algorithm is a smooth version of the perceptron and von Neumann algorithms. Our algorithm retains the simplicity of these algorithms but has a significantly improved convergence rate. Our approach also extends to more general conic systems provided a suitable smoothing oracle is available.

Negar Soheili

Papers

A Level-Set Method for Convex Optimization with a Feasible Solution Path

A Smooth Perceptron Algorithm

A deterministic rescaled perceptron algorithm

Solving Conic Systems via Projection and Rescaling

A Primal–Dual Smooth Perceptron–von Neumann Algorithm