Stochastic First- and Zeroth-Order Methods for Nonconvex Stochastic Programming
Saeed Ghadimi,Guanghui Lan +1 more
Reads0
Chats0
TLDR
The randomized stochastic gradient (RSG) algorithm as mentioned in this paper is a type of approximation algorithm for non-convex nonlinear programming problems, and it has a nearly optimal rate of convergence if the problem is convex.Abstract:
In this paper, we introduce a new stochastic approximation type algorithm, namely, the randomized stochastic gradient (RSG) method, for solving an important class of nonlinear (possibly nonconvex) stochastic programming problems. We establish the complexity of this method for computing an approximate stationary point of a nonlinear programming problem. We also show that this method possesses a nearly optimal rate of convergence if the problem is convex. We discuss a variant of the algorithm which consists of applying a postoptimization phase to evaluate a short list of solutions generated by several independent runs of the RSG method, and we show that such modification allows us to improve significantly the large-deviation properties of the algorithm. These methods are then specialized for solving a class of simulation-based optimization problems in which only stochastic zeroth-order information is available.read more
Citations
More filters
Proceedings ArticleDOI
ZOO: Zeroth Order Optimization Based Black-box Attacks to Deep Neural Networks without Training Substitute Models
TL;DR: An effective black-box attack that also only has access to the input (images) and the output (confidence scores) of a targeted DNN is proposed, sparing the need for training substitute models and avoiding the loss in attack transferability.
Proceedings ArticleDOI
ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Models
TL;DR: Zeroth order optimization (ZOO) as discussed by the authors was proposed to estimate the gradients of the target DNN for generating adversarial examples, which was shown to be as effective as the state-of-the-art white-box attack.
Journal ArticleDOI
Accelerated gradient methods for nonconvex nonlinear and stochastic programming
Saeed Ghadimi,Guanghui Lan +1 more
TL;DR: The AG method is generalized to solve nonconvex and possibly stochastic optimization problems and it is demonstrated that by properly specifying the stepsize policy, the AG method exhibits the best known rate of convergence for solving general non Convex smooth optimization problems by using first-order information, similarly to the gradient descent method.
Posted Content
Certifying Some Distributional Robustness with Principled Adversarial Training
TL;DR: In this paper, a training procedure that augments model parameter updates with worst-case perturbations of training data is proposed to guarantee moderate levels of robustness with little computational or statistical cost relative to empirical risk minimization.
Posted Content
Federated Optimization in Heterogeneous Networks
TL;DR: FedProx as discussed by the authors is a generalization and re-parametrization of FedAvg, which is the state-of-the-art method for federated learning.