We consider the class of iterative shrinkage-thresholding algorithms (ISTA) for solving linear inverse problems arising in signal/image processing. This class of methods, which can be viewed as an extension of the classical gradient algorithm, is attractive due to its simplicity and thus is adequate for solving large-scale problems even with dense matrix data. However, such methods are also known to converge quite slowly. In this paper we present a new fast iterative shrinkage-thresholding algorithm (FISTA) which preserves the computational simplicity of ISTA but with a global rate of convergence which is proven to be significantly better, both theoretically and practically. Initial promising numerical results for wavelet-based image deblurring demonstrate the capabilities of FISTA which is shown to be faster than ISTA by several orders of magnitude.

http://people.rennes.inria.fr/Cedric.Herzet/Cedric.Herzet/Sparse_Seminar/Entrees/2012/11/12_A_Fast_Iterative_Shrinkage-Thresholding_Algorithmfor_Linear_Inverse_Problems_(A._Beck,_M._Teboulle)_files/Breck_2009.pdf

A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems

This monograph is about a class of optimization algorithms called proximal algorithms. Much like Newton's method is a standard tool for solving unconstrained smooth optimization problems of modest size, proximal algorithms can be viewed as an analogous tool for nonsmooth, constrained, large-scale, or distributed versions of these problems. They are very generally applicable, but are especially well-suited to problems of substantial recent interest involving large or high-dimensional datasets. Proximal methods sit at a higher level of abstraction than classical algorithms like Newton's method: the base operation is evaluating the proximal operator of a function, which itself involves solving a small convex optimization problem. These subproblems, which generalize the problem of projecting a point onto a convex set, often admit closed-form solutions or can be solved very quickly with standard or simple specialized methods. Here, we discuss the many different interpretations of proximal operators and algorithms, describe their connections to many other topics in optimization and applied mathematics, survey some popular algorithms, and provide a large number of examples of proximal operators that commonly arise in practice.

Proximal Algorithms

In this paper we propose a new approach for constructing efficient schemes for non-smooth convex optimization. It is based on a special smoothing technique, which can be applied to functions with explicit max-structure. Our approach can be considered as an alternative to black-box minimization. From the viewpoint of efficiency estimates, we manage to improve the traditional bounds on the number of iterations of the gradient schemes from ** keeping basically the complexity of each iteration unchanged.

/pdf/smooth-minimization-of-non-smooth-functions-2gbnzqokeh.pdf

Smooth minimization of non-smooth functions

A tutorial account of variable structure control with sliding mode is presented. The purpose is to introduce in a concise manner the fundamental theory, main results, and practical applications of this powerful control system design approach. This approach is particularly attractive for the control of nonlinear systems. Prominent characteristics such as invariance, robustness, order reduction, and control chattering are discussed in detail. Methods for coping with chattering are presented. Both linear and nonlinear systems are considered. Future research areas are suggested and an extensive list of references is included. >

Variable structure control: a survey

The Theory of Matrices. By F R. Gantmacher. Two volumes, pp. 374 and 276. 1959. (Translated from the Russian by K. A. Hirsch; Chelsea Publishing Company, New York)

Preface. MATHEMATICAL REVIEW. Methods of Proof and Some Notation. Vector Spaces and Matrices. Transformations. Concepts from Geometry. Elements of Calculus. UNCONSTRAINED OPTIMIZATION. Basics of Set--Constrained and Unconstrained Optimization. One--Dimensional Search Methods. Gradient Methods. Newton's Method. Conjugate Direction Methods. Quasi--Newton Methods. Solving Ax = b. Unconstrained Optimization and Neural Networks. Genetic Algorithms. LINEAR PROGRAMMING. Introduction to Linear Programming. Simplex Method. Duality. Non--Simplex Methods. NONLINEAR CONSTRAINED OPTIMIZATION. Problems with Equality Constraints. Problems with Inequality Constraints. Convex Optimization Problems. Algorithms for Constrained Optimization. References. Index.

An introduction to optimization

A Lyapunov-based stabilizing control design method for uncertain nonlinear dynamical systems using fuzzy models is proposed. The controller is constructed using a design model of the dynamical process to be controlled. The design model is obtained from the truth model using a fuzzy modeling approach. The truth model represents a detailed description of the process dynamics. The truth model is used in a simulation experiment to evaluate the performance of the controller design. A method for generating local models that constitute the design model is proposed. Sufficient conditions for stability and stabilizability of fuzzy models using fuzzy state feedback controllers are given. The results obtained are illustrated with a numerical example involving a four-dimensional nonlinear model of a stick balancer.

Stabilizing controller design for uncertain nonlinear systems using fuzzy models

Methods for identification and control of dynamical systems by adalines, two-layer, and three-layer feedforward neural networks (FNNs) using generalized weight adaptation algorithms are discussed. The FNNs considered contain odd nonlinear operators in both the neurons and the weight adaptation algorithms. Two application examples, each involving a nonlinear dynamical system, are considered. The first is identification of the system's forward and inverse dynamics. The second is control of the system using coordination of feedforward and feedback control combined with inverse system dynamics identification. Simulation results are used to verify the method's feasibility and to examine the effect of ENN parameter changes. Specifically the effect that the type of nonlinear activation functions present in the neurons and the type of nonlinear functions present in the weight adaptation algorithms have on FNN system dynamics identification performance is investigated. >

Application of feedforward neural networks to dynamical system identification and control

A class of variable structure output feedback controllers for uncertain dynamic systems with bounded uncertainties is proposed. No statistical information about the uncertain elements is assumed. A variable structure systems approach and a geometric approach to the analysis and synthesis of system zeros are employed in the synthesis of the proposed controllers. >

On variable structure output feedback controllers for uncertain dynamic systems

In this paper we study three different classes of neural network models for solving linear programming problems. We investigate the following characteristics of each model: model complexity, complexity of individual neurons, and accuracy of solutions. Simulation examples are given to illustrate the dynamical behavior of each model. >

Stanislaw H. Zak

Papers

An introduction to optimization

Stabilizing controller design for uncertain nonlinear systems using fuzzy models

Application of feedforward neural networks to dynamical system identification and control

On variable structure output feedback controllers for uncertain dynamic systems

Solving linear programming problems with neural networks: a comparative study