We consider the class of iterative shrinkage-thresholding algorithms (ISTA) for solving linear inverse problems arising in signal/image processing. This class of methods, which can be viewed as an extension of the classical gradient algorithm, is attractive due to its simplicity and thus is adequate for solving large-scale problems even with dense matrix data. However, such methods are also known to converge quite slowly. In this paper we present a new fast iterative shrinkage-thresholding algorithm (FISTA) which preserves the computational simplicity of ISTA but with a global rate of convergence which is proven to be significantly better, both theoretically and practically. Initial promising numerical results for wavelet-based image deblurring demonstrate the capabilities of FISTA which is shown to be faster than ISTA by several orders of magnitude.

http://people.rennes.inria.fr/Cedric.Herzet/Cedric.Herzet/Sparse_Seminar/Entrees/2012/11/12_A_Fast_Iterative_Shrinkage-Thresholding_Algorithmfor_Linear_Inverse_Problems_(A._Beck,_M._Teboulle)_files/Breck_2009.pdf

A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems

This monograph is about a class of optimization algorithms called proximal algorithms. Much like Newton's method is a standard tool for solving unconstrained smooth optimization problems of modest size, proximal algorithms can be viewed as an analogous tool for nonsmooth, constrained, large-scale, or distributed versions of these problems. They are very generally applicable, but are especially well-suited to problems of substantial recent interest involving large or high-dimensional datasets. Proximal methods sit at a higher level of abstraction than classical algorithms like Newton's method: the base operation is evaluating the proximal operator of a function, which itself involves solving a small convex optimization problem. These subproblems, which generalize the problem of projecting a point onto a convex set, often admit closed-form solutions or can be solved very quickly with standard or simple specialized methods. Here, we discuss the many different interpretations of proximal operators and algorithms, describe their connections to many other topics in optimization and applied mathematics, survey some popular algorithms, and provide a large number of examples of proximal operators that commonly arise in practice.

Proximal Algorithms

Splitting algorithms for the sum of two monotone operators.We study two splitting algorithms for (stationary and evolution) problems involving the sum of two monotone operators. These algorithms ar...

Splitting Algorithms for the Sum of Two Nonlinear Operators

Due to their extraordinary utility and broad applicability in many areas of classical mathematics and modern physical sciences (most notably, computerized tomography), algorithms for solving convex feasibility problems continue to receive great attention. To unify, generalize, and review some of these algorithms, a very broad and flexible framework is investigated. Several crucial new concepts which allow a systematic discussion of questions on behaviour in general Hilbert spaces and on the quality of convergence are brought out. Numerous examples are given.

/pdf/on-projection-algorithms-for-solving-convex-feasibility-2wyotvnuad.pdf

On Projection Algorithms for Solving Convex Feasibility Problems

Iterative algorithms for nonexpansive mappings and maximal monotone operators are investigated. Strong convergence theorems are proved for nonexpansive mappings, including an improvement of a result of Lions. A modification of Rockafellar’s proximal point algorithm is obtained and proved to be always strongly convergent. The ideas of these algorithms are applied to solve a quadratic minimization problem.

Iterative Algorithms for Nonlinear Operators

Let C be a closed convex subset of the Banach space X. A subset F of C is called a nonexpansive retract of C if either F = 0 or there exists a retraction of C onto F which is a nonexpansive mapping. The main theorem of this paper is that if T : C -» C is nonexpansive and satisfies a conditional fixed point property, then the fixed-point set of T is a nonexpansive retract of C. This result is used to generalize a theorem of Belluce and Kirk on the existence of a common fixed point of a finite family of commuting nonexpansive mappings. Introduction. In this paper, A' always denotes a Banach space (either real or complex) and C a nonempty, closed and convex subset of X. Most published results on nonexpansive mappings have centered on existence theorems for fixed points of nonexpansive F : C —> X. In this paper we initiate the study of the structure of the fixed-point set F(T) = {x : Tx = x}. In this connection it is useful to assume that a conditional fixed point property is satisfied: Either F has no fixed points, or F has a fixed (CFP) point in every nonempty bounded closed convex set that F leaves invariant. It is obvious that existence theorems serve to define classes of nonexpansive mappings which satisfy (CFP). However, (CFP) holds even in contexts where no existence theorem can be hoped for. For example (Theorem 4), if C is locally weakly compact and X is strictly convex, then every nonexpansive F : C -» X satisfies (CFP). Our principal structure result is Theorem 2: if C is locally weakly compact, T : C -» C is nonexpansive, and F satisfies (CFP), then F(T) is a nonexpansive retract of C. Therefore our approach is to study the structure of F(T) by studying nonexpansive retracts of C. Although we are not primarily interested in existence results, our main structure theorem does permit us to prove an existence result (Theorem 7) under more general hypotheses than before, by a more transparent argument. Received by the editors November 13, 1970 and, in revised form, May 19, 1972. A MS (MOS) subject classifications (1970). Primary 47H10.

/pdf/properties-of-fixed-point-sets-of-nonexpansive-mappings-in-maipics1ul.pdf

Properties of fixed-point sets of nonexpansive mappings in Banach spaces

Asymptotic convergence of nonlinear contraction semigroups in Hilbert space

for every x ∈ C, and that T be continuous for some N ≥ 1. The stronger definition (cf. Goebel and Kirk [8]) requires that each iterate T be Lipschitzian with Lipschitz constants Ln → 1 as n → ∞. For our iteration method we find it convenient to introduce a definition somewhere between these two: T is asymptotically nonexpansive in the intermediate sense provided T is uniformly continuous and lim sup n→∞ sup x,y∈C (‖Tx− Ty‖ − ‖x− y‖) ≤ 0 .

/pdf/convergence-of-iterates-of-asymptotically-nonexpansive-1g04hdsxb5.pdf

Convergence of iterates of asymptotically nonexpansive mappings in Banach spaces with the uniform Opial property

The following theorem is proven:if E is a uniformly rotund Banach space with a Frechet differentiable norm, C is a bounded nonempty closed convex subset of E, and T: C→C is a contraction, then the iterates {Tnx} are weakly almost-convergent to a fixed-point of T.

A simple proof of the mean ergodic theorem for nonlinear contractions in Banach spaces

/pdf/nonexpansive-projections-on-subsets-of-banach-spaces-e8wq1k8ucw.pdf

Ronald E. Bruck

Papers

Properties of fixed-point sets of nonexpansive mappings in Banach spaces

Asymptotic convergence of nonlinear contraction semigroups in Hilbert space

Convergence of iterates of asymptotically nonexpansive mappings in Banach spaces with the uniform Opial property

A simple proof of the mean ergodic theorem for nonlinear contractions in Banach spaces

Nonexpansive projections on subsets of Banach spaces.