Local Linear Convergence for Alternating and Averaged Nonconvex Projections

TL;DR: It is proved that von Neumann’s method of “alternating projections” converges locally to a point in the intersection, at a linear rate associated with a modulus of regularity.

Abstract: The idea of a finite collection of closed sets having “linearly regular intersection” at a point is crucial in variational analysis. This central theoretical condition also has striking algorithmic consequences: in the case of two sets, one of which satisfies a further regularity condition (convexity or smoothness, for example), we prove that von Neumann’s method of “alternating projections” converges locally to a point in the intersection, at a linear rate associated with a modulus of regularity. As a consequence, in the case of several arbitrary closed sets having linearly regular intersection at some point, the method of “averaged projections” converges locally at a linear rate to a point in the intersection. Inexact versions of both algorithms also converge linearly.

Summary

1 Introduction

• The authors interest here is not in the development of practical numerical methods.
• Notwithstanding linear convergence proofs, basic alternating and averaged projection schemes may be slow in practice.
• Rather the authors aim to study the interplay between a simple, popular, fundamental algorithm and a variety of central ideas from variational analysis.
• Whether such an approach can help in the design and analysis of more practical algorithms remains to be seen.

Corollary 4.10 (approximate monotonicity)

• If the authors replace the normal cone N C in the property described in the result above by its convex hull, the "Clarke normal cone", they obtain a stronger property, called "subsmoothness" in [4] .
• Similar proofs to those above show that, like super-regularity, subsmoothness is a consequence of either amenability or prox-regularity.
• Subsmoothness is strictly stronger than superregularity.
• In a certain sense, however, the distinction between subsmoothness and super-regularity is slight.
• Since super-regularity implies Clarke regularity, the normal cone and Clarke normal cone coincide throughout F ∩ U, and hence F is also subsmooth throughout F ∩ U.

Theorem 5.2 (linear convergence of alternating projections)

• Adding this inequality to the previous inequality then gives the right-hand side of (5.7), as desired.
• The authors can now easily check that the sequence (x k ) is Cauchy and therefore converges.
• Then any alternating projection sequence with initial point sufficiently near x must converge to a point in F ∩ C with R-linear rate √ c. Proof.
• The authors have shown that c also controls the speed of linear convergence for the method of alternating projections applied to the sets F and C. Inevitably, Theorem 5.16 concerns local convergence: it relies on finding an initial point x 0 sufficiently close to a point of linearly regular intersection.
• One example is the case of two manifolds [30] .

8 Prox-regularity and averaged projections

• For each iteration k. Random examples are interesting for their simple test of averaged projections: the challenging question of checking a priori the linear regularity of the intersection of the three sets is open, but randomness seems to prevent irregular solutions, providing α is not too small.
• So in this situation, the authors would hope that the algorithm will converge locally linearly; this is indeed what the numerical results in Figure 9 suggest.
• The authors observed that the method still appears locally linearly convergent in practice, and again, that the rate is better than for averaged projections.
• This example illustrates how the projection algorithm behaves on random feasibility problems of this type.
• Further study and more complete testing have to be done for these questions; this is beyond the scope of this paper.

Figures

Figure 1: Convergence of averaged projection algorithm for designing compression matrix in compressed sensing.

Full text

HAL Id: hal-00389555
https://hal.archives-ouvertes.fr/hal-00389555
Submitted on 2 Jun 2009
HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entic research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.
L’archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diusion de documents
scientiques de niveau recherche, publiés ou non,
émanant des établissements d’enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
Local linear convergence of alternating and averaged
nonconvex projections
Adrian Lewis, David Russel Luke, Jérôme Malick
To cite this version:
Adrian Lewis, David Russel Luke, Jérôme Malick. Local linear convergence of alternating and averaged
nonconvex projections. Foundations of Computational Mathematics, Springer Verlag, 2009, 9 (4),
pp.485-513. �10.1007/s10208-008-9036-y�. �hal-00389555�

Local linear convergence for alternating and
averaged nonconvex projections
A.S. Lewis
∗
D.R. Luke
†
J. Malick
‡
September 16, 2008
Key words: alternating projections, averaged projections, linear conver-
gence, metric regularity, distance to ill-posedness, variational analysis, non-
convexity, extremal principle, prox-regularity
AMS 2000 Subject Classification: 49M20, 65K10, 90C30
Abstract
The idea of a fin ite collection of closed sets having “linearly regular
intersection” at a point is crucial in variational analysis. This central
theoretical condition also has striking algorithmic consequences: in the
case of two sets, one of which satisfies a further regularity condition
(convexity or smoothness for example), we prove that von Neumann’s
method of “alternating projections” converges locally to a point in the
intersection, at a linear rate associated with a modulus of regularity.
As a consequence, in the case of several arbitrary closed sets having
linearly regular intersection at some point, the method of “averaged
projections” converges locally at a linear rate to a point in the inter-
section. Inexact versions of b oth algorithms also converge linearly.
∗
ORIE, Cornell University, Ithaca, NY 14853, U.S.A. aslewis@orie.cornell.edu
people.orie.cornell.edu/~ aslewis. Research supported in part by National Science
Foundation Grant DMS-0504032.
†
Department of Mathematical Sciences, University of Delaware.
rluke@math.udel.edu
‡
CNRS, Lab. Jean Kunztmann, University of Grenoble. jerome.malick@inria.fr
1

1 Introduction
An important theme in computational mathematics is the relationship be-
tween “conditioning” of a problem instance and speed o f convergence of iter-
ative solution algorithms on that instance. A classical example is the method
of conjugate gradients for a positive definite system of linear equations: the
relative condition number of the associated matrix gives a bound on the lin-
ear convergence rate. More generally, Renegar [41–43] showed that the rate
of convergence of interior-point methods for conic convex programming can
be bounded in terms of the “distance to ill-posedness” of the program.
In studying the convergence of iterative algorithms for nonconvex min-
imization problems or nonmonoto ne variational inequalities, we must con-
tent ourselves with a local theory. A suitable analo gue of the distance to
ill-posedness is then the notion of “metric regularity”, fundamenta l in vari-
ational analysis. Loosely speaking, a constraint system, such as a system of
inequalities, for example, is metrically regular when, locally, we can bound
the distance from a trial solution to an exact solution by a constant multiple
of the error in the equation generated by the trial solution. The constant
needed is called the “regularity modulus”, and its reciprocal has a natural
interpretation as a distance to ill-posedness for the equation [19]. While
not a ppropriate as a universal condition on general variational systems [34],
metric regularity is often a reasonable assumption f or constraint systems.
This philosophy suggests understanding the speed of convergence of algo-
rithms for solving constraint systems in terms of the regularity modulus at a
solution. Recent literature focuses in particular on the proximal point algo-
rithm (see for example [1,13,26,37]). After the initial version [29] of this arti-
cle, an independent but related, proximal-type development was announced
in [2]. A unified approach to the relationship between metric regularity and
the linear convergence of a family of conceptual algorithms appears in [27].
We here study a very basic algorithm for a very basic problem. We
consider the problem of finding a point in the intersection o f several closed
sets, using the method of averaged projections: at each step, we project the
current iterate onto each set, and average the results to obtain the next
iterate. Global convergence of this method for convex sets was proved in
1969 in [3]. Here we show, in complete generality, that this method converges
locally to a point in the intersection of the sets, at a linear rate governed by an
associated regularity modulus. Our linear convergence proof is elementary:
although we use the idea of the normal cone, we apply only the definition,
2

and we discuss metric regularity only to illuminate the rate of convergence.
Finding a point in the intersection of several sets is a problem of fun-
damental computational significance. In the case of closed halfspaces, fo r
example, the problem is equivalent to linear programming. We mention
some nonconvex examples below.
Our approach to the convergence of the method of averaged projections
is standard [5, 38,39]: we identify the method with von Neumann’s alternat-
ing projections algorithm [49] on two closed sets (one of which is a linear
subspace) in a suitable product space. A nice development of the classical
method of alternating proj ections in the convex case may be found in [15].
The convergence of the method for two intersecting closed convex sets was
proved in [8], and linear convergence under a regular intersection assumption
was proved in [5], strengthening a classical result of [25]. Our algorithmic con-
tribution is to show that, assuming linear regularity, local linear convergence
does not depend on convexity of both sets, but rather on a good geometric
property (such as convexity, smoothness, or more generally, “amenability ”
or “prox-regularity”) of just one of the two.
One consequence of our convergence proof is a n algorithmic demonstra-
tion for the “exact extremal principle” o f [31] (see also [33, Theorem 2.8]).
This result, a unifying theme in [33], asserts tha t if several sets have linearly
regular intersection at a point, then that point is not “locally extremal”:
that is, translating the sets by sufficiently small vectors cannot render t he
intersection empty locally. To prove this result, we simply apply the method
of averaged projections, starting from the point of regular intersection. In
a further section, we show that inexact versions of the method of averaged
projections, closer to practical implementations, also converge linearly.
The method of averaged projections is a conceptual algorithm that might
appear hard to implement on concrete nonconvex problems. However, the
projection problem for some nonconvex sets is relatively easy. A good exam-
ple is the set of matrices of some fixed rank: g iven a singular value decompo-
sition of a matrix, projecting it onto this set is immediate. Furthermore, non-
convex alternating projection algorithms and analogous heuristics are quite
popular in practice, in areas such as inverse eigenvalue problems [10,11], pole
placement [35,51], information theory [48], low-order control design [23,24,36]
and image processing [7, 50]. Previous convergence results on nonconvex al-
ternating projection algorithms have been uncommon, and have either fo-
cussed on a very special case (see for example [10, 30]), or have been much
weaker than for the convex case [14, 48]. For more discussion, see [30].
3

Our results primarily concern R -linear convergence: we show that our se-
quences of iterates converge, with error bounded by a geometric sequence. In
a final section, we employ a completely different approach to show that the
method of averaged projections, for prox-regular sets with regular intersec-
tion, has a Q-linear convergence property: each iteration guarantees a fixed
rate of improvement. In a final section, we illustrate these theoretical results
with an elementary numerical example coming from signal processing.
Our interest here is not in the development of practical numerical meth-
ods. Notwithstanding linear convergence proofs, basic alternating and aver-
aged projection schemes may be slow in practice. Rather we aim to study the
interplay between a simple, popular, fundamental algorithm and a variety of
centr al ideas f r om variational analysis. Whether such an approach can help
in the design and analysis of more practical algorithms remains to be seen.
2 Notation and definitions
We fix some notation and definitions. Our underlying setting throughout
this work is a Euclidean space E with corresponding closed unit ball B. For
any point x ∈ E and radius ρ > 0 , we write B
ρ
(x) for the set x + ρB.
Consider first two sets F, G ⊂ E. A point ¯x ∈ F ∩G is locally extremal [33]
for this pair of sets if there exists a constant ρ > 0 and a sequence of vectors
z
r
→ 0 in E such that (F + z
r
) ∩G ∩B
ρ
(¯x) = ∅ for all r = 1, 2, . . .. In other
words, restricting to a neighborhood of ¯x and then translating the sets by
arbitrarily small distances can render their intersection empty. Clearly ¯x is
not locally extremal if and only if
0 ∈ int

((F − ¯x) ∩ ρB) − ((G − ¯x) ∩ ρB)

for all ρ > 0.
For recognition purposes, it is easier to study a weaker property than local
extremality. We say that two sets F, G ⊂ E have linearly reg ular intersection
at the point ¯x ∈ F ∩ G if there exist constants α, δ > 0 such that for all
points x ∈ F ∩ B
δ
(¯x) and z ∈ G ∩ B
δ
(¯x), and all ρ ∈ (0, δ], we have
αρB ⊂ ((F − x) ∩ ρB) − ((G −z) ∩ ρB).
(In [28] this property is called “strong regularity”.) By considering the case
x = z = ¯x, we see that linear regularity implies that ¯x is not locally extremal.
This “ primal” definition of linear r egularity is often not the most convenient
4

Frequently asked questions

Q1. What contributions have the authors mentioned in the paper "Local linear convergence of alternating and averaged nonconvex projections" ?

This central theoretical condition also has striking algorithmic consequences: in the case of two sets, one of which satisfies a further regularity condition ( convexity or smoothness for example ), the authors prove that von Neumann ’ s method of “ alternating projections ” converges locally to a point in the intersection, at a linear rate associated with a modulus of regularity. As a consequence, in the case of several arbitrary closed sets having linearly regular intersection at some point, the method of “ averaged projections ” converges locally at a linear rate to a point in the intersection. 

Q2. What is the main theme in computational mathematics?

An important theme in computational mathematics is the relationship between “conditioning” of a problem instance and speed of convergence of iterative solution algorithms on that instance. 

Q3. What is the definition of Clarke regularity?

“Clarke regularity” is a basic variation-geometric property of sets, shared in particular by closed convex sets and smooth manifolds. 

Q4. What is the main result of the method of averaged projections?

Their main result shows, assuming only linear regularity, that providing the initial point x0 is sufficiently near x̄, any sequence x1, x2, x3, . . . generated by the method of averaged projections converges linearly to a point in the intersection ∩iFi, at a rate governed by the condition modulus. 

Q5. What are some examples of nonconvex alternating projection algorithms?

nonconvex alternating projection algorithms and analogous heuristics are quite popular in practice, in areas such as inverse eigenvalue problems [10,11], pole placement [35,51], information theory [48], low-order control design [23,24,36] and image processing [7, 50]. 

Q6. What is the sequence of even iterates for the latter method?

If x0, x1, x2, . . . is a possible sequence of iterates for the former method, then a possible sequence of even iterates for the latter method is Ax0, Ax1, Ax2, . . .. 

Q7. What is the case of exact projection on the super-regular set C?

The authors might reasonably consider the case of exact projection on the super-regular set C: for example, in the next section, for the method of averaged projections, C is a subspace and computing projections is trivial. 

Q8. How can the authors understand the rate of convergence of conic convex programming?

More generally, Renegar [41–43] showed that the rate of convergence of interior-point methods for conic convex programming can be bounded in terms of the “distance to ill-posedness” of the program. 

Q9. how many ms is the optimal value of the condition modulus?

By equation (3.3) and the definition of the condition modulus, the optimal value of this new problem is1 − 1 m · cond2(F1, F2, . . . , Fm|x̄)as required. 

Q10. What is the proof of linear convergence?

Their linear convergence proof is elementary: although the authors use the idea of the normal cone, the authors apply only the definition,and the authors discuss metric regularity only to illuminate the rate of convergence. 

Q11. What is the prox-regular function of the first set L?

The first set L is a subspace, the second set M is a smooth manifold while the third C is convex; hence the three are prox-regular. 

Q12. What is the definition of linear regularity?

The notion of linear regularity is well-known to be closely related to another central idea in variational analysis: “metric regularity”.