The Tortoise and the Hare restart GMRES

TL;DR: Two simple examples are presented where GM RES(1) converges exactly in three iterations, while GMRES(2) stagnates, revealing that GMRES (1) convergence can be extremely sensitive to small changes in the initial residual.

Abstract: When solving large nonsymmetric systems of linear equations with the restarted GMRES algorithm, one is inclined to select a relatively large restart parameter in the hope of mimicking the full GMRES process. Surprisingly, cases exist where small values of the restart parameter yield convergence in fewer iterations than larger values. Here, two simple examples are presented where GMRES(1) converges exactly in three iterations, while GMRES(2) stagnates. One of these examples reveals that GMRES(1) convergence can be extremely sensitive to small changes in the initial residual.

Figures

Fig. 4. Comparison of GMRES(1) (left) and GMRES(2) (right) as in Figure 2, but for thematrix in (3.1). (Double and quadruple pre ision omputations di er notably only at the boundariesbetween onvergen e and stagnation.)

Fig. 5. Close-up of the left plot of Figure 4. The white urve denotes unstable xed points ofthe map s(1)k+1.

Full text

THE TORTOISE AND THE HARE RESTART GMRES
MARK EMBREE

Abstrat.
When solving large nonsymmetri systems of linear equations with the restarted
GMRES algorithm, one is inlined to selet a relatively large restart parameter in the hop e of
mimiking the full GMRES pro ess. Surprisingly, ases exist where small values of the restart
parameter yield onvergene in fewer iterations than larger values. Here, two simple examples are
presented where GMRES(1) onverges exatly in three iterations, while GMRES(2) stagnates. One
of these examples reveals that GMRES(1) onvergene an b e extremely sensitive to small hanges
in the initial residual.
Key words.
Restarted GMRES, Krylov subspae methods.
AMS sub jet lassiations.
65F10, 37N30
1. Intro dution.
GMRES is an iterative method for solving large nonsymmetri
systems of linear equations,
Ax
=
b
[8℄. Throughout siene and engineering, this
algorithm and its variants routinely solve problems with millions of degrees of freedom.
Its p opularity is rooted in an optimality ondition: At the
k
th iteration, GMRES
omputes the solution estimate
x
k
that minimizes the Eulidean norm of the residual
r
k
=
Ax
k

b
over a subspae of dimension
k
,
k
r
k
k
= min
p
2
P
k
p
(0)=1
k
p
(
A
)
r
0
k
;
(1.1)
where
P
k
denotes those polynomials with degree not exeeding
k
, and
r
0
=
b

Ax
0
is the initial residual. As eah iteration enlarges the minimizing subspae, the residual
norm dereases monotonially.
GMRES optimality omes at a ost, however, sine eah iteration demands both
more arithmeti and memory than the one b efore it. A standard work-around is
to restart the pro ess after some xed number of iterations,
m
. The resulting algo-
rithm, GMRES(
m
), uses the approximate solution
x
m
as the initial guess for a new
run of GMRES, ontinuing this proess until onvergene. The global optimality of
the original algorithm is lost, so although the residual norms remain monotoni, the
restarted pro ess an stagnate with a non-zero residual, failing to ever onverge [8℄.
Sine GMRES(
m
) enfores loal optimality on
m
-dimensional spaes, one antiipates
that inreasing
m
will yield onvergene in fewer iterations. Many pratial examples
onrm this intuition.
We denote the
k
th residual of GMRES(
m
) by
r
(
m
)
k
. To b e preise, one yle
between restarts of GMRES(
m
) is ounted as
m
individual iterations. Conventionally,
then, one exp ets
k
r
(
m
)
k
k  k
r
(
`
)
k
k
for
` < m
. Indeed, this must be true when
k

m
.
Surprisingly, inreasing the restart parameter sometimes leads to
slower
onver-
gene:
k
r
(
m
)
k
k
>
k
r
(
`
)
k
k
for
` < m < k
. The author enountered this phenomenon
while solving a disretized onvetion-diusion equation desribed in [4℄. In unpub-
lished exp eriments, de Sturler [1℄ and Walker and Watson [11℄ observed similar b e-
havior arising in pratial appliations. One wonders, how muh smaller than
k
r
(
m
)
k
k
might
k
r
(
`
)
k
k
be? The smallest possible ases ompare GMRES(1) to GMRES(2) for
3-by-3 matries. Eiermann, Ernst, and Shneider present suh an example for whih

Oxford University Computing Laboratory, Wolfson Building, Parks Road, Oxford OX1 3QD,
United Kingdom (mark.embreeomlab.ox.a.uk). Supp orted by UK Engineering and Physial Si-
enes Researh Counil Grant GR/M12414.
1

2
MARK EMBREE
k
r
(1)
4
k
=
k
r
(2)
4
k
= 0
:
2154
: : :
[2, pp. 284{285℄. Otherwise, the phenomenon we desrib e
has apparently reeived little attention in the literature.
The purp ose of this artile is twofold. First, we desribe a pair of extreme ex-
amples where GMRES(1) onverges exatly at the third iteration, while GMRES(2)
seems to never onverge. The seond example leads to our seond p oint: Small p er-
turbations to the initial residual an dramatially alter the onvergene b ehavior of
GMRES(1).
2. First Example.
Consider using restarted GMRES to solve
Ax
=
b
for
A
=
0

1 1 1
0 1 3
0 0 1
1
A
;
b
=
0

2

4
1
1
A
:
(2.1)
Taking
x
0
=
0
yields the initial residual
r
0
=
b
. Using the fat that
A
and
r
0
are
real, we an derive expliit formulas for GMRES(1) and GMRES(2) diretly from the
GMRES optimality ondition (1.1). The reurrene for GMRES(1),
r
(1)
k
+1
=
r
(1)
k

r
(1)T
k
Ar
(1)
k
r
(1)T
k
A
T
Ar
(1)
k
Ar
(1)
k
;
(2.2)
was studied as early as the 1950s [3,
x
71℄,[7℄. For the
A
and
r
0
=
b
dened in (2.1),
this iteration onverges
exatly
at the third step:
r
(1)
1
=
0

3

3
0
1
A
;
r
(1)
2
=
0

3
0
0
1
A
;
r
(1)
3
=
0

0
0
0
1
A
:
Expressions for one GMRES(2) yle an likewise b e derived using elementary alu-
lus. The up dated residual takes the form
r
(2)
k
+2
=
p
(
A
)
r
(2)
k
, where
p
(
z
) = 1 +
z
+
 z
2
is a quadrati whose o eÆients

=

(
A
;
r
(2)
k
) and

=

(
A
;
r
(2)
k
) are given by

=
(
r
(2)T
k
AAr
(2)
k
)(
r
(2)T
k
A
T
AAr
(2)
k
)

(
r
(2)T
k
Ar
(2)
k
)(
r
(2)T
k
A
T
A
T
AAr
(2)
k
)
(
r
(2)T
k
A
T
Ar
(2)
k
)(
r
(2)T
k
A
T
A
T
AAr
(2)
k
)

(
r
(2)T
k
A
T
AAr
(2)
k
)(
r
(2)T
k
A
T
AAr
(2)
k
)
;

=
(
r
(2)T
k
Ar
(2)
k
)(
r
(2)T
k
A
T
AAr
(2)
k
)

(
r
(2)T
k
AAr
(2)
k
)(
r
(2)T
k
A
T
Ar
(2)
k
)
(
r
(2)T
k
A
T
Ar
(2)
k
)(
r
(2)T
k
A
T
A
T
AAr
(2)
k
)

(
r
(2)T
k
A
T
AAr
(2)
k
)(
r
(2)T
k
A
T
AAr
(2)
k
)
:
Exeuting GMRES(2) on the matrix and right hand side (2.1) reveals
r
(2)
1
=
0

3

3
0
1
A
;
r
(2)
2
=
1
2
0

3
0
3
1
A
;
r
(2)
3
=
1
28
0

24

27
33
1
A
;
r
(2)
4
=
1
122
0

81

108
162
1
A
:
The inferiority of GMRES(2) ontinues well b eyond the fourth iteration. For example:
k
k
r
(2)
k
k
=
k
r
0
k
5 0.376888290025532. . .
10 0.376502488858910. . .
15 0.376496927936533. . .
20 0.376496055944867. . .
25 0.376495995285626. . .
30 0.376495984909087. . .

RESTARTED GMRES
3
k
r
(
m
)
k
k
k
r
0
k
iteration,
k
GMRES(1)
GMRES(2)
0 5 10 15 20 25 30
10
0
10

5
10

10
10

15
Fig. 1
.
Convergene urves for GMRES
(1)
and GMRES
(2)
applied to
(2.1)
with
x
0
=
0
.
The entire onvergene urve for the rst thirty iterations is shown in Figure 1, based
on p erforming GMRES(2) in exat arithmeti using Mathematia.
The partiular value of
b
(and thus
r
0
) studied ab ove is exeptional, as it is
unusual for GMRES(1) to onverge exatly in three iterations. Remarkably, though,
GMRES(1) maintains sup eriority over GMRES(2) for a wide range of initial residuals.
For this matrix
A
, GMRES(2) onverges exatly in one yle for any initial residual
with zero in the third omp onent, so we restrit attention to residuals normalized to
the form
r
0
= (
 ;  ;
1)
T
. Figure 2 indiates that GMRES(2) makes little progress for
most suh residuals, while GMRES(1) onverges to high auray for the vast ma jor-
ity of these
r
0
values. The olor in eah plot reets the magnitude of
k
r
(
m
)
100
k
=
k
r
0
k
:
Blue indiates satisfatory onvergene, while red signals little progress in one hun-
dred iterations. (To ensure this data's delity, we performed these omputations in
both double and quadruple preision arithmeti; dierenes b etween the two were
negligible.)
To gain an appreiation for the dynamis behind Figure 2, we rst examine the
ation of a single GMRES(1) step. From (2.2) it is lear that GMRES(1) will om-
pletely stagnate only when
r
T
0
Ar
0
= 0. For the matrix
A
speied in (2.1) and
r
0
= (
 ;  ;
1)
T
, this ondition redues to

2
+
 
+

2
+

+ 3

+ 1 = 0
;
(2.3)
the equation for an oblique ellipse in the (
 ; 
) plane.
Now writing
r
(1)
k
= (
 ;  ;
1)
T
, onsider the map
r
(1)
k
7!
s
(1)
k
+1
that pro jets
r
(1)
k
+1
into the (
 ; 
) plane,
s
(1)
k
+1
= (
r
(1)
k
+1
)

1
3
0

(
r
(1)
k
+1
)
1
(
r
(1)
k
+1
)
2
1
A
;

4
MARK EMBREE

10

5 0 5 10

10

5
0
5
10

10

5 0 5 10

10

5
0
5
10

10

15
10

10
10

5
10
0
 
 
Fig. 2
.
Convergene of GMRES
(1) (
left
)
and GMRES
(2) (
right
)
for the matrix in
(2.1)
over
a range of initial residuals of the form
r
0
= (
 ; ;
1)
T
. The olor indiates
k
r
(
m
)
100
k
=
k
r
0
k
on a loga-
rithmi sale: blue regions orrespond to initial residuals that onverge satisfatorily, while the red
regions show residuals that stagnate or onverge very slow ly.
where (
r
(1)
k
+1
)
j
denotes the
j
th entry of
r
(1)
k
+1
, whih itself is derived from
r
(1)
k
via (2.2).
For the present example, we have
s
(1)
k
+1
=
0
B
B
B



3

4

2
+ 3
 
+ 9


4


1

2
+
 
+

+ 5

+ 10

3
+
 
2

3

2
+ 2

2

2
 

3

+


3

2
+
 
+

+ 5

+ 10
1
C
C
C
A
:
(2.4)
We an lassify the xed p oints (
 ; 
) satisfying (2.3) by investigating the Jaobian
of (2.4). One of its eigenvalues is always one, while the other eigenvalue varies ab ove
and b elow one in magnitude. In the left plot of Figure 2, we show the stable portion
of the ellipse (2.3) in blak and the unstable part in white.
We an similarly analyze GMRES(2). This iteration will never progress when, in
addition to the stagnation ondition for GMRES(1),
r
0
also satises
r
T
0
AAr
0
= 0.
For the present example, this requirement implies

2
+ 2
 
+

2
+ 5

+ 6

+ 1 = 0
;
the equation for an oblique parab ola. This urve intersets the ellipse (2.3) at two
points, drawn as dots in the right plot of Figure 2, the only stagnating residuals
(
 ;  ;
1)
T
for GMRES(2). We an analyze their stability as done above for GMRES(1).
The pro jeted map for this iteration,
r
(2)
k
7!
s
(2)
k
+2
, takes the form
s
(2)
k
+2
=
0
B
B

3

2

3

+ 4

+ 9



4

2

3

+ 4

+ 9
1
C
C
A
:
(2.5)
Analyzing the Jaobian for this GMRES(2) map at the pair of xed p oints, we nd one
to b e unstable (shown in blak in the right plot of Figure 2) while the other is stable
(shown in white). This stable xed p oint is an attrator for stagnating residuals.

RESTARTED GMRES
5
k
r
(
m
)
k
k
k
r
0
k
iteration,
k
GMRES(1)
GMRES(2)
0 5 10 15 20 25 30
10
0
10

5
10

10
10

15
Fig. 3
.
Convergene urves for GMRES
(1)
and GMRES
(2)
applied to
(3.1)
with
x
0
=
0
.
We return briey to the initial residual
r
0
= (2
;

4
;
1)
T
. After the rst few itera-
tions, the angle between
r
(2)
k
and the xed vetor steadily onverges to zero at the
rate 0
:
6452
: : :
suggested by the Jaobian's dominant eigenvalue. We onlude with
high ondene that GMRES(2) never onverges for this initial residual. (If one yle
of GMRES(
m
) pro dues a residual parallel to
r
0
, then either
r
(
m
)
m
=
r
0
or
r
(
m
)
m
=
0
.
Thus a residual an't remain xed in the nite (
 ; 
) plane, but still onverge to
0
.)
3. Seond Example.
The matrix
A
in (2.1) is nondiagonalizable, and one might
be tempted to blame its surprising onvergene behavior on this fat. To demonstrate
that nondiagonalizablity is not an essential requirement, we exhibit a diagonalizable
matrix with eigenvalues
f
1
;
2
;
3
g
for whih restarted GMRES also pro dues extreme
behavior. Take
A
=
0
B

1 2

2
0 2 4
0 0 3
1
C
A
;
b
=
0

3
1
1
1
A
;
(3.1)
with
x
0
=
0
. Again, we onstrut the rst few residuals. For GMRES(1),
r
(1)
1
=
0

2

1
0
1
A
;
r
(1)
2
=
0

2
0
0
1
A
;
r
(1)
3
=
0

0
0
0
1
A
;
while GMRES(2) yields
r
(2)
1
=
0

2

1
0
1
A
;
r
(2)
2
=
0

1
0

1
1
A
;
r
(2)
3
=
1
17
0

8
12

8
1
A
;
r
(2)
4
=
1
67
0


12
12

28
1
A
:
Figure 3 illustrates the onvergene urve for thirty iterations, again omputed using
exat arithmeti.

Frequently asked questions

Q1. What is the olor of a GMRES?

The olor indi ates kr(m)100 k=kr0k on a loga-rithmi s ale: blue regions orrespond to initial residuals that onverge satisfa torily, while the redregions show residuals that stagnate or onverge very slowly. 

Q2. What is the way to do this?

For the onve tion-di usion dis retization des ribed in [4℄,GMRES(1) or GMRES(5) an outperform GMRES(20) on moderately re ned grids.