On the exponential convergence of matching pursuits in quasi-incoherent dictionaries

TLDR: The main contribution is a detailed analysis of the approximation and stability properties of MP with quasi-incoherent dictionaries, and a bound on the number of steps sufficient to reach an error no larger than a penalization factor times the best m-term approximation error.

Abstract: The purpose of this correspondence is to extend results by Villemoes and Temlyakov about exponential convergence of Matching Pursuit (MP) with some structured dictionaries for "simple" functions in finite or infinite dimension. The results are based on an extension of Tropp's results about Orthogonal Matching Pursuit (OMP) in finite dimension, with the observation that it does not only work for OMP but also for MP. The main contribution is a detailed analysis of the approximation and stability properties of MP with quasi-incoherent dictionaries, and a bound on the number of steps sufficient to reach an error no larger than a penalization factor times the best m-term approximation error.

Full text

HAL Id: inria-00544945
https://hal.inria.fr/inria-00544945
Submitted on 11 Dec 2010
HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entic research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.
L’archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diusion de documents
scientiques de niveau recherche, publiés ou non,
émanant des établissements d’enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
On the exponential convergence of Matching Pursuits in
quasi-incoherent dictionaries
Rémi Gribonval, Pierre Vandergheynst
To cite this version:
Rémi Gribonval, Pierre Vandergheynst. On the exponential convergence of Matching Pursuits in
quasi-incoherent dictionaries. IEEE Transactions on Information Theory, Institute of Electrical and
Electronics Engineers, 2006, 52 (1), pp.255–261. �10.1109/TIT.2005.860474�. �inria-00544945�

IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 52, NO. 1, JANUARY 2006
255
[10] S. Natarajan, “Large deviations, hypothesis testing, and source coding
for finite Markov chains,” IEEE Trans. Inf. Theory, vol. IT-31, no. 3, pp.
360–365, May 1985.
[11] O. Shalvi and E. Weinstein, “New criteria for blind deconvolution of
minimum phase systems,” IEEE Trans. Inf. Theory, vol. 36, no. 2, pp.
312–321, Mar. 1990.
[12] S. M. Sowelam and A. H. Tewfik, “Waveform selection in radar target
classification,” IEEE Trans. Inf. Theory, vol. 46, no. 3, pp. 1014–1029,
May 2000.
[13] L. Tong, G. Xu, and T. Kailath, “Blind identification and equalization
based on second-order statistics: A time domain approach,” IEEE Trans.
Inf. Theory, vol. 40, no. 2, pp. 340–349, Mar. 1994.
[14] H. L. Van Trees, Detection, Estimation, and Modulation Theory.New
York: Wiley, 1968, pt. 1.
On the Exponential Convergence of Matching Pursuits in
Quasi-Incoherent Dictionaries
Rémi Gribonval, Member, IEEE, and
Pierre Vandergheynst, Member, IEEE
Abstract—The purpose of this correspondence is to extend results by
Villemoes and Temlyakov about exponential convergence of Matching Pur-
suit (MP) with some structured dictionaries for “simple” functions in finite
or infinite dimension. The results are based on an extension of Tropp’s re-
sults about Orthogonal Matching Pursuit (OMP) in finite dimension, with
the observation that it does not only work for OMP but also for MP. The
main contribution is a detailed analysis of the approximation and stability
properties of MP with quasi-incoherent dictionaries, and a bound on the
number of steps sufficient to reach an error no larger than a penalization
factor times the best
-term approximation error.
Index Terms—Dictionary, greedy algorithm, matching pursuit (MP),
nonlinear approximation, sparse representation.
I. INTRODUCTION
In a Hilbert space
H
of finite or infinite dimension, we consider the
problem of getting
m
-term approximants of a function
f
from a pos-
sibly redundant dictionary
D
=
f
g
k
;k
2 g
of unit norm basis func-
tions also called atoms. It will often be convenient to see a dictionary
as a synthesis operator (or, in finite dimension, as a matrix)
D
:
cc
c
=(
c
k
)
7!
D
cc
c
=
k
c
k
g
k
that maps sequences to vectors in
H
. A special class of dictionaries that
is widely used in signal and image processing is the family of frames:
a dictionary
D
is a frame for
H
if, and only if,
D
is a bounded oper-
ator from
`
2
onto
H
[2]. However, in this correspondence, we consider
dictionaries that may not be frames, hence
D
shall be defined essen-
tially on sequences
cc
c
with a finite number of nonzero entries. For any
Manuscript received April 9, 2004; revised October 13, 2005.
R. Gribonval is with IRISA-INRIA, Campus de Beaulieu, 35042 Rennes
Cedex, France (e-mail: remi.gribonval@irisa.fr).
P. Vandergheynst is with the Signal Processing Institute, the Swiss Federal
Institute of Technology (EPFL), CH-1015 Lausanne, Switzerland (e-mail:
pierre.vandergheynst@epfl.ch).
Communicated by G. Battail, Associate Editor At Large.
Digital Object Identifier 10.1109/TIT.2005.860474
index set
I
(not necessarily finite) we will also consider the restricted
synthesis operator
D
I
:
cc
c
7!
D
I
cc
c
=
k
2
I
c
k
g
k
that corresponds to the subset
D
I
=
f
g
k
;k
2
I
g
of the full dictionary.
When
D
is an orthonormal basis for
H
, it is well known how to get
the best
m
-term approximant to any
f
: the solution is to keep the
m
atoms of the basis which have the largest inner products
jh
f;g
k
ij
with
f
. However, for arbitrary redundant dictionaries, the problem becomes
NP-hard [3]. In the recent years, much effort has been made to under-
stand what structure should be imposed on
f
(for a given dictionary)
or on the dictionary itself so that good approximants can be obtained
with computationally feasible algorithms.
One of the first algorithms that appeared in the signal processing
community for approximating signals from a redundant dictionary was
the Matching Pursuit (MP) algorithm of Mallat and Zhang [25], which
iteratively decomposes the analyzed function
f
into an
m
-term approx-
imant
f
m
=
m
n
=1

n
g
k
and a residual
r
m
=
f
0
f
m
. MP is also
known as Projection Pursuit in the statistics community [10], [22] and
as a Pure Greedy Algorithm [27] in the approximation community. In
finite dimension, MP is known to converge exponentially, i.e., for some
0
<<
1
k
r
m
k
2
=
k
f
m
0
f
k
2


m
1k
f
k
2
;m

1
:
In infinite-dimensional Hilbert spaces, Jones [24] proved that MP is
still convergent, i.e.,
k
f
m
0
f
k!
0
, but gave no estimate of the
speed of convergence. DeVore and Temlyakov [4] exhibited a “bad”
dictionary
D
where there exists a “simple” function (sum of two dictio-
nary elements) for which MP gives “bad” approximations (i.e., with a
slow convergence
k
f
m
0
f
k
Cm
0
1
=
2
). On the positive side, Ville-
moes [30] showed that for Walsh wavelet packets, MP on “simple”
functions (
f
=
c
i
g
i
+
c
j
g
j
any sum of any two wavelet packets)
was exponentially convergent (just as MP in finite dimension) with
k
f
m
0
f
k
2

(3
=
4)
m
k
f
k
2
. Temlyakov obtained similar results [26]:
in particular, for
f
a function on the erval
[0
;
1)
taking constant values
on a partition of
[0
;
1)
into
n
disjoint intervals, and
D
a highly redun-
dant dictionary containing all (normalized) characteristic functions of
intervals
I

[0
;
1)
k
f
m
0
f
k
2

(1
0
1
=n
)
m=
2
k
f
k
2
:
In this correspondence, we extend Villemoes and Temlyakov results
about MP to more general dictionaries and “simple functions,” as stated
in the following featured theorem.
Featured Theorem 1: Let
D
be a dictionary in a finite- or infinite-
dimensional Hilbert space and
I
an index set such that the stability
condition (SC)

(
I
):=sup
k=
2
I
k
(
D
I
)
y
g
k
k
1
<
1
(1)
is met, where
(
1
)
y
denotes pseudoinversion.
1
Then, for any
f
=
k
2
I
c
k
g
k
2
span(
g
k
;k
2
I
)
,MP
1) picks up only correct atoms at each step:
(
8
n; k
n
2
I
)
;
2) if
I
is a finite set, then the residual
r
m
converges exponentially
to zero.
The stability condition (1) may look fairly abstract, but for so-called
quasi-incoherent dictionaries, one can obtain more explicit sufficient
conditions [28]. For such dictionaries, we derive estimates of the rate of
1
Basic reminders on pseudoinversion are given in Section II-E
0018-9448/$20.00 © 2006 IEEE

256 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 52, NO. 1, JANUARY 2006
exponential convergence of MP, and we obtain the following featured
theorem.
Featured Theorem 2: Let
D
be a dictionary in a finite- or infinite-di-
mensional Hilbert space and let

:max
k
6
=
l
jh
g
k
;g
l
ij
be its coherence.
For any finite index set
I
of size card
(
I
)=
m<
(1 + 1
=
)
=
2
and
any
f
=
k
2
I
c
k
g
k
2
span (
g
k
;k
2
I
)
,MP
1) picks up only correct atoms at each step:
(
8
n; k
n
2
I
)
;
2) converges exponentially
k
f
n
0
f
k
2

((1
0
1
=m
)(1 +

))
n
k
f
k
2
:
The proof of this theorem is based on an argument given by Tropp
[28] where the condition (1) is called “Exact Recovery Condition”
(ERC) because it ensures that Orthonormal Matching Pursuit (OMP)
and Basis Pursuit (BP) exactly recover any
f
=
k
2
I
c
k
g
k
2
span (
g
k
;k
2
I
)
:
We have chosen to rename the ERC a “stability condition.” Indeed, for
MP one cannot strictly speak about recovery, however, the theorem is
definitely a stability result since all residuals remain in the subspace
span
(
g
k
;k
2
I
)
H
. Tropp’s result was the last of a series of
“recovery” results: first with the BP “algorithm”—which was intro-
duced [1] as an alternative to MP since the latter cannot resolve close
atoms—under some assumptions on both the analyzed function and the
dictionary [6]–[9], [20], [19]; then with variants of the MP [12], [13].
After the first draft of this manuscript was submitted for publication, it
came to our attention that Donoho, Elad, and Temlyakov also consider
stability and recovery properties of MP in incoherent dictionaries [5].
We discuss in more details in Section V how our results are connected
to other approaches.
The previous theorems only explain the behavior of MP on exact ex-
pansions, i.e., they require that the approximated function
f
be exactly
expressed as an expansion from a “good” set of atoms. However, real
signals or images almost never have such a simple expansion in prac-
tical dictionaries. Fortunately, just as for OMP [28], the analysis of MP
as an approximation algorithm can be carried out by taking into account
how well a function is approximated by an expansion from a good set
of atoms. In particular, our results lead to the following theorem (with
the notations of Featured Theorem 2)
Featured Theorem 3: Let
f
f
n
g
be a sequence of approximants to
f
2H
produced with MP with
g
k
the corresponding atoms. Let
m<
(1+1
=
)
=
4
and let
f
?
m
=
k
2
I
c
k
g
k
be a best
m
-term approximant
to
f
from
D
, i.e.,
k
f
?
m
0
f
k
=

m
(
f
):=inf
fk
f
0
D
I
cc
c
k
;
card(
I
)

m;cc
c
2
I
g
:
Then, there is a number
N
m
such that
1) the error after
N
m
steps of MP satisfies
k
f
N
0
f
k
p
1+4
m
m
;
2) during the first
N
m
steps, MP picks up atoms from the best
m
-term approximant:
k
n
2
I
?
m
;
3) if

2
m
<
3

2
1
=m
then
N
m
is no larger than
N
m

2+
m
1
4
3
1
ln
3

2
1
m
2
m
:
In the course of this correspondence, we actually prove slightly more
general results (Theorems 1–4) and particularize them to get our fea-
tured results (Featured Theorems 1 and 3). The structure of this corre-
spondence is as follows. In Section II, we recall the definition of MP
and several variants thereof, and prove the stability result (Featured
Theorem 1). In Section III, we particularize this result to a special class
of dictionaries, quasi-incoherent dictionaries. This allows us to obtain
constraints on the dictionary so that the stability condition is met and
we also give estimates on the rate of convergence of MP in these cases
(Featured Theorem 2). Finally, in Section IV, we explore the approxi-
mation properties of various flavors of MP. In particular, we show that
greedy algorithms may robustly select atoms participating in a near best
m
-term approximation and give the resulting approximation bounds
(Featured Theorem 3).
The proof of Featured Theorem 1 is merely a rewriting of Tropp’s
proof with the observation that it does not only work for OMP but
also for MP. Thus, the main contribution of this correspondence is in
the study of the approximation and stability properties of greedy algo-
rithms with quasi-incoherent dictionaries.
II. M
ATCHING PURSUIT(S) ON “SIMPLE”EXPANSIONS
In this section, we first recall the definition of MP and several vari-
ants thereof, then we prove the stability of all these variants, in the sense
of Featured Theorem 1.
A. Matching Pursuit (MP)
MP is an iterative algorithm that builds
n
-term approximants
f
m
and
residuals
r
n
=
f
0
f
n
by adding one term at a time in the approximant.
It works as follows. At the beginning, we set
f
0
=0
and
r
0
=
f
;
assuming
f
n
and
r
n
are defined, we set
jh
r
n
;g
k
ij
=sup
k
jh
r
n
;g
k
ij
(2)
f
n
+1
=
f
n
+
h
r
n
;g
k
i
g
k
(3)
and compute a new residual as
r
n
+1
=
f
0
f
n
+1
.
B. Weak MPs
When the dictionary is infinite, the supremum in (2) may not be at-
tained, so one may have to consider the so-called weak selection rule
jh
r
n
;g
k
ij 

sup
k
jh
r
n
g
k
ij
(4)
with some fixed
0
<

1
independent of
n
. Corresponding variants
of MP will be called Weak MP with weakness parameter

, or in short
Weak
(

)
MP or even Weak MP when the value of

does not need to
be specified.
C. Orthonormal MP
Moreover, once
m
atoms have been selected, the approximant
f
m
=
m
0
1
n
=0
h
r
n
;g
k
i
g
k
is generally not the best approximant to
f
from the finite-dimensional
subspace
V
m
:= span(
g
k
;
...
;g
k
)
. OMP, respectively, Weak
(

)
OMP, replaces the update rule (3) with
f
n
+1
=
P
V
f
(5)
where
P
V
is the orthonormal projector onto the finite dimensional sub-
space
V
.
D. General MPs
More generally, one can consider the family of approximation algo-
rithms based on the repeated application of two steps:
1) a (weak) selection step according to (4);
2) an update step where a new approximant
f
n
+1
2V
n
+1
is
chosen.
Algorithms from this larger family will be called General MP, Weak
(

)
General MP, or Weak General MP. Examples of Weak
(

)
General

IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 52, NO. 1, JANUARY 2006
257
MP algorithms include the High Resolution Pursuits [23], [16], which
were introduced to attenuate the lack of resolution of plain MP with
time–frequency dictionaries in the time domain.
E. Stability of Weak
(

)
General MP
The major result of Tropp [28] is what he calls the “Exact Recovery
Condition”

(
I
):=sup
k=
2
I
k
(
D
I
)
y
g
k
k
1
<
(6)
(where
(
1
)
y
denotes pseudoinversion, see below): when the Exact Re-
covery Condition is met, Weak
(

)
OMP “exactly recovers” any linear
combinations of atoms from the subdictionary
D
I
, which means that
Weak
(

)
OMP can only pick up correct atoms at each step. Tropp’s
proof indeed works for Weak
(

)
General MP, with the only difference
that we no longer get exact recovery but only stability of the Pursuit,
as stated in the following theorem.
Theorem 1: Let
I
be an index set (finite or infinite) with

(
I
)
<
1
.
For any
f
=
k
2
I
c
k
g
k
and
>
(
I
)
, Weak
(

)
General MP picks
up a correct atom at each step, i.e., for all
n

1
;k
n
2
I
.
Note that we do not assume that the elements
f
g
k
g
k
2
I
are linearly
independent for the result to hold true. Before giving the proof of the
theorem, let us give a quick reminder on the notion of pseudoinverse.
Most of this material can be found in the usual suspects [14], [21].
Let
A
be a linear operator and let Range
A
be its range. The pseu-
doinverse
A
y
is the left inverse that is zero on
f
Range
A
g
?
. It is also
the left inverse of minimal sup norm. In the case of general
p
by
q
ma-
trices, we will make use of the Moore–Penrose pseudoinverse. It is the
unique
q
by
p
matrix that satisfies the following properties:
AA
y
A
=
A
A
y
AA
y
=
A
y
(
AA
y
)
3
=
AA
y
and
(
A
y
A
)
3
=
A
y
A
where
(
1
)
3
denotes the adjoint. In particular,
AA
y
is an orthonormal
projection onto Range
A
. If the inverse of
A
3
A
exists, the Moore–Pen-
rose pseudoinverse can simply be written
A
y
=(
A
3
A
)
0
1
A
3
:
Proof of Theorem 1: Just as the proof of exactness of OMP by
Tropp (which is a special case), we can show by induction that at each
step MP picks up an atom
k
n
2
I
, so the residual
r
n
remains in the
finite-dimensional space
V
I
= span(
g
k
;k
2
I
)
. Initially, we have by
assumption
r
0
=
f
2V
I
. Assuming that
r
n
2V
I
, we notice that the
inner products
fh
r
n
;g
k
g
k
2
I
i
between
r
n
and
f
g
k
;k
2
I
g
are listed
in the vector
D
?
I
r
n
while those with
f
g
k
;k=
2
I
g
are listed in
D
?

I
r
n
where
I
=
f
k;k
62
I
g
. Thus, the atom
g
k
is a correct one (i.e.,
k
n
+1
2
I
) if and only if

(
I; r
n
):=
k
D
?
I
r
n
k
1
k
D
?
I
r
n
k
1
<:
The core of of the proof of [28, Theorem 3.1] yields

(
I; r
n
)


(
I
)
.
From the assumption

(
I
)
<
, we can infer that
k
n
+1
2
I
and
r
n
+1
2V
I
, and we get the theorem.
F. Recovery and Convergence
Suppose that the analyzed function
f
belongs to
span(
g
k
;k
2
I
)
where
I
satisfies

(
I
)
<
1
, and that we perform some Weak
(

)
Gen-
eral MP with
>
(
I
)
: Theorem 1 states that the Pursuit will only
pick up correct atoms.
In the particular case of an Orthogonal Pursuit, since each residual
r
n
is orthogonal to previously selected atoms
g
k
;
...
g
k
, any atom
can only be picked up once by the Pursuit. As a result, if in addition
I
is a finite set of cardinality
m
, the Orthogonal Pursuit exactly recovers
f
in
m
iterations: this is the main result formalized by Tropp and al-
ready present—though not with such a clear statement—in the results
of Gilbert et al. [12], [13].
If the Pursuit we are performing on
f
is not orthogonal, it is known
that convergence does not generally occur in a finite number of steps.
However, if
I
is a finite set, the stability condition implies that the Pur-
suit is actually performed in the finite-dimensional space
V
I
. In the
case of Weak MP, it follows [25] that we have exponential convergence,
just as stated in Featured Theorem 1. In the next section, we provide
some tools to estimate the rate of this convergence, and it will turn out
that they also make it possible to estimate the speed of convergence of
(Weak) OMP.
III. MP
IN QUASI-INCOHERENT DICTIONARIES
In the previous section, we have given fairly abstract conditions to
ensure stability of Weak General MP, exact recovery with Weak OMP,
and exponential convergence of Weak MP toward the approximated
function. However, the quantity

(
I
)
that appears in the stability con-
dition (6) is not very explicit, and we did not yet provide estimates for
the rate of exponential convergence.
In this section, we will show that we can use the so-called cumulative
coherence function
2
of the dictionary to estimate

(
I
)
—and check the
Stability Condition—as well as the rate of exponential convergence of
Plain MP.
A. Cumulative Coherence Function and Coherence
Definition 1: Let
D
be a dictionary. Its cumulative coherence func-
tion is defined for each integer
m

1
as

1
(
m
):= max
I
j
card(
I
)

m
max
k=
2
I
l
2
I
jh
g
l
;g
k
ij
:
(7)
As a special case, for
m
=1
, the value of the cumulative coherence
function is the so-called coherence of the dictionary

=

1
(1) = max
k
6
=
l
jh
g
l
;g
k
ij
:
(8)
One easily observes that the cumulative coherence function is subad-
ditive

1
(
k
+
l
)


1
(
k
)+

1
(
l
)
;
8
k;l
hence, we have

1
(
m
)


1
m; m

1
. A dictionary is called in-
coherent if

is small: typically, in finite dimension
N
, any dictionary
that strictly contains an orthonormal basis has coherence


1
=
p
N
.
The union of the Dirac and the Fourier bases is an incoherent dictionary
where indeed

=1
=
p
N
and

1
(
m
)=

1
m
. When the cumulative
coherence function grows no faster than

1
m
, we say that the dictio-
nary is quasi-incoherent.
B. Explicit Stability Condition and Rate of Convergence
Using Neumann series, Tropp proved that whenever
I
is of size
m
such that

1
(
m
0
1)
<
1
, we have the upper bound

(
I
)


1
(
m
)
1
0

1
(
m
0
1)
:
(9)
From this estimate, we can derive the following theorem which shows
that the cumulative coherence function

1
can provide both a practical
Stability Condition for Weak General MP and an estimate of the rate
of exponential convergence for Weak MP.
2
Formerly known as the Babel function.

258 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 52, NO. 1, JANUARY 2006
Theorem 2: Let
m
be an integer such that

1
(
m
)+

1
(
m
0
1)
<
1
:
(10)
Then for any index set
I
of size at most
m
,any
f
2
span (
g
k
;k
2
I
)
,
and
>
1
(
m
)
=
(1
0

1
(
m
0
1))
.
1) Weak
(

)
General MP picks up a correct atom at each step, i.e.,
for all
n

1
;k
n
2
I
.
2) Weak
(

)
MP/OMP converge exponentially to
f
: more pre-
cisely, we have
k
f
0
f
n
k
2

(

m
(

))
n
1k
f
k
2
with

m
(

):=1
0

2
(1
0

1
(
m
0
1))
=m:
(11)
Before we prove the theorem, we need a few lemmas.
Lemma 1: For any index set
I
with card
(
I
)=
m
, the squared
singular values of
D
I
exceed
1
0

1
(
m
0
1)
.
The proof relies on Gershgorin Disc Theorem and can be found in
[12], [6], [15], [28], see for example [28, Lemma 2.3]. The second im-
portant lemma is due to DeVore and Temlyakov [4]; it gives a lower
estimate on the amount of energy of a signal that can be removed in
one step of MP.
Lemma 2 (DeVore,Temlyakov): For any
I
and
cc
c
sup
k
2
I
jh
D
I
cc
c
;g
k
ij 
k
D
I
cc
c
k
2
k
cc
c
k
1
:
We can now prove Theorem 2.
Proof of Theorem 2: The stability result is trivial using the esti-
mate (9) together with Theorem 1. Let us proceed with the exponential
convergence of Weak
(

)
MP/OMP. From the stability part we know
that at each step the residual
r
n
=
f
0
f
n
of Weak
(

)
MP/OMP is in
V
I
. Thus,
r
n
=
D
I
cc
c
n
for some sequence
cc
c
n
with at most
m
nonzero
elements. Denoting

, the smallest nonzero singular value of
D
I
,it
follows using Lemma 1 that
k
cc
c
n
k
2
1

m
k
cc
c
n
k
2
2

m

2
k
D
I
cc
c
n
k
2
2

m
1
0

1
(
m
0
1)
k
r
n
k
2
:
Then, by Lemma 2, we obtain
sup
k
2
I
jh
r
n
;g
k
ij 
k
r
n
k
2
k
cc
c
n
k
1
k
r
n
k
1
0

1
(
m
0
1)
m
:
We conclude by noticing that
k
r
n
+1
k
2
(a)
k
r
n
k
2
0jh
r
n
;g
k
ij
2
k
r
n
k
2
0

2
sup
k
jh
r
n
;g
k
ij
2

1
0

2
(1
0

1
(
m
0
1))
=m
1k
r
n
k
2


m
(

)
1k
r
n
k
2
111
(

m
(

))
n
+1
k
r
0
k
2
=(

m
(

))
n
+1
k
f
k
2
:
Notice that
(a)
is an equality for MP and an inequality for OMP.
The above estimate is valid for the whole range of admissible weak-
ness parameter

:

=1
corresponds to the standard “full search” Pur-
suit while

=

1
(
m
)
=
(1
0

1
(
m
0
1))
gives the worst case estimate
corresponding to the limiting case of the weakest allowable Pursuit. To
avoid carrying unnecessary heavy notations throughout the rest of the
correspondence, from now on we will only consider the case of a full
search Pursuit.
C. Estimates Based on the Coherence
For any dictionary, we have seen that the cumulative coherence func-
tion can be bounded using the coherence as

1
(
m
)


1
m; m

1
.
Thus, a sufficient condition to get the stability condition (10) with the
cumulative coherence function becomes a condition based on the co-
herence:
m<
1
2
1+
1

:
(12)
If the dictionary is a union of incoherent orthonormal bases in finite
dimension
N
[20], then indeed

1
(
m
)=

1
m
for
1

m

N
and (12) is equivalent to (10). In any case, the rate

m
=

m
(1)
of
exponential convergence of a (full search) MP is estimated from above
by

m
:=

m
(1) = 1
0
(1
0

1
(
m
0
1))
=m

(1
0
1
=m
)(1+

)
:
(13)
The combination of (3) with Theorem 2 yields our Featured Theorem 2.
IV. MP
AS AN APPROXIMATION ALGORITHM
So far we have considered the behavior of (Weak) MP on exact sparse
expansions in the dictionary. However, the set of functions with an
exact sparse expansion
f
2
Range
D
I
;
card
(
I
)
<
dim
H
is negli-
gible, hence, it is more interesting to know what is the behavior of Pur-
suits on more general vectors, typically on
f
“close enough” to some
f
?
with an exact sparse expansion.
A. Best
M
-Term Approximation
For any
f
2H
and
m
, the error of best
m
-term approximation to
f
from the dictionary is

m
(
f
):=inf
fk
f
0
D
I
cc
c
k
;
card(
I
)

m; c
k
2 g
:
(14)
When there is no ambiguity about which
f
is considered, we will
simply write

m
.For
f
2H
, let
f
?
m
=
k
2
I
c
k
g
k
be a best
m
-term
approximation to
f
, i.e., with card
(
I
m
)

m
and
k
f
0
f
?
m
k
=

m
.
If a best
m
-term approximant does not exist (because the infimum
in the definition of

m
is not reached), one can consider a near
best
m
-term approximant by letting
>
0
and only requiring
k
f
0
f
?
m
k
=(1+

)

m
. In any case, without loss of generality, we
can assume that
1) the atoms
f
g
k
;k
2
I
m
g
are linearly independent;
2)
f
?
m
is the orthogonal projection of
f
onto span
(
g
k
;k
2
I
m
)
;
else, we could easily replace
f
?
m
with a better
m
-term approximant to
f
by either changing the coefficients
c
k
or selecting a subset
I

I
m
corresponding to linearly independent atoms with
span(
g
k
;k
2
I
) = span(
g
k
;k
2
I
m
)
:
B. Robustness Theorem
From Theorem 1, we know that if
I
m
satisfies the stability condition,
then General MP performed on
f
?
m
is stable. The following theorem is
a robustness result which shows that if
f
is “close enough” to
f
?
m
, the
atoms selected during “the first iterations” of a Pursuit will coincide
with those which would be selected by a Pursuit on
f
?
m
, which can be
considered as the correct ones.

Frequently asked questions

Q1. What is the stability condition for a Pursuit?

if The authoris a finite set, the stability condition implies that the Pursuit is actually performed in the finite-dimensional space VI . 

Q2. What is the significance of the correspondence?

This correspondence provides insights that one of the most widely used heuristics, the Matching Pursuit algorithm, is stable, and offers good approximation properties when the dictionary is sufficiently incoherent. 

Q3. What is the main result of the Featured Theorem?

The major result of Tropp [28] is what he calls the “Exact Recovery Condition”(I) := sup k=2Ik(DI)ygkk1 < (6)(where ( )y denotes pseudoinversion, see below): when the Exact Recovery Condition is met, Weak ( ) OMP “exactly recovers” any linear combinations of atoms from the subdictionary DI , which means that Weak ( ) OMP can only pick up correct atoms at each step. 

Q4. What is the bestm-term approximation to f?

For any f 2 H andm, the error of bestm-term approximation to f from the dictionary ism(f) := inffkf DIck; card(I) m; ck 2 g: (14)When there is no ambiguity about which f is considered, the authors will simply write m. 

Q5. What is the approximant to f from the finite-dimensional subspace?

once m atoms have been selected, the approximantfm =m 1n=0hrn; gk igkis generally not the best approximant to f from the finite-dimensional subspace Vm := span(gk ; . . . ; gk ).