Journal ArticleDOI

# A curse-of-dimensionality-free numerical method for a class of HJB PDE's

01 Jan 2005-IFAC Proceedings Volumes (Elsevier)-Vol. 38, Iss: 1, pp 532-537

AbstractMax-plus methods have been explored for solution of first-order, nonlinear Hamilton-Jacobi-Bellman partial differential equations (HJB PDEs) and corresponding nonlinear control problems. These methods exploit the max-plus linearity of the associated semigroups. Although these methods provide advantages, they still suffer from the curse-of-dimensionality. Here we consider HJB PDEs where the Hamiltonian takes the form of a (pointwise) maximum of linear/quadratic forms. We obtain a numerical method not subject to the curse-of-dimensionality. The method is based on construction of the dual-space semigroup corresponding to the HJB PDE. This dual-space semigroup is constructed from the dual-space semigroups corresponding to the constituent Hamiltonians.

Topics: Semigroup (57%), Pointwise (51%), Nonlinear system (50%)

### 1. INTRODUCTION

• One approach to nonlinear control is through Dynamic Programming (DP).
• Various approaches have been taken to solution of the HJB PDE.
• If the state dimension is n, then one has 100n grid points.
• They employ a max-plus basis function expansion of the solution, and the numerical methods obtain the coefficients in the basis expansion.

### 2. REVIEW OF THEORY

• Note that due to space limitations, the proofs of the results cannot be included here.
• As indicated above, the authors suppose the individual Hm are linear/quadratic Hamiltonians.
• These assumptions guarantee the existence of the V m as locally bounded functions which are zero at the origin (cf. (McEneaney, 1998)).
• Pm is the smallest symmetric, positive definite solution of (6) The method the authors will use to obtain these value functions/HJB PDE solutions will be through the associated semigroups.

### 3. MAX-PLUS DUAL OPERATORS

• LetSβ = Sβ(IRn) be the set of functions mapping IRn into IR which are uniformly semiconvex with constant β.
• The following semiconvex duality result (Fleming and McEneaney, 2000), (McEneaney, 2003) requires only a small modification of convex duality and Legendre/Fenchel transform results (c.f. (Rockafellar and Wets, 1997)).
• Semiconcavity is the obvious analogue of semiconvexity.
• In particular, one has the following Theorem 3.8.

### 4. DISCRETE TIME APPROXIMATION

• The method developed here will not involve any discretization over space nor any basis functions.
• Of course this is obvious since otherwise one could not avoid the curse-of-dimensionality.
• The discretization will be over time, where approximate µ processes will be constant over the length of each time-step.
• The corresponding max-plus linear operator is B̂τ = ⊕ m∈M B̂mτ .
• The following result on propagation of the semiconvex dual will also come in handy.

### 5. ALGORITHM AND EXAMPLES

• Due to space limitations, the authors cannot give the steps in the actual algorithm that is generated by the above theory.
• Further, the computational cost growth in space dimension n is limited to cubic growth.
• The quoted computational times were obtained with a standard 2001 PC.
• The backsubstitution error has been scaled by dividing by |x|2 + 10−5.

Did you find this useful? Give us your feedback

Content maybe subject to copyright    Report

A CURSE-OF-DIMENSIONALITY-FREE
NUMERICAL METHOD FOR A CLASS OF HJB
PDE’S
William M. McEneaney
,1
Dept. of Mech. and Aero. Eng. and Dept. of Math.,
University of California San Diego, wmceneaney@ucsd.edu
Abstract: Max-plus methods have been explored for solution of ﬁrst-order, nonlin-
ear Hamilton-Jacobi-Bellman partial diﬀerential equations (HJB PDEs) and corre-
sponding nonlinear control problems. These methods exploit the max-plus linearity
of the associated semigroups. Although these methods provide advantages, they
still suﬀer from the curse-of-dimensionality. Here we consider HJB PDEs where
the Hamiltonian takes the form of a (pointwise) maximum of linear/quadratic
forms. We obtain a numerical method not subject to the curse-of-dimensionality.
The method is based on construction of the dual-space semigroup corresponding
to the HJB PDE. This dual-space semigroup is constructed from the dual-space
semigroups corresponding to the constituent Hamiltonians. Copyright 2005 IFAC.
Keywords: partial diﬀerential equations, curse-of-dimensionality, dynamic
programming, max-plus algebra, Hamilton-Jacobi-Bellman equations
1. INTRODUCTION
One approach to nonlinear control is through
Dynamic Programming (DP). With DP, solution
of the control problem “reduces” to solution of the
corresponding partial diﬀerential equation (PDE).
In the case of Deterministic Optimal Control or
Deterministic Games (such as H
control) where
one player’s feedback is prespeciﬁed, the PDE
is a Hamilton-Jacobi-Bellman (HJB) PDE. The
diﬃculty is that one must solve the HJB PDE.
Various approaches have been taken to solution of
the HJB PDE. The most common methods by far
all fall into the class of ﬁnite element methods (cf.
(Bardi and Capuzzo-Dolcetta, 1997), (Dupuis and
Bou´e, 1999), among many others). These require
that one generate a grid over some bounded region
of the state-space. Suppose the region over which
1
Research supported by NSF grant DMS-0307229. The
author also thanks Prof. J. William Helton for helpful
discussions.
one constructs the grid is rectangular. Suppose
one uses 100 grid points per dimension. If the
state dimension is n, then one has 100
n
grid
points. Thus the computations grow exponentially
in state-space dimension n.
In recent years, an entirely new class of numer-
ical methods for HJB PDEs has emerged (c.f.
(Fleming and McEneaney, 2000), (McEneaney,
2003), (McEneaney, 2004), (Akian, Gaubert and
Lakhouat, 2004)). These metho ds exploit the
max-plus linearity of the associated semigroup.
They employ a max-plus basis function expansion
of the solution, and the numerical methods ob-
tain the coeﬃcients in the basis expansion. Much
of the work has concentrated on the (harder)
steady-state HJB PDE class. With the max-plus
methods, the numb er of basis functions required
still typically grows exponentially with space di-
mension. For instance, one might use 25 basis
functions per space dimension to cover a rectan-
gular region well. Consequently, one still has the

curse-of-dimensionality. Even with the max-plus
approach, one cannot expect to solve problems
of more than say dimension 4 or 5 on current
machinery.
This paper discusses an approach to certain
nonlinear HJB PDEs which is not subject to
the curse-of-dimensionality. In fact, the compu-
tational growth in state-space dimension is on
the order of n
3
. However, there is exponential
computational growth in a certain measure of
complexity of the Hamiltonian. Under this mea-
sure, the minimal complexity Hamiltonian is the
solution by a Riccati equation. If the Hamiltonian
is given as a maximum of M linear/quadratic
Hamiltonians, then one could say the complexity
of the Hamiltonian is M.
The approach has been applied on some sim-
ple nonlinear problems. A few simple examples
comprised of 3 linear/quadratic components were
solved in 10-20 seconds over R
3
and 10-45 seconds
over R
4
. For these particular problems, the solu-
tion was obtained over the entire space with the
resulting errors in the gradients growing linearly
in |x|. (See Section 5 for speciﬁc examples.) These
speeds are of course unprecedented. This co de was
not optimized. Further, the computational growth
in going from n = 4 up to say n = 6 would be on
the order of 6
3
/4
3
' 4 as opposed to say more
than 10
4
for a ﬁnite element method.
We will consider HJB PDEs given as
0=
e
H(x, V ) = max
m∈{1,2,...,M}
{H
m
(x, V )} (1)
with boundary data V (0) = 0 (V being zero at the
origin). In order to make the problem tractable,
we will concentrate on a single class of HJB PDEs
of form (1). However, the theory can obviously be
expanded to a much larger class.
2. REVIEW OF THEORY
Note that due to space limitations, the proofs of
the results cannot be included here.
As indicated above, we suppose the individual H
m
consider a ﬁnite set of linear systems
˙
ξ
m
= A
m
ξ
m
+ σ
m
w, ξ
m
0
= x IR
n
. (2)
Let w ∈W
.
=L
loc
2
([0, ); IR
m
). Let the cost
functionals and value functions be
J
m
(x, T ; w)
.
=
T
Z
0
1
2
ξ
m
t
D
m
ξ
m
t
γ
2
2
|w
t
|
2
dt, (3)
V
m
(x) = lim
T →∞
sup
w∈W
J
m
(x, T ; w). (4)
Obviously J
m
and V
m
require some assumptions
in order to guarantee their existence.
Assume that there exists c
A
(0, )
such that x
T
A
m
x ≤−c
A
|x|
2
for all
x IR
n
and m ∈M. Assume that
there exists c
σ
< such that |σ
m
|≤
c
σ
m∈M. Assume that all D
m
are
positive deﬁnite, symmetric, and let c
D
be such that x
T
D
m
x c
D
|x|
2
for all
x IR
n
and m ∈M. Lastly, assume that
γ
2
/c
2
σ
>c
D
/c
2
A
.
(A.m)
These assumptions guarantee the existence of the
V
m
as locally bounded functions which are zero
at the origin (cf. (McEneaney, 1998)).
The corresponding HJB PDEs are
0=H
m
(x, V )
=
1
2
x
T
D
m
x +(A
m
x)
T
V +
1
2
V
T
Σ
m
V
V(0) = 0
(5)
where Σ
m
.
=
1
γ
2
σ
m
(σ
m
)
T
. Let G
δ
be the subset
of C(IR
n
) such that 0 V (x)
c
A
(γδ)
2
c
2
σ
|x|
2
.
For m ∈M, let P
m
satisfy the algebraic Riccati
equations
0=(A
m
)
T
P
m
+P
m
A
m
+D
m
+P
m
Σ
m
P
m
.(6)
Theorem 2.1. Each value function (4) is the
unique classical solution of its corresponding HJB
PDE (5) in the class G
δ
for suﬃciently small δ>0.
Further, V
m
(x)=
1
2
x
T
P
m
xwhere P
m
is the
smallest symmetric, positive deﬁnite solution of
(6) In particular, there exists symmetric, positive
deﬁnite
C such that V
m
(x)
1
2
x
T
Cx is convex for
all m ∈M.
The method we will use to obtain these value
functions/HJB PDE solutions will be through
the associated semigroups. For each m deﬁne the
semigroup
S
m
T
[φ]
.
=sup
w∈W
T
Z
0
1
2
(ξ
m
t
)
T
D
m
ξ
m
t
γ
2
2
|w
t
|
2
dt+φ(ξ
m
T
)
where ξ
m
satisﬁes (2). By (McEneaney, 1998), the
domain of S
m
T
includes G
δ
for all δ>0.
Theorem 2.2. Fix any T>0. Each value function,
V
m
, is the unique smooth solution of V = S
m
T
[V ]
in the class G
δ
for suﬃciently small δ>0. Further,
given any V ∈G
δ
, lim
T →∞
S
m
T
[V ](x)=V
m
(x)
(uniformly on compact sets).
Recall that the HJB PDE of interest is (1) with
H
m
given by (5). The corresponding value func-
tion is

e
V (x) = sup
w∈W
sup
µ∈D
e
J(x, w, µ)
.
= sup
w∈W
sup
µ∈D
sup
T<
T
Z
0
l
µ
t
(ξ
t
)
γ
2
2
|w
t
|
2
dt (7)
where
l
µ
t
(x)=
1
2
x
T
D
µ
t
x,
D
= {µ :[0,)→M : measurable },
and ξ satisﬁes
˙
ξ = A
µ
t
ξ + σ
µ
t
w
t
0
=x. (8)
Deﬁne the semigroup
e
S
T
[φ] = sup
w∈W
sup
µ∈D
T
T
Z
0
l
µ
t
(ξ
t
)
γ
2
2
|w
t
|
2
dt + φ(ξ
T
)
where D
T
= {µ :[0,T)→M : measurable }.
Theorem 2.3. Fix any T>0. Value function
e
V is
the unique continuous solution of V =
e
S
T
[V ]in
the class G
δ
for suﬃciently small δ>0. Further,
given any V ∈G
δ
, lim
T →∞
e
S
T
[V ](x)=
e
V(x)
(uniformly on compact sets). Lastly, there exists
c
V
> 0 such that
e
V (x)
1
2
c
V
|x|
2
is convex.
3. MAX-PLUS DUAL OPERATORS
We use , to indicate max-plus addition and
multiplication; max-plus integration (supremiza-
tion) is indicated by an superscript on the
integral sign. Let
IR = IR {−∞}. Recall that
a function, φ : IR
n
IR is semiconvex if given
any R (0, ) there exists β
R
IR such that
φ(x)+
β
R
2
|x|
2
is convex over B
R
(0) = {x IR
n
:
|x|≤R}.Wesayφis uniformly semiconvex with
constant β if φ(x)+
β
2
|x|
2
is convex over IR
n
. Let
S
β
= S
β
(IR
n
) be the set of functions mapping IR
n
into IR which are uniformly semiconvex with con-
stant β. Note that S
β
is a max-plus vector space
(also known as a moduloid) (Fleming and McE-
neaney, 2000), (McEneaney, 2003), (Baccelli, Co-
hen, Olsder and Quadrat, 1992), (Cohen, Gaubert
and Quadrat, 2004), (Litvinov, Maslov and Sh-
piz, 2001). Combining Theorems 2.1 and 2.3, we
have
Theorem 3.1. There exists
β IR such that given
any β>
β,
e
V ∈S
β
and V
m
∈S
β
for all m ∈M.
Further, one may take β<0 (i.e.
e
V,V
m
convex).
The following semiconvex duality result (Fleming
and McEneaney, 2000), (McEneaney, 2003) re-
quires only a small modiﬁcation of convex dual-
ity and Legendre/Fenchel transform results (c.f.
(Rockafellar and Wets, 1997)).
Theorem 3.2. Let φ ∈S
β
. Let C be a symmetric
matrix such that C + βI > 0 (i.e. C + βI positive
deﬁnite) with either C>0orC<0. Deﬁne
ψ : IR
n
×IR
n
IR by ψ(x, z)=
1
2
(xz)
T
C(x
z). Then, for all x IR
n
,
φ(x)= max
zIR
n
[ψ(x, z)+a(z)] (9)
.
=
Z
IR
n
ψ(x, z) a(z) dz
.
= ψ(x, ·) ¯ a(·)
where for all z IR
n
a(z)=
Z
IR
n
ψ(x, z) [φ(x)] dx (10)
= −{ψ(·,z)¯[φ(·)]}
.
=
ψ(·,z)¯[φ
(·)]
ª
.
We will refer to a as the semiconvex dual of φ.
Semiconcavity is the obvious analogue of semicon-
vexity. Let S
β
be the set of functions mapping IR
n
into IR ∪{+∞} which are uniformly semiconcave
with constant β (φ(x) (β/2)|x|
2
concave over all
of IR
n
).
Lemma 3.3. Let φ ∈S
β
, and let a be the semi-
convex dual of φ. Then a ∈S
β
. Further, suppose
b ∈S
β
is such that φ = ψ(x, ·) ¯ b(·). Then b = a.
For simplicity, we will henceforth specialize to the
case where ψ(x, z)
.
=(c/2)|x z|
2
. It will be
critical to the method that
e
S
τ
[ψ(·,z)] ∈S
(c+ε)
for some ε>0. This is the subject of the next
theorem.
Theorem 3.4. We may choose c>0 such that
e
V,V
m
∈S
c
, and such that there exists τ>0
and η>0 such that ,
e
S
τ
[ψ(·,z)],S
m
τ
[ψ(·,z)] ∈S
(c+ητ)
τ [0, τ].
Henceforth, we suppose c, τ, η chosen so that the
results of Theorem 3.4 hold. Now for each z IR
n
,
e
S
τ
[ψ(·,z)] ∈S
(c+ητ)
. Therefore, by Theorem 3.2
e
S
τ
[ψ(·,z)](x)=ψ(x, ·) ¯
e
B
τ
(·,z) (11)
where for all y IR
n
e
B
τ
(y, z)=
ψ(·,y)¯[
e
S
τ
[ψ(·,z)](·)]
ª
(12)
It is handy to deﬁne the max-plus linear operator
with “kernel”
e
B
τ
as
b
e
B
τ
[a](z)
.
=
e
B
τ
(z,·) ¯ a(·) for
all a ∈S
c
. Note that (11), (12) introduce the
dual-space operator kernel
e
B
τ
which propagates
the dual equivalently to propagation in the origi-
nal space by
e
S
τ
.

Proposition 3.5. Let φ ∈S
c
with semiconvex
dual denoted by a. Deﬁne φ
1
=
e
S
τ
[φ]. Then
φ
1
∈S
(c+ητ)
, and φ
1
(x)=ψ(x, ·) ¯ a
1
(·) where
a
1
(x)=
e
B
τ
(x, ·) ¯ a(·).
Theorem 3.6. Let V ∈S
c
, and let a be its
semiconvex dual (with respect to ψ). Then V =
e
S
τ
[V ] if and only if
a(z)=
Z
IR
n
e
B
τ
(z,y) a(y)dy
=
e
B
τ
(z,·) ¯ a(·)=
b
e
B
τ
[a](z) z IR
n
.
Corollary 3.7. The value function
e
V is given by
e
V (x)=ψ(x, ·)¯ ea(·) where ea is the unique solution
of ea(y)=
e
B
τ
(y, ·) ¯ ea(·) y IR
n
or equivalently,
ea =
b
e
B
τ
[ea].
Similarly, for each m ∈Mand z IR
n
,
S
m
τ
[ψ(·,z)] ∈S
(c+ητ)
and
S
m
τ
[ψ(·,z)](x)=ψ(x, ·) ¯B
m
τ
(·,z) x IR
n
where
B
m
τ
(y, z)=
n
ψ(·,y)¯
£
S
m
τ
[ψ(·,z)]
¤
(·)
o
.
As before, it will be handy to deﬁne the max-plus
linear operator with “kernel” B
m
τ
as
b
B
m
τ
[a](z)
.
=
B
m
τ
(z,·) ¯ a(·) for all a ∈S
c
. Further, one also
obtains analogous results (by similar proofs). In
particular, one has the following
Theorem 3.8. Let V ∈S
c
, and let a be its
semiconvex dual (with respect to ψ). Then V =
S
m
τ
[V ] if and only if a(z)=B
m
τ
(z,·) ¯ a(·) z
IR
n
.
Corollary 3.9. Each value function V
m
is given
by V
m
(x)=ψ(x, ·) ¯ a
m
(·) where each a
m
is
the unique solution of a
m
(y)=B
m
τ
(y, ·) ¯ a
m
(·)
y IR
n
.
4. DISCRETE TIME APPROXIMATION
The method develop ed here will not involve any
discretization over space nor any basis functions.
Of course this is obvious since otherwise one could
not avoid the curse-of-dimensionality. The dis-
cretization will be over time, where approximate µ
processes will be constant over the length of each
time-step.
We deﬁne the operator
¯
S
τ
on G
δ
by
¯
S
τ
[φ](x)= sup
w∈W
max
m∈M
·
τ
Z
0
l
m
(ξ
m
t
)
γ
2
2
|w
t
|
2
dt
+φ(ξ
m
τ
)
¸
(x)
= max
m∈M
S
m
τ
[φ](x)
where ξ
m
satisﬁes (2). Let
B
τ
(y, z)
.
= max
m∈M
B
m
τ
(y, z)=
M
m∈M
B
m
τ
(y, z).
The corresponding max-plus linear operator is
b
B
τ
=
M
m∈M
b
B
m
τ
.
Lemma 4.1. For all z IR
n
,
¯
S
τ
[ψ(·,z)]
S
(c+ητ)
. Further,
¯
S
τ
[ψ(·,z)](x)=ψ(x, ·) ¯
B
τ
(·,z).
One has S
m
τ
¯
S
τ
e
S
τ
for all m ∈M. With τ
acting as a time-discretization step-size, let
D
τ
=
n
µ :[0,)→M|for each n N ∪{0},
there exists m
n
∈Msuch that
µ(t)=m
n
t[, (n +1)τ)
o
,
and for T with ¯n N deﬁne D
τ
T
similarly
but with domain [0,T) rather than [0, ). Let
M
¯n
denote the outer product of Mntimes. Let
T , and deﬁne
¯
¯
S
τ
T
[φ](x) = max
{m
k
}
¯n1
k=0
∈M
¯n
(
¯n1
Y
k=0
S
m
k
τ
)
[φ](x)
where the
Q
indicates operator composition.
We will be approximating
e
V by solving V =
¯
S
τ
[V ]
via its dual problem a =
b
B
τ
[a] for small τ .
Consequently, we will need to show that there
exists a solution to V =
¯
S
τ
[V ], that the solution
is unique, and that it can be found by solving the
dual problem. We begin with existence.
Theorem 4.2. Let
V (x)
.
= lim
N→∞
¯
¯
S
τ
[0](x) (13)
for all x IR
n
where 0 here represents the zero-
function. Then,
V satisﬁes
V =
¯
S
τ
[V ],V(0) = 0. (14)
Further, 0 V
m
V
e
V for all m ∈M, and
consequently,
V ∈G
δ
.
Similar techniques to those used for V
m
and
e
V
will prove uniqueness for (14) within G
δ
.
Theorem 4.3.
V is the unique solution of (14)
within the class G
δ
for suﬃciently small δ>0.
Further, given any V ∈G
δ
, lim
N→∞
¯
¯
S
τ
[V ](x)=
V(x) for all x IR
n
(uniformly on compact sets).

Henceforth, we let δ>0 be suﬃciently small such
that V
m
,
e
V,V ∈G
δ
for all m ∈M.
Theorem 4.4. Let V ∈S
c
, and let a be its
semiconvex dual. Then V =
¯
S
τ
[V ] if and only if
a(y)=
B
τ
(y, ·) ¯ a(¯) y IR
n
.
Corollary 4.5. Value function
V given by (13) is
in S
c
, and has representation V (x)=ψ(x, ·) ¯
a(·) where a is the unique solution of a =
b
B
τ
[a].
The following result on propagation of the semi-
convex dual will also come in handy.
Proposition 4.6. Let φ ∈S
c
with semiconvex
dual denoted by a. Deﬁne φ
1
=
¯
S
τ
[φ]. Then
φ
1
∈S
(c+ητ)
, and φ
1
(x)=ψ(x, ·) ¯ a
1
(·) where
a
1
(y)=B
τ
(y, ·) ¯ a(·) y IR
n
.
The next result indicates that one may approxi-
mate
e
V , the solution of V =
e
S
τ
[V ], to as accurate
a level as one desires by solving V =
¯
S
τ
[V ] for
suﬃciently small τ . Recall that if V =
¯
S
τ
[V ], then
it satisﬁes V =
¯
¯
S
τ
[V ] for all N>0 (while
e
V
satisﬁes V =
e
S
[V ]), and so this is essentially
equivalent to introducing a discrete-time
µ ∈D
τ
approximation to the µ process in
e
S
.
Theorem 4.7. Given
ε>0 and R<, there
exists τ>0 such that
e
V (x)
ε V (x)
e
V (x) x B
R
(0).
5. ALGORITHM AND EXAMPLES
Due to space limitations, we cannot give the
steps in the actual algorithm that is generated
by the above theory. However, we note that at
each time step, one generates a set of quadratic
functions, where the coeﬃcients in these functions
are obtained purely analytically. The approximate
solution can be obtained at any point by taking
the maximum of these quadratic functions. In the
absence of any pruning techniques, the number of
quadratic functions at iteration N grows like M
N
.
For large N, this is indeed a large number. There
exist means (such as pruning) for reducing this
growth, but we do not discuss them here. Never-
theless, for small values of M, we obtain a very
rapid solution of such nonlinear HJB PDEs, as
will be indicated in the example to follow. Further,
the computational cost growth in space dimension
n is limited to cubic growth. We emphasize that
the existence of an algorithm avoiding the curse-
of-dimensionality is signiﬁcant regardless of the
practical issues.
A number of examples have so far been tested.
In these tests, the computational speeds were
very great. (Again, some practical issues involving
pruning and initialization are not discussed here
due to space limitations.) This is due to the fact
that M =#Mwas small. The algorithm as
described above was coded in MATLAB (with a
very simple pruning technique and initialization).
The quoted computational times were obtained
with a standard 2001 PC. The times correspond
to the time to compute
V
N
.
=
¯
¯
S
τ
[0]. The
plots below require one to compute the value
function and/or gradients pointwise on planes in
the state space. These plotting computations are
not included in the quoted computational times.
Consider a four-dimensional example with con-
stituent Hamiltonians, H
m
, whose A
m
are
A
1
=
1.00.50.00.1
0.11.00.20.0
0.20.01.50.1
0.00.10.01.5
,
A
2
=(A
1
)
T
,
A
3
=
1.00.50.00.1
0.11.00.20.0
0.20.01.60.1
0.00.05 0.1 1.5
.
The D
m
and Σ
m
were simply
D
1
= D
2
= D
3
=
1.50.20.10.0
0.21.50.00.1
0.10.01.50.0
0.00.10.01.5
,
and
Σ
1
2
3
=
0.2 0.01 0.02 0.01
0.01 0.20.00.0
0.02 0.00.25 0.0
0.01 0.00.00.25
.
The results of this four-dimensional example ap-
pear in Figures 1–4. In this case, the results have
been plotted over the region of the aﬃne plane
x
3
=3,x
4
=0.5 given by x
1
[10, 10]
and x
2
[10, 10]. The backsubstitution error
has been scaled by dividing by |x|
2
+10
5
. The
computations required approximately 40 seconds.
−10
−5
0
5
10
−10
−5
0
5
10
0
20
40
60
80
100
120
140
160
180
Fig. 1. Value function (4-D case)

##### Citations
More filters

Journal ArticleDOI
TL;DR: This scheme is shown to be capable of learning the optimal control without requiring an initial guess and to enhance the efficiency of the proposed scheme when treating more complex nonlinear systems, an iterative algorithm based on Girsanov's theorem on the change of measure is derived.
Abstract: The aim of this work is to present a novel sampling-based numerical scheme designed to solve a certain class of stochastic optimal control problems, utilizing forward and backward stochastic differential equations (FBSDEs). By means of a nonlinear version of the Feynman–Kac lemma, we obtain a probabilistic representation of the solution to the nonlinear Hamilton–Jacobi–Bellman equation, expressed in the form of a system of decoupled FBSDEs. This system of FBSDEs can be solved by employing linear regression techniques. The proposed framework relaxes some of the restrictive conditions present in recent sampling based methods within the Linearly Solvable Optimal Control framework, and furthermore addresses problems in which the time horizon is not prespecified. To enhance the efficiency of the proposed scheme when treating more complex nonlinear systems, we then derive an iterative algorithm based on Girsanov’s theorem on the change of measure, which features importance sampling. This scheme is shown to be capable of learning the optimal control without requiring an initial guess.

46 citations

### Cites methods from "A curse-of-dimensionality-free nume..."

• ...In the first case, several attempts have been made to address the difficulty inherent in solving such nonlinear PDEs, aswell as the curse of dimensionality, with various different methods and approaches (Beard, Saridis, & Wen, 1997; Lasserre, Henrion, Prieur, & Trelat, 2008; McEneaney, 2007) for deterministic control problems, while a stochastic setting is considered in Gorodetsky, Karaman, and Marzouk (2015), Horowitz and Burdick (2014) and Horowitz, Damle, and Burdick (2014)....

[...]

Journal ArticleDOI
Abstract: The matrix differential Riccati equation (DRE) is ubiquitous in control and systems theory. The presence of the quadratic term implies that a simple linear-systems fundamental solution does not exist. Of course it is well-known that the Bernoulli substitution may be applied to obtain a linear system of doubled size. Here, however, tools from max-plus analysis and semiconvex duality are brought to bear on the DRE. We consider the DRE as a finite-dimensional solution to a deterministic linear/quadratic control problem. Taking the semiconvex dual of the associated semigroup, one obtains the solution operator as a max-plus integral operator with quadratic kernel. The kernel is equivalently represented as a matrix. Using the semigroup property of the dual operator, one obtains a matrix operation whereby the kernel matrix propagates as a semigroup. The propagation forward is through some simple matrix operations. This time-indexed family of matrices forms a new fundamental solution for the DRE. Solution for any initial condition is obtained by a few matrix operations on the fundamental solution and the initial condition. In analogy with standard-algebra linear systems, the fundamental solution can be viewed as an exponential form over a certain idempotent semiring. This fundamental solution has a particularly nice control interpretation, and might lead to improved DRE solution speeds.

42 citations

Journal ArticleDOI
TL;DR: This work obtains specific error bounds for a previously obtained numerical method not subject to the curse-of-dimensionality of HJB PDEs where the Hamiltonian takes the form of a (pointwise) maximum of linear/quadratic forms.
Abstract: In previous work of the first author and others, max-plus methods have been explored for solution of first-order, nonlinear Hamilton-Jacobi-Bellman partial differential equations (HJB PDEs) and corresponding nonlinear control problems. Although max-plus basis expansion and max-plus finite-element methods can provide substantial computational-speed advantages, they still generally suffer from the curse-of-dimensionality. Here we consider HJB PDEs where the Hamiltonian takes the form of a (pointwise) maximum of linear/quadratic forms. The approach to solution will be rather general, but in order to ground the work, we consider only constituent Hamiltonians corresponding to long-run average-cost-per-unit-time optimal control problems for the development. We consider a previously obtained numerical method not subject to the curse-of-dimensionality. The method is based on construction of the dual-space semigroup corresponding to the HJB PDE. This dual-space semigroup is constructed from the dual-space semigroups corresponding to the constituent linear/quadratic Hamiltonians. The dual-space semigroup is particularly useful due to its form as a max-plus integral operator with kernel obtained from the originating semigroup. One considers repeated application of the dual-space semigroup to obtain the solution. Although previous work indicated that the method was not subject to the curse-of-dimensionality, it did not indicate any error bounds or convergence rate. Here we obtain specific error bounds.

32 citations

### Cites methods from "A curse-of-dimensionality-free nume..."

• ...In [29], [30], [31], [32], a new class of methods for first-order HJB PDEs was introduced, and these methods are not subject to the curse-of-dimensionality....

[...]

Proceedings ArticleDOI
01 Dec 2009
TL;DR: It is shown that a similar, albeit more abstract, approach can be applied to deterministic game problems by finding reduced-complexity approximations to min-max sums of max-plus affine functions.
Abstract: In recent years, idempotent methods (specifically, max-plus methods) have been developed for solution of nonlinear control problems. It was thought that idempotent linearity of the associated semigroup was required for application of these techniques. It is now known that application of the max-plus distributive property allows one to apply the max-plus curse-of-dimensionality-free approach to stochastic control problems. Here, we see that a similar, albeit more abstract, approach can be applied to deterministic game problems. The main difficulty is a curse-of-complexity growth in the computational cost. Attenuation of this effect requires finding reduced-complexity approximations to min-max sums of max-plus affine functions. We demonstrate that this problem can be reduced to a pruning problem.

9 citations

##### References
More filters

Book
18 Dec 1997
Abstract: Preface.- Basic notations.- Outline of the main ideas on a model problem.- Continuous viscosity solutions of Hamilton-Jacobi equations.- Optimal control problems with continuous value functions: unrestricted state space.- Optimal control problems with continuous value functions: restricted state space.- Discontinuous viscosity solutions and applications.- Approximation and perturbation problems.- Asymptotic problems.- Differential Games.- Numerical solution of Dynamic Programming.- Nonlinear H-infinity control by Pierpaolo Soravia.- Bibliography.- Index

2,561 citations

### "A curse-of-dimensionality-free nume..." refers methods in this paper

• ...The most common methods by far all fall into the class of finite element methods (cf. (Bardi and Capuzzo-Dolcetta, 1997), (Dupuis and Boué, 1999), among many others)....

[...]

• ...(Bardi and Capuzzo-Dolcetta, 1997), (Dupuis and Boué, 1999), among many others)....

[...]

Journal ArticleDOI
Abstract: This paper introduces random versions of successive approximations and multigrid algorithms for computing approximate solutions to a class of finite and infinite horizon Markovian decision problems (MDPs). We prove that these algorithms succeed in breaking the curse of dimensionality for a subclass of MDPs known as discrete decision processes (DDPs).

425 citations

Book
30 Apr 1997
Abstract: Preface. 1. Idempotent Analysis. 2. Analysis of Operators on Idempotent Semimodules. 3. Generalized Solutions of Bellman's Differential Equation. 4. Quantization of the Bellman Equation and Multiplicative Asymptotics. References. Appendix: (P. Del Moral) Maslov Optimziation Theory. Optimality versus Randomness. Index.

398 citations

Journal ArticleDOI
Abstract: We consider subsemimodules and convex subsets of semimodules over semirings with an idempotent addition. We introduce a nonlinear projection on subsemimodules: the projection of a point is the maximal approximation from below of the point in the subsemimodule. We use this projection to separate a point from a convex set. We also show that the projection minimizes the analogue of Hilbert’s projective metric. We develop more generally a theory of dual pairs for idempotent semimodules. We obtain as a corollary duality results between the row and column spaces of matrices with entries in idempotent semirings. We illustrate the results by showing polyhedra and half-spaces over the max-plus semiring.

259 citations

Journal ArticleDOI
Grigory L. Litvinov
Abstract: This paper is devoted to Idempotent Functional Analysis, which is an “abstract” version of Idempotent Analysis developed by V. P. Maslov and his collaborators. We give a brief survey of the basic ideas of Idempotent Analysis. The correspondence between concepts and theorems of traditional Functional Analysis and its idempotent version is discussed in the spirit of N. Bohr's correspondence principle in quantum theory. We present an algebraic approach to Idempotent Functional Analysis. Basic notions and results are formulated in algebraic terms; the essential point is that the operation of idempotent addition can be defined for arbitrary infinite sets of summands. We study idempotent analogs of the basic principles of linear functional analysis and results on the general form of a linear functional and scalar products in idempotent spaces.

209 citations