Finance Stoch (2017) 21:331–360
DOI 10.1007/s00780-017-0327-5
On time-inconsistent stochastic control in continuous
time
Tomas Björk
1
·Mariana Khapko
2
·
Agatha Murgoci
3
Received: 9 April 2014 / Accepted: 29 November 2016 / Published online: 13 March 2017
© The Author(s) 2017. This article is published with open access at Springerlink.com
Abstract In this paper, which is a continuation of the discrete-time paper (Björk
and Murgoci in Finance Stoch. 18:545–592, 2004), we study a class of continuous-
time stochastic control problems which, in various ways, are time-inconsistent in the
sense that they do not admit a Bellman optimality principle. We study these prob-
lems within a game-theoretic framework, and we look for Nash subgame perfect
equilibrium points. For a general controlled continuous-time Markov process and a
fairly general objective functional, we derive an extension of the standard Hamilton–
Jacobi–Bellman equation, in the form of a system of nonlinear equations, for the
determination of the equilibrium strategy as well as the equilibrium value function.
The main theoretical result is a verification theorem. As an application of the gen-
eral theory, we study a time-inconsistent linear-quadratic regulator. We also present
a study of time-inconsistency within the framework of a general equilibrium produc-
tion economy of Cox–Ingersoll–Ross type (Cox et al. in Econometrica 53:363–384,
1985).
B
T. Björk
tomas.bjork@hhs.se
M. Khapko
mariana.khapko@rotman.utoronto.ca
A. Murgoci
agatha.murgoci@econ.au.dk
1
Department of Finance, Stockholm School of Economics, Box 6501, 113 83 Stockholm, Sweden
2
Department of Management (UTSc), Rotman School of Management, University of Toronto,
105 St. George Street, Toronto, ON, M5S 3E6, Canada
3
Department of Economics and Business Economics, Aarhus University, Fuglesangs Allé 4,
8210 Aarhus V, Denmark
332 T. Björk et al.
Keywords Time-consistency · Time-inconsistency · Time-inconsistent control ·
Dynamic programming · Stochastic control · Bellman equation · Hyperbolic
discounting · Mean-variance · Equilibrium
Mathematics Subject Classification 49L99 · 49N90 · 60J70 · 91A10 · 91A80 ·
91B02 · 91B25 · 91B51 · 91G80
JEL Classification C61 · C72 · C73 · D5 · G11 · G12
1 Introduction
The purpose of this paper is to study a class of stochastic control problems in con-
tinuous time which have the property of being time-inconsistent in the sense that
they do not allow a Bellman optimality principle. As a consequence, the very con-
cept of optimality becomes problematic, since a strategy which is optimal given a
specific starting point in time and space may be non-optimal when viewed from a
later date and a different state. In this paper, we attack a fairly general class of time-
inconsistent problems by using a game-theoretic approach; so instead of searching for
optimal strategies, we search for subgame perfect Nash equilibrium strategies. The
paper presents a continuous-time version of the discrete-time theory developed in our
previous paper [5]. Since we build heavily on the discrete-time paper, the reader is
referred to that for motivating examples and more detailed discussions on conceptual
issues.
1.1 Previous literature
For a detailed discussion of the game-theoretic approach to time-inconsistency using
Nash equilibrium points as above, the reader is referred to [5].Alistofsomeofthe
most important papers on the subject is given by [2, 6, 8–14, 16, 18–25].
All the papers above deal with particular model choices, and different authors
use different methods in order to solve the problems. To our knowledge, the present
paper, which is the continuous-time part of the working paper [4], is the first attempt
to study a reasonably general (albeit Markovian) class of time-inconsistent control
problems in continuous time. We should, however, like to stress that for the present
paper, we have been greatly inspired by [2, 9, 11].
1.2 Structure of the paper
The structure of the paper is roughly as follows.
– In Sect. 2, we present the basic setup, and in Sect. 3, we discuss the concept of equi-
librium. This replaces in our setting the optimality concept for a standard stochastic
control problem, and in Definition 3.4, we give a precise definition of the equilib-
rium control and the equilibrium value function.
Time-inconsistent control 333
– Since the equilibrium concept in continuous time is quite delicate, we build the
continuous-time theory on the discrete-time theory previously developed in [5]. In
Sect. 4, we start to study the continuous-time problem by going to the limit for a
discretized problem, and using the results from [5]. This leads to an extension of
the standard HJB equation to a system of equations with an embedded static opti-
mization problem. The limiting procedure described above is done in an informal
manner. It is largely heuristic, and it thus remains to clarify precisely how the de-
rived extended HJB system is related to the precisely defined equilibrium problem
under consideration.
– The needed clarification is in fact delivered in Sect. 5. In Theorem 5.2, which is
the main theoretical result of the paper, we give a precise statement and proof of a
verification theorem. This theorem says that a solution to the extended HJB system
does indeed deliver the equilibrium control and equilibrium value function to our
original problem.
– In Sect. 6, the results of Sect. 5 are extended to a more general reward functional.
– Section 7 treats the infinite-horizon case.
– In Sect. 8, we study a time-inconsistent version of the linear-quadratic regulator to
illustrate how the theory works in a concrete case.
– Section 9 is devoted to a rather detailed study of a general equilibrium model for a
production economy with time-inconsistent preferences.
– In Sect. 10, we review some remaining open problems.
For extensions of the theory as well as worked out examples such as point process
models, non-exponential discounting, mean-variance control, and state-dependent
risk, see the working paper overview [3].
2 The model
We now turn to the formal continuous-time theory. In order to present this, we need
some input data.
Definition 2.1 The following objects are given exogenously:
1. A drift mapping μ :R
+
×R
n
×R
k
→R
n
.
2. A diffusion mapping σ : R
+
×R
n
×R
k
→ M(n,d), where M(n,d) denotes the
set of all n ×d matrices.
3. A control constraint mapping U :R
+
×R
n
→2
R
k
.
4. A mapping F :R
n
×R
n
→R.
5. A mapping G : R
n
×R
n
→R.
We now consider, on the time interval [0,T], a controlled SDE of the form
dX
t
=μ(t, X
t
,u
t
)dt +σ(t,X
t
,u
t
)dW
t
, (2.1)
where the state process X is n-dimensional, the Wiener process W is d-dimensional,
and the control process u is k-dimensional, with the constraint u
t
∈U(t,X
t
).
334 T. Björk et al.
Loosely speaking, our objective is to maximize, for every initial point (t, x),a
reward functional of the form
E
t,x
[
F(x,X
T
)
]
+G(x, E
t,x
[
X
T
]
).
This functional is not of a form which is suitable for dynamic programming, and this
will be discussed in detail below, but first we need to specify our class of controls. In
this paper, we restrict the controls to admissible feedback control laws.
Definition 2.2 An admissible control law is a map u :[0,T]×R
n
→R
k
satisfying
the following conditions:
1. For each (t, x) ∈[0,T]×R
n
,wehaveu(t, x) ∈U(t,x).
2. For each initial point (s, y) ∈[0,T]×R
n
, the SDE
dX
t
=μ
t,X
t
, u(t, X
t
)
dt +σ
t,X
t
, u(t, X
t
)
dW
t
,X
s
=y
has a unique strong solution denoted by X
u
.
The class of admissible control laws is denoted by U. We sometimes use the notation
u
t
(x) instead of u(t, x).
We now go on to define the controlled infinitesimal generator of the SDE above. In
the present paper, we use the (somewhat non-standard) convention that the infinitesi-
mal operator acts on the time variable as well as on the space variable; so it includes
the term
∂
∂t
.
Definition 2.3 Consider the SDE (2.1), and let
denote matrix transpose.
– For any fixed u ∈R
k
, the functions μ
u
, σ
u
and C
u
are defined by
μ
u
(t, x) = μ(t, x, u), σ
u
(t, x) =σ(t,x,u),
C
u
(t, x) = σ(t,x,u)σ(t,x,u)
.
– For any admissible control law u, the functions μ
u
, σ
u
, C
u
(t, x) are defined by
μ
u
(t, x) = μ
t,x,u(t, x)
,σ
u
(t, x) =σ
t,x,u(t, x)
,
C
u
(t, x) = σ
t,x,u(t, x)
σ
t,x,u(t, x)
.
– For any fixed u ∈R
k
, the operator A
u
is defined by
A
u
=
∂
∂t
+
n
i=1
μ
u
i
(t, x)
∂
∂x
i
+
1
2
n
i,j=1
C
u
ij
(t, x)
∂
2
∂x
i
∂x
j
.
– For any admissible control law u, the operator A
u
is defined by
A
u
=
∂
∂t
+
n
i=1
μ
u
i
(t, x)
∂
∂x
i
+
1
2
n
i,j=1
C
u
ij
(t, x)
∂
2
∂x
i
∂x
j
.
Time-inconsistent control 335
3 Problem formulation
In order to formulate our problem, we need an objective functional. We thus consider
the two functions F and G from Definition 2.1.
Definition 3.1 For a fixed (t, x) ∈[0,T]×R
n
and a fixed admissible control law u,
the corresponding reward functional J is defined by
J(t,x,u) =E
t,x
[F(x,X
u
T
)]+G(x, E
t,x
[X
u
T
]). (3.1)
Remark 3.2 In Sect. 6, we consider a more general reward functional. The restriction
to the functional (3.1) above is done in order to minimize the notational complexity
of the derivations below, which otherwise would be somewhat messy.
In order to have a nondegenerate problem, we need a formal integrability assump-
tion.
Assumption 3.3 We assume that for each initial point (t, x) ∈[0,T]×R
n
and each
admissible control law u,wehave
E
t,x
[|F(x,X
u
T
)|]< ∞,E
t,x
[|X
u
T
|]< ∞
and hence
G(x, E
t,x
[X
u
T
])<∞.
Our objective is loosely that of maximizing J(t,x,u) for each (t, x), but concep-
tually this turns out to be far from trivial, so instead of optimal controls we will study
equilibrium controls. The equilibrium concept is made precise in Definition 3.4 be-
low, but in order to motivate that definition, we need a brief discussion concerning
the reward functional above.
We immediately note that in contrast to a standard optimal control problem, the
family of reward functionals above are not connected by a Bellman optimality prin-
ciple. The reasons for this are as follows:
– The present state x appears in the function F .
– In the second term, we have (even apart from the appearance of the present state x)
a nonlinear function G operating on the expected value E
t,x
[X
u
T
].
Since we do not have a Bellman optimality principle, it is in fact unclear what we
should mean by the term “optimal”, since the optimality concept would differ at dif-
ferent initial times t and for different initial states x.
The approach of this paper is to adopt a game-theoretic perspective and look for
subgame perfect Nash equilibrium points. Loosely speaking, we view the game as
follows:
– Consider a non-cooperative game where we have one player for each point in
time t. We refer to this player as “Player t”.
– For each fixed t, Player t can only control the process X exactly at time t .He/she
does that by choosing a control function u(t, ·); so the action taken at time t with
state X
t
is given by u(t, X
t
).