Discretisation of stochastic control problems for continuous time dynamics with delay
20 Aug 2007-Journal of Computational and Applied Mathematics (North-Holland)-Vol. 205, Iss: 2, pp 969-981
TL;DR: In this article, the existence of an optimal strategy with respect to the cost functional can be guaranteed in the class of relaxed controls in continuous-time control problems in continuous time, where the controlled process is approximated by sequences of controlled Markov chains, thus discretising time and space.
About: This article is published in Journal of Computational and Applied Mathematics.The article was published on 2007-08-20 and is currently open access. It has received 10 citations till now. The article focuses on the topics: Markov process & Markov chain.
Citations
More filters
TL;DR: The equivalence between the one-dimensional delay problem and the associated infinite-dimensional problem without delay is shown and it is proved that the value function is continuous in this infinite- dimensional setting.
Abstract: This paper deals with the optimal control of a stochastic delay differential equation arising in the management of a pension fund with surplus. The problem is approached by the tool of a representation in infinite dimension. We show the equivalence between the one-dimensional delay problem and the associated infinite-dimensional problem without delay. Then we prove that the value function is continuous in this infinite-dimensional setting. These results represent a starting point for the investigation of the associated infinite-dimensional Hamilton–Jacobi–Bellman equation in the viscosity sense and for approaching the problem by numerical algorithms. Also an example with complete solution of a simpler but similar problem is provided.
89 citations
Cites background from "Discretisation of stochastic contro..."
...More recent references on this approach for stochastic delay control problems are the papers [24, 25, 38]....
[...]
TL;DR: In this article, the authors studied stochastic delay differential equations (SDDE) where the coefficients depend on the moving averages of the state process and provided sufficient conditions under which the solution of the SDDE and a linear path functional of it admit a finite-dimensional Markovian representation.
Abstract: We study stochastic delay differential equations (SDDE) where the coefficients depend on the moving averages of the state process. As a first contribution, we provide sufficient conditions under which the solution of the SDDE and a linear path functional of it admit a finite-dimensional Markovian representation. As a second contribution, we show how approximate finite-dimensional Markovian representations may be constructed when these conditions are not satisfied, and provide an estimate of the error corresponding to these approximations. These results are applied to optimal control and optimal stopping problems for stochastic systems with delay.
12 citations
Cites methods from "Discretisation of stochastic contro..."
...problem reduces to a finite-dimensional one [2, 9, 28, 30]. In the general case, [27] extends the Markov chain approximation method to stochastic equations with delay. A similar method is developed in [32], and [17] establish convergence rates for an approximation of this kind. The infinitedimensional Hilbertian approach to controlled deterministic and stochastic systems with delays in the state variabl...
[...]
TL;DR: In this article, a semi-discretisation scheme for stochastic optimal control problems whose dynamics are given by controlled stochiastic delay (or functional) differential equations with bounded memory is studied.
Abstract: We study a semi-discretisation scheme for stochastic optimal control problems whose dynamics are given by controlled stochastic delay (or functional) differential equations with bounded memory. Performance is measured in terms of expected costs. By discretising time in two steps, we construct a sequence of approximating finite-dimensional Markovian optimal control problems in discrete time. The corresponding value functions converge to the value function of the original problem, and we derive an upper bound on the discretisation error or, equivalently, a worst-case estimate for the rate of convergence.
10 citations
Cites methods from "Discretisation of stochastic contro..."
...The techniques used in [22] and related work for the proofs of convergence are based on weak convergence of measures; they can be extended to cover control problems with delay, see [11, 20]....
[...]
Posted Content•
TL;DR: In this article, the authors study stochastic delay differential equations (SDDE) where the coefficients depend on the moving averages of the state process and provide sufficient conditions under which a linear path functional of the solution of a SDDE admits a finite-dimensional Markovian representation.
Abstract: We study stochastic delay differential equations (SDDE) where the coefficients depend on the moving averages of the state process. As a first contribution, we provide sufficient conditions under which a linear path functional of the solution of a SDDE admits a finite-dimensional Markovian representation. As a second contribution, we show how approximate finite-dimensional Markovian representations may be constructed when these conditions are not satisfied, and provide an estimate of the error corresponding to these approximations. These results are applied to optimal control and optimal stopping problems for stochastic systems with delay.
5 citations
TL;DR: In this paper, the authors employ neural networks for sequence modeling (e.g., recurrent neural networks such as long short-term memory) to parameterize the policy and optimize the objective function.
Abstract: Stochastic control problems with delay are challenging due to the path-dependent feature of the system and thus its intrinsic high dimensions. In this paper, we propose and systematically study deep neural network-based algorithms to solve stochastic control problems with delay features. Specifically, we employ neural networks for sequence modeling (e.g., recurrent neural networks such as long short-term memory) to parameterize the policy and optimize the objective function. The proposed algorithms are tested on three benchmark examples: a linear-quadratic problem, optimal consumption with fixed finite delay, and portfolio optimization with complete memory. Particularly, we notice that the architecture of recurrent neural networks naturally captures the path-dependent feature with much flexibility and yields better performance with more efficient and stable training of the network compared to feedforward networks. The superiority is even evident in the case of portfolio optimization with complete memory, which features infinite delay.
5 citations
References
More filters
TL;DR: A verification theorem of variational inequality type is proved and is applied to solve explicitly some classes of optimal harvesting delay problems.
Abstract: We consider optimal harvesting of systems described by stochastic differential equations with delay. We focus on those situations where the value function of the harvesting problem depends on the initial path of the process in a simple way, namely through its value at 0 and through some weighted averages A verification theorem of variational inequality type is proved. This is applied to solve explicitly some classes of optimal harvesting delay problems
159 citations
"Discretisation of stochastic contro..." refers methods in this paper
...The mathematical analysis of stochastic control problems with time delay in the state equation has been the object of recent works, see e. g. Elsanosi et al. (2000) for certain explicitly available solutions, Øksendal and Sulem (2001) for the derivation of a maximum principle and Larssen (2002) for…...
[...]
01 Jan 2000
TL;DR: In this article, the authors consider optimal control problems for systems described by stochastic differential equations with delay and prove two (sufficient) maximum principles for certain classes of such systems.
Abstract: We consider optimal control problems for systems described by stochastic differential equations with delay. We prove two (sufficient) maximum principles for certain classes of such systems, one for ordinary stochastic delay control and one which also includes singular stochastic delay control. As an application we find explicitly the optimal consumption rate from an economic quantity described by a stochastic delay equation of a certain type. We also solve a Merton type optimal portfolio problem in a market with delay.
130 citations
TL;DR: In this paper, a strong Milstein approximation scheme for solving stochastic delay differential equations (SDDEs) was developed, which has convergence order 1. In order to establish the scheme, they proved an infinite-dimensional Ito formula for "tame" functions acting on the segment process of the solution of an SDDE.
Abstract: In this paper, we develop a strong Milstein approximation scheme for solving stochastic delay differential equations (SDDEs). The scheme has convergence order 1. In order to establish the scheme, we prove an infinite-dimensional Ito formula for "tame'' functions acting on the segment process of the solution of an SDDE. It is interesting to note that the presence of the memory in the SDDE requires the use of the Malliavin calculus and the anticipating stochastic analysis of Nualart and Pardoux. Given the nonanticipating nature of the SDDE, the use of anticipating calculus methods in the context of strong approximation schemes appears to be novel.
126 citations
"Discretisation of stochastic contro..." refers methods in this paper
...The development of numerical methods for SDDEs has attracted much attention recently, see Buckwar (2000), Hu et al. (2004) and the references therein....
[...]
01 Jan 1996
TL;DR: In this paper, the authors define a notion of an integral J: H.dW, where H is a stochastic process; or more generally an indefinite integral J; H.w.d W, o t < 00.
Abstract: Let W denote a standard Wiener process with Wo = O. For a variety of reasons, it is desirable to have a notion of an integral J: H.dW., where H is a stochastic process; or more generally an indefinite integral J; H.dW., o t < 00. IfH is a process with continuous paths, an obvious way to define a stochastic integral is by a limit of sums: let 7l"n [0, t] be a sequence of partitions of [O,t], with mesh (7l"n) = SUPi(ti+l td, where 0 = to < t 1 < ... < t« = t are the successive points of the partition. Then one could define
124 citations
TL;DR: This paper analyzes the equilibrium dynamics of an AK-type endogenous growth model with vintage capital, which leads to oscillatory dynamics governed by replacement echoes, which additionally influence the intercept of the balanced growth path.
124 citations