Discretisation of stochastic control problems for continuous time dynamics with delay
20 Aug 2007-Journal of Computational and Applied Mathematics (North-Holland)-Vol. 205, Iss: 2, pp 969-981
TL;DR: In this article, the existence of an optimal strategy with respect to the cost functional can be guaranteed in the class of relaxed controls in continuous-time control problems in continuous time, where the controlled process is approximated by sequences of controlled Markov chains, thus discretising time and space.
About: This article is published in Journal of Computational and Applied Mathematics.The article was published on 2007-08-20 and is currently open access. It has received 10 citations till now. The article focuses on the topics: Markov process & Markov chain.
Citations
More filters
TL;DR: The equivalence between the one-dimensional delay problem and the associated infinite-dimensional problem without delay is shown and it is proved that the value function is continuous in this infinite- dimensional setting.
Abstract: This paper deals with the optimal control of a stochastic delay differential equation arising in the management of a pension fund with surplus. The problem is approached by the tool of a representation in infinite dimension. We show the equivalence between the one-dimensional delay problem and the associated infinite-dimensional problem without delay. Then we prove that the value function is continuous in this infinite-dimensional setting. These results represent a starting point for the investigation of the associated infinite-dimensional Hamilton–Jacobi–Bellman equation in the viscosity sense and for approaching the problem by numerical algorithms. Also an example with complete solution of a simpler but similar problem is provided.
89 citations
Cites background from "Discretisation of stochastic contro..."
...More recent references on this approach for stochastic delay control problems are the papers [24, 25, 38]....
[...]
TL;DR: In this article, the authors studied stochastic delay differential equations (SDDE) where the coefficients depend on the moving averages of the state process and provided sufficient conditions under which the solution of the SDDE and a linear path functional of it admit a finite-dimensional Markovian representation.
Abstract: We study stochastic delay differential equations (SDDE) where the coefficients depend on the moving averages of the state process. As a first contribution, we provide sufficient conditions under which the solution of the SDDE and a linear path functional of it admit a finite-dimensional Markovian representation. As a second contribution, we show how approximate finite-dimensional Markovian representations may be constructed when these conditions are not satisfied, and provide an estimate of the error corresponding to these approximations. These results are applied to optimal control and optimal stopping problems for stochastic systems with delay.
12 citations
Cites methods from "Discretisation of stochastic contro..."
...problem reduces to a finite-dimensional one [2, 9, 28, 30]. In the general case, [27] extends the Markov chain approximation method to stochastic equations with delay. A similar method is developed in [32], and [17] establish convergence rates for an approximation of this kind. The infinitedimensional Hilbertian approach to controlled deterministic and stochastic systems with delays in the state variabl...
[...]
TL;DR: In this article, a semi-discretisation scheme for stochastic optimal control problems whose dynamics are given by controlled stochiastic delay (or functional) differential equations with bounded memory is studied.
Abstract: We study a semi-discretisation scheme for stochastic optimal control problems whose dynamics are given by controlled stochastic delay (or functional) differential equations with bounded memory. Performance is measured in terms of expected costs. By discretising time in two steps, we construct a sequence of approximating finite-dimensional Markovian optimal control problems in discrete time. The corresponding value functions converge to the value function of the original problem, and we derive an upper bound on the discretisation error or, equivalently, a worst-case estimate for the rate of convergence.
10 citations
Cites methods from "Discretisation of stochastic contro..."
...The techniques used in [22] and related work for the proofs of convergence are based on weak convergence of measures; they can be extended to cover control problems with delay, see [11, 20]....
[...]
Posted Content•
TL;DR: In this article, the authors study stochastic delay differential equations (SDDE) where the coefficients depend on the moving averages of the state process and provide sufficient conditions under which a linear path functional of the solution of a SDDE admits a finite-dimensional Markovian representation.
Abstract: We study stochastic delay differential equations (SDDE) where the coefficients depend on the moving averages of the state process. As a first contribution, we provide sufficient conditions under which a linear path functional of the solution of a SDDE admits a finite-dimensional Markovian representation. As a second contribution, we show how approximate finite-dimensional Markovian representations may be constructed when these conditions are not satisfied, and provide an estimate of the error corresponding to these approximations. These results are applied to optimal control and optimal stopping problems for stochastic systems with delay.
5 citations
TL;DR: In this paper, the authors employ neural networks for sequence modeling (e.g., recurrent neural networks such as long short-term memory) to parameterize the policy and optimize the objective function.
Abstract: Stochastic control problems with delay are challenging due to the path-dependent feature of the system and thus its intrinsic high dimensions. In this paper, we propose and systematically study deep neural network-based algorithms to solve stochastic control problems with delay features. Specifically, we employ neural networks for sequence modeling (e.g., recurrent neural networks such as long short-term memory) to parameterize the policy and optimize the objective function. The proposed algorithms are tested on three benchmark examples: a linear-quadratic problem, optimal consumption with fixed finite delay, and portfolio optimization with complete memory. Particularly, we notice that the architecture of recurrent neural networks naturally captures the path-dependent feature with much flexibility and yields better performance with more efficient and stable training of the network compared to feedforward networks. The superiority is even evident in the case of portfolio optimization with complete memory, which features infinite delay.
5 citations
References
More filters
TL;DR: An approximation scheme for a nonlinear filtering problem when the state process X is the solution of a stochastic delay diffusion equation and the observation process is a noisy function of X for t-tau, where $\tau$ is a constant.
Abstract: We study an approximation scheme for a nonlinear filtering problem when the state process $X$ is the solution of a stochastic delay diffusion equation and the observation process is a noisy function of $X(s)$ for $s\in [t-\tau,t]$, where $\tau$ is a constant. The approximating state is the piecewise linear Euler-Maruyama scheme, and the observation process is a noisy function of the approximating state. The rate of convergence of this scheme is computed.
12 citations
"Discretisation of stochastic contro..." refers background in this paper
...Elsanosi et al. (2000) for certain explicitly available solutions, Øksendal and Sulem (2001) for the derivation of a maximum principle and Larssen (2002) for the dynamic programming approach....
[...]