Discretisation of stochastic control problems for continuous time dynamics with delay
20 Aug 2007-Journal of Computational and Applied Mathematics (North-Holland)-Vol. 205, Iss: 2, pp 969-981
TL;DR: In this article, the existence of an optimal strategy with respect to the cost functional can be guaranteed in the class of relaxed controls in continuous-time control problems in continuous time, where the controlled process is approximated by sequences of controlled Markov chains, thus discretising time and space.
About: This article is published in Journal of Computational and Applied Mathematics.The article was published on 2007-08-20 and is currently open access. It has received 10 citations till now. The article focuses on the topics: Markov process & Markov chain.
Citations
More filters
TL;DR: The equivalence between the one-dimensional delay problem and the associated infinite-dimensional problem without delay is shown and it is proved that the value function is continuous in this infinite- dimensional setting.
Abstract: This paper deals with the optimal control of a stochastic delay differential equation arising in the management of a pension fund with surplus. The problem is approached by the tool of a representation in infinite dimension. We show the equivalence between the one-dimensional delay problem and the associated infinite-dimensional problem without delay. Then we prove that the value function is continuous in this infinite-dimensional setting. These results represent a starting point for the investigation of the associated infinite-dimensional Hamilton–Jacobi–Bellman equation in the viscosity sense and for approaching the problem by numerical algorithms. Also an example with complete solution of a simpler but similar problem is provided.
89 citations
Cites background from "Discretisation of stochastic contro..."
...More recent references on this approach for stochastic delay control problems are the papers [24, 25, 38]....
[...]
TL;DR: In this article, the authors studied stochastic delay differential equations (SDDE) where the coefficients depend on the moving averages of the state process and provided sufficient conditions under which the solution of the SDDE and a linear path functional of it admit a finite-dimensional Markovian representation.
Abstract: We study stochastic delay differential equations (SDDE) where the coefficients depend on the moving averages of the state process. As a first contribution, we provide sufficient conditions under which the solution of the SDDE and a linear path functional of it admit a finite-dimensional Markovian representation. As a second contribution, we show how approximate finite-dimensional Markovian representations may be constructed when these conditions are not satisfied, and provide an estimate of the error corresponding to these approximations. These results are applied to optimal control and optimal stopping problems for stochastic systems with delay.
12 citations
Cites methods from "Discretisation of stochastic contro..."
...problem reduces to a finite-dimensional one [2, 9, 28, 30]. In the general case, [27] extends the Markov chain approximation method to stochastic equations with delay. A similar method is developed in [32], and [17] establish convergence rates for an approximation of this kind. The infinitedimensional Hilbertian approach to controlled deterministic and stochastic systems with delays in the state variabl...
[...]
TL;DR: In this article, a semi-discretisation scheme for stochastic optimal control problems whose dynamics are given by controlled stochiastic delay (or functional) differential equations with bounded memory is studied.
Abstract: We study a semi-discretisation scheme for stochastic optimal control problems whose dynamics are given by controlled stochastic delay (or functional) differential equations with bounded memory. Performance is measured in terms of expected costs. By discretising time in two steps, we construct a sequence of approximating finite-dimensional Markovian optimal control problems in discrete time. The corresponding value functions converge to the value function of the original problem, and we derive an upper bound on the discretisation error or, equivalently, a worst-case estimate for the rate of convergence.
10 citations
Cites methods from "Discretisation of stochastic contro..."
...The techniques used in [22] and related work for the proofs of convergence are based on weak convergence of measures; they can be extended to cover control problems with delay, see [11, 20]....
[...]
Posted Content•
TL;DR: In this article, the authors study stochastic delay differential equations (SDDE) where the coefficients depend on the moving averages of the state process and provide sufficient conditions under which a linear path functional of the solution of a SDDE admits a finite-dimensional Markovian representation.
Abstract: We study stochastic delay differential equations (SDDE) where the coefficients depend on the moving averages of the state process. As a first contribution, we provide sufficient conditions under which a linear path functional of the solution of a SDDE admits a finite-dimensional Markovian representation. As a second contribution, we show how approximate finite-dimensional Markovian representations may be constructed when these conditions are not satisfied, and provide an estimate of the error corresponding to these approximations. These results are applied to optimal control and optimal stopping problems for stochastic systems with delay.
5 citations
TL;DR: In this paper, the authors employ neural networks for sequence modeling (e.g., recurrent neural networks such as long short-term memory) to parameterize the policy and optimize the objective function.
Abstract: Stochastic control problems with delay are challenging due to the path-dependent feature of the system and thus its intrinsic high dimensions. In this paper, we propose and systematically study deep neural network-based algorithms to solve stochastic control problems with delay features. Specifically, we employ neural networks for sequence modeling (e.g., recurrent neural networks such as long short-term memory) to parameterize the policy and optimize the objective function. The proposed algorithms are tested on three benchmark examples: a linear-quadratic problem, optimal consumption with fixed finite delay, and portfolio optimization with complete memory. Particularly, we notice that the architecture of recurrent neural networks naturally captures the path-dependent feature with much flexibility and yields better performance with more efficient and stable training of the network compared to feedforward networks. The superiority is even evident in the case of portfolio optimization with complete memory, which features infinite delay.
5 citations
References
More filters
Book•
01 Jan 1990
TL;DR: In this article, the authors propose a method for general stochastic integration and local times, which they call Stochastic Differential Equations (SDEs), and expand the expansion of Filtrations.
Abstract: I Preliminaries.- II Semimartingales and Stochastic Integrals.- III Semimartingales and Decomposable Processes.- IV General Stochastic Integration and Local Times.- V Stochastic Differential Equations.- VI Expansion of Filtrations.- References.
5,254 citations
Book•
01 Jan 1999
TL;DR: In this article, the authors consider the problem of deterministic control problems in the context of stochastic control systems and show that the optimal control problem can be formulated in a deterministic manner.
Abstract: 1. Basic Stochastic Calculus.- 1. Probability.- 1.1. Probability spaces.- 1.2. Random variables.- 1.3. Conditional expectation.- 1.4. Convergence of probabilities.- 2. Stochastic Processes.- 2.1. General considerations.- 2.2. Brownian motions.- 3. Stopping Times.- 4. Martingales.- 5. Ito's Integral.- 5.1. Nondifferentiability of Brownian motion.- 5.2. Definition of Ito's integral and basic properties.- 5.3. Ito's formula.- 5.4. Martingale representation theorems.- 6. Stochastic Differential Equations.- 6.1. Strong solutions.- 6.2. Weak solutions.- 6.3. Linear SDEs.- 6.4. Other types of SDEs.- 2. Stochastic Optimal Control Problems.- 1. Introduction.- 2. Deterministic Cases Revisited.- 3. Examples of Stochastic Control Problems.- 3.1. Production planning.- 3.2. Investment vs. consumption.- 3.3. Reinsurance and dividend management.- 3.4. Technology diffusion.- 3.5. Queueing systems in heavy traffic.- 4. Formulations of Stochastic Optimal Control Problems.- 4.1. Strong formulation.- 4.2. Weak formulation.- 5. Existence of Optimal Controls.- 5.1. A deterministic result.- 5.2. Existence under strong formulation.- 5.3. Existence under weak formulation.- 6. Reachable Sets of Stochastic Control Systems.- 6.1. Nonconvexity of the reachable sets.- 6.2. Noncloseness of the reachable sets.- 7. Other Stochastic Control Models.- 7.1. Random duration.- 7.2. Optimal stopping.- 7.3. Singular and impulse controls.- 7.4. Risk-sensitive controls.- 7.5. Ergodic controls.- 7.6. Partially observable systems.- 8. Historical Remarks.- 3. Maximum Principle and Stochastic Hamiltonian Systems.- 1. Introduction.- 2. The Deterministic Case Revisited.- 3. Statement of the Stochastic Maximum Principle.- 3.1. Adjoint equations.- 3.2. The maximum principle and stochastic Hamiltonian systems.- 3.3. A worked-out example.- 4. A Proof of the Maximum Principle.- 4.1. A moment estimate.- 4.2. Taylor expansions.- 4.3. Duality analysis and completion of the proof.- 5. Sufficient Conditions of Optimality.- 6. Problems with State Constraints.- 6.1. Formulation of the problem and the maximum principle.- 6.2. Some preliminary lemmas.- 6.3. A proof of Theorem 6.1.- 7. Historical Remarks.- 4. Dynamic Programming and HJB Equations.- 1. Introduction.- 2. The Deterministic Case Revisited.- 3. The Stochastic Principle of Optimality and the HJB Equation.- 3.1. A stochastic framework for dynamic programming.- 3.2. Principle of optimality.- 3.3. The HJB equation.- 4. Other Properties of the Value Function.- 4.1. Continuous dependence on parameters.- 4.2. Semiconcavity.- 5. Viscosity Solutions.- 5.1. Definitions.- 5.2. Some properties.- 6. Uniqueness of Viscosity Solutions.- 6.1. A uniqueness theorem.- 6.2. Proofs of Lemmas 6.6 and 6.7.- 7. Historical Remarks.- 5. The Relationship Between the Maximum Principle and Dynamic Programming.- 1. Introduction.- 2. Classical Hamilton-Jacobi Theory.- 3. Relationship for Deterministic Systems.- 3.1. Adjoint variable and value function: Smooth case.- 3.2. Economic interpretation.- 3.3. Methods of characteristics and the Feynman-Kac formula.- 3.4. Adjoint variable and value function: Nonsmooth case.- 3.5. Verification theorems.- 4. Relationship for Stochastic Systems.- 4.1. Smooth case.- 4.2. Nonsmooth case: Differentials in the spatial variable.- 4.3. Nonsmooth case: Differentials in the time variable.- 5. Stochastic Verification Theorems.- 5.1. Smooth case.- 5.2. Nonsmooth case.- 6. Optimal Feedback Controls.- 7. Historical Remarks.- 6. Linear Quadratic Optimal Control Problems.- 1. Introduction.- 2. The Deterministic LQ Problems Revisited.- 2.1. Formulation.- 2.2. A minimization problem of a quadratic functional.- 2.3. A linear Hamiltonian system.- 2.4. The Riccati equation and feedback optimal control.- 3. Formulation of Stochastic LQ Problems.- 3.1. Statement of the problems.- 3.2. Examples.- 4. Finiteness and Solvability.- 5. A Necessary Condition and a Hamiltonian System.- 6. Stochastic Riccati Equations.- 7. Global Solvability of Stochastic Riccati Equations.- 7.1. Existence: The standard case.- 7.2. Existence: The case C = 0, S = 0, and Q, G ?0.- 7.3. Existence: The one-dimensional case.- 8. A Mean-variance Portfolio Selection Problem.- 9. Historical Remarks.- 7. Backward Stochastic Differential Equations.- 1. Introduction.- 2. Linear Backward Stochastic Differential Equations.- 3. Nonlinear Backward Stochastic Differential Equations.- 3.1. BSDEs in finite deterministic durations: Method of contraction mapping.- 3.2. BSDEs in random durations: Method of continuation.- 4. Feynman-Kac-Type Formulae.- 4.1. Representation via SDEs.- 4.2. Representation via BSDEs.- 5. Forward-Backward Stochastic Differential Equations.- 5.1. General formulation and nonsolvability.- 5.2. The four-step scheme, a heuristic derivation.- 5.3. Several solvable classes of FBSDEs.- 6. Option Pricing Problems.- 6.1. European call options and the Black--Scholes formula.- 6.2. Other options.- 7. Historical Remarks.- References.
2,364 citations
"Discretisation of stochastic contro..." refers background in this paper
...We consider control problems in the weak formulation (cf. Yong and Zhou, 1999: p. 64)....
[...]
...It then follows from a theorem by Weierstraÿ that Ĵ(ϕ, .) attains its minimum at some point of its compact domain (cf. Yong and Zhou, 1999: p. 65)....
[...]
Book•
01 Jan 2000TL;DR: In this paper, a Markov chain is used to approximate the solution of the optimal stochastic control problem for diffusion, reflected diffusion, or jump-diffusion models, and a general method for obtaining a useful approximation is given.
Abstract: A powerful and usable class of methods for numerically approximating the solutions to optimal stochastic control problems for diffusion, reflected diffusion, or jump-diffusion models is discussed. The basic idea involves uconsistent approximation of the model by a Markov chain, and then solving an appropriate optimization problem for the Murkoy chain model. A general method for obtaining a useful approximation is given. All the standard classes of cost functions can be handled here, for illustrative purposes, discounted and average cost per unit time problems with both reflecting and nonreflecting diffusions are concentrated on. Both the drift and the variance can be controlled. Owing to its increasing importance and to lack of material on numerical methods, an application to the control of queueing and production systems in heavy traffic is developed in detail. The methods of proof of convergence are relatively simple, using only some basic ideas in the theory of weak convergence of a sequence of probabi...
1,767 citations
Book•
01 Jan 1997
1,170 citations
"Discretisation of stochastic contro..." refers background in this paper
...For an exposition of the general theory of SDDEs see Mohammed (1984) or Mao (1997)....
[...]
Book•
01 Jan 1984
639 citations
"Discretisation of stochastic contro..." refers background in this paper
...For an exposition of the general theory of SDDEs see Mohammed (1984) or Mao (1997)....
[...]