Bellman equation

About: Bellman equation is a research topic. Over the lifetime, 5884 publications have been published within this topic receiving 135589 citations.

When Are HJB-Equations in Stochastic Control of Delay Systems Finite Dimensional?

Abstract: We consider optimal control problems where the state X(t) at time t of the system is given by a stochastic differential delay equation. The growth at time t not only depends on the present value X(t), but also on X(t-δ) and some sliding average of previous values. Moreover, this dependence may be nonlinear. Using the dynamic programming principle we derive an associated (finite dimensional) Hamilton-Jacobi-Bellman equation for the value function of such problems. This (finite dimensional) HJB equation has solutions if and only if the coefficients satisfy a particular system of first order PDEs. We introduce viscosity solutions for the type of HJB-equations that we consider, and prove that under certain conditions, the value function is the unique viscosity solution to the HJB-equation. We also give numerical examples for two cases where the HJB-equation reduces to a finite dimensional one.

A primal–dual algorithm for bsdes

Abstract: We generalize the primal–dual methodology, which is popular in the pricing of early-exercise options, to a backward dynamic programming equation associated with time discretization schemes of (reflected) backward stochastic differential equations (BSDEs). Taking as an input some approximate solution of the backward dynamic program, which was precomputed, e.g., by least-squares Monte Carlo, this methodology enables us to construct a confidence interval for the unknown true solution of the time-discretized (reflected) BSDE at time 0. We numerically demonstrate the practical applicability of our method in two 5-dimensional nonlinear pricing problems where tight price bounds were previously unavailable.

Optimization Techniques for State-Constrained Control and Obstacle Problems

Abstract: The design of control laws for systems subject to complex state constraints still presents a significant challenge. This paper explores a dynamic programming approach to a specific class of such problems, that of reachability under state constraints. The problems are formulated in terms of nonstandard minmax and maxmin cost functionals, and the corresponding value functions are given in terms of Hamilton-Jacobi-Bellman (HJB) equations or variational inequalities. The solution of these relations is complicated in general; however, for linear systems, the value functions may be described also in terms of duality relations of convex analysis and minmax theory. Consequently, solution techniques specific to systems with a linear structure may be designed independently of HJB theory. These techniques are illustrated through two examples.

Categories, Relations and Dynamic Programming

Abstract: Dynamic programming is a strategy for solving optimisation problems. In this paper, we show how many problems that may be solved by dynamic programming are instances of the same abstract specification. This specification is phrased using the calculus of relations offered by topos theory. The main theorem underlying dynamic programming can then be proved by straightforward equational reasoning.The generic specification of dynamic programming makes use of higher-order operators on relations, akin to the fold operators found in functional programming languages. In the present context, a data type is modelled as an initial F-algebra, where F is an endofunctor on the topos under consideration. The mediating arrows from this initial F-algebra to other F-algebras are instances of fold – but only for total functions. For a regular category e, it is possible to construct a category of relations Rel(e). When a functor between regular categories is a so-called relator, it can be extended (in some canonical way) to a functor between the corresponding categories of relations. Applied to an endofunctor on a topos, this process of extending functors preserves initial algebras, and hence fold can be generalised from functions to relations.It is well-known that the use of dynamic programming is governed by the principle of optimality. Roughly, the principle of optimality says that an optimal solution is composed of optimal solutions to subproblems. In a first attempt, we formalise the principle of optimality as a distributivity condition. This distributivity condition is elegant, but difficult to check in practice. The difficulty arises because we consider minimum elements with respect to a preorder, and therefore minimum elements are not unique. Assuming that we are working in a Boolean topos, it can be proved that monotonicity implies distributivity, and this monotonicity condition is easy to verify in practice.