Bellman equation

About: Bellman equation is a research topic. Over the lifetime, 5884 publications have been published within this topic receiving 135589 citations.

Convergent Numerical Scheme for Singular Stochastic Control with State Constraints in a Portfolio Selection Problem

Abstract: We consider a singular stochastic control problem with state constraints that arises in problems of optimal consumption and investment under transaction costs. Numerical approximations for the value function using the Markov chain approximation method of Kushner and Dupuis are studied. The main result of the paper shows that the value function of the Markov decision problem (MDP) corresponding to the approximating controlled Markov chain converges to that of the original stochastic control problem as various parameters in the approximation approach suitable limits. All our convergence arguments are probabilistic; the main assumption that we make is that the value function be finite and continuous. In particular, uniqueness of the solutions of the associated HJB equations is neither needed nor available (in the generality under which the problem is considered). Specific features of the problem that make the convergence analysis nontrivial include unboundedness of the state and control space and the cost function; degeneracies in the dynamics; mixed boundary (Dirichlet-Neumann) conditions; and presence of both singular and absolutely continuous controls in the dynamics. Finally, schemes for computing the value function and optimal control policies for the MDP are presented and illustrated with a numerical study.

Some Results on Risk-Sensitive Control with Full Observation

Abstract: The Bellman equation of the risk-sensitive control problem with full observation is considered. It appears as an example of a quasi-linear parabolic equation in the whole space, and fairly general growth assumptions with respect to the space variable x are permitted. The stochastic control problem is then solved, making use of the analytic results. The case of large deviation with small noises is then treated, and the limit corresponds to a differential game.

The Hamilton-Jacobi-Bellman equation for time-optimal control

Abstract: In this paper several assertions concerning viscosity solutions of the Hamilton–Jacobi–Bellman equation for the optimal control problem of steering a system to zero in minimal time are proved. First two rather general uniqueness theorems are established, asserting that any positive viscosity solution of the HJB equation must, in fact, agree with the minimal time function near zero; if also a boundary condition introduced by Bardi [SIAM J Control Optim., 27 (1988), pp. 776–785] is satisfied, then the agreement is global. Additionally, the Holder continuity of any subsolution of the HJB equation is proved in the case where the related dynamics satisfy a Hormander-type hypothesis. This last assertion amounts to a “half-derivative” analogue of a theorem of Crandall and Lions [Traps. Amer. Math. Soc., 277 (1.983), pp. 1–42] concerning Lipschitz viscosity solutions.

Approximate Linear Programming for First-order MDPs

Abstract: We introduce a new approximate solution technique for first-order Markov decision processes (FOMDPs). Representing the value function linearly w.r.t. a set of first-order basis functions, we compute suitable weights by casting the corresponding optimization as a first-order linear program and show how off-the-shelf theorem prover and LP software can be effectively used. This technique allows one to solve FOMDPs independent of a specific domain instantiation; furthermore, it allows one to determine bounds on approximation error that apply equally to all domain instantiations. We apply this solution technique to the task of elevator scheduling with a rich feature space and multi-criteria additive reward, and demonstrate that it outperforms a number of intuitive, heuristicallyguided policies.