Bellman equation

About: Bellman equation is a research topic. Over the lifetime, 5884 publications have been published within this topic receiving 135589 citations.

Optimality of Reinforcement Learning Algorithms with Linear Function Approximation

Abstract: There are several reinforcement learning algorithms that yield approximate solutions for the problem of policy evaluation when the value function is represented with a linear function approximator. In this paper we show that each of the solutions is optimal with respect to a specific objective function. Moreover, we characterise the different solutions as images of the optimal exact value function under different projection operations. The results presented here will be useful for comparing the algorithms in terms of the error they achieve relative to the error of the optimal approximate solution.

A new look at Bellman's principle of optimality

Abstract: New light is shed on Bellman's principle of optimality and the role it plays in Bellman's conception of dynamic programming. It is argued that a failure to recognize the special features of the model in the context of which the principle was stated has resulted in the latter being misconstrued in the dynamic programming literature.

Stochastic optimization theory of backward stochastic differential equations with jumps and viscosity solutions of Hamilton–Jacobi–Bellman equations

Abstract: In this paper we study stochastic optimal control problems with jumps with the help of the theory of Backward Stochastic Differential Equations (BSDEs) with jumps. We generalize the results of Peng [S. Peng, BSDE and stochastic optimizations, in: J. Yan, S. Peng, S. Fang, L. Wu, Topics in Stochastic Analysis, Science Press, Beijing, 1997 (Chapter 2) (in Chinese)] by considering cost functionals defined by controlled BSDEs with jumps. The application of BSDE methods, in particular, the use of the notion of stochastic backward semigroups introduced by Peng in the above-mentioned work allows a straightforward proof of a dynamic programming principle for value functions associated with stochastic optimal control problems with jumps. We prove that the value functions are the viscosity solutions of the associated generalized Hamilton–Jacobi–Bellman equations with integral-differential operators. For this proof, we adapt Peng’s BSDE approach, given in the above-mentioned reference, developed in the framework of stochastic control problems driven by Brownian motion to that of stochastic control problems driven by Brownian motion and Poisson random measure.

Optimal Control Problems of Fully Coupled FBSDEs and Viscosity Solutions of Hamilton--Jacobi--Bellman Equations

Abstract: In this paper we study stochastic optimal control problems of fully coupled forward-backward stochastic differential equations (FBSDEs). The recursive cost functionals are defined by controlled fully coupled FBSDEs. We use a new method to prove that the value functions are deterministic, satisfy the dynamic programming principle, and are viscosity solutions to the associated generalized Hamilton--Jacobi--Bellman (HJB) equations. For this we generalize the notion of stochastic backward semigroup introduced by Peng Topics on Stochastic Analysis, Science Press, Beijing, 1997, pp. 85--138. We emphasize that when $\sigma$ depends on the second component of the solution $(Y, Z)$ of the BSDE it makes the stochastic control much more complicated and has as a consequence that the associated HJB equation is combined with an algebraic equation. We prove that the algebraic equation has a unique solution, and moreover, we also give the representation for this solution. On the other hand, we prove a new local existence...

Symmetry in Markov Decision Processes and its Implications for Single Agent and Multiagent Learning

Abstract: This paper examines the notion of symmetry in Markov decision processes (MDPs). We define symmetry for an MDP and show how it can be exploited for more effective learning in single agent systems as well as multiagent systems and multirobot systems. We prove that if an MDP possesses a symmetry, then the optimal value function andQ function are similarly symmetric and there exists a symmetric optimal policy. If an MDP is known to possess a symmetry, this knowledge can be applied to decrease the number of training examples needed for algorithms like Q learning and value iteration. It can also be used to directly restrict the hypothesis space.