Bellman equation

About: Bellman equation is a research topic. Over the lifetime, 5884 publications have been published within this topic receiving 135589 citations.

Finite Difference Methods for Mean Field Games

Abstract: Mean field type models describing the limiting behavior of stochastic differential game problems as the number of players tends to + ∞, have been recently introduced by J-M. Lasry and P-L. Lions. They may lead to systems of evolutive partial differential equations coupling a forward Bellman equation and a backward Fokker–Planck equation. The forward-backward structure is an important feature of this system, which makes it necessary to design new strategies for mathematical analysis and numerical approximation. In this survey, several aspects of a finite difference method used to approximate the previously mentioned system of PDEs are discussed, including: existence and uniqueness properties, a priori bounds on the solutions of the discrete schemes, convergence, and algorithms for solving the resulting nonlinear systems of equations. Some numerical experiments are presented. Finally, the optimal planning problem is considered, i.e. the problem in which the positions of a very large number of identical rational agents, with a common value function, evolve from a given initial spatial density to a desired target density at the final horizon time.

Optimal Control of Jump-Markov Processes and Viscosity Solutions

Abstract: We investigate the Bellman equation that arises in the optimal control of Markov processes This is a fully nonlinear integro-differential equation The notion of viscosity solutions is introduced and then existence and uniqueness results are obtained Also, the connection between the optimal control problem and the Bellman equation is developed

Resource Allocation Optimization for Delay-Sensitive Traffic in Fronthaul Constrained Cloud Radio Access Networks

Abstract: The cloud radio access network (C-RAN) provides high spectral and energy efficiency performances, low expenditures, and intelligent centralized system structures to operators, which have attracted intense interests in both academia and industry. In this paper, a hybrid coordinated multipoint transmission (H-CoMP) scheme is designed for the downlink transmission in C-RANs and fulfills the flexible tradeoff between cooperation gain and fronthaul consumption. The queue-aware power and rate allocation with constraints of average fronthaul consumption for the delay-sensitive traffic are formulated as an infinite horizon constrained partially observed Markov decision process, which takes both the urgent queue state information and the imperfect channel state information at transmitters (CSIT) into account. To deal with the curse of dimensionality involved with the equivalent Bellman equation, the linear approximation of postdecision value functions is utilized. A stochastic gradient algorithm is presented to allocate the queue-aware power and transmission rate with H-CoMP, which is robust against unpredicted traffic arrivals and uncertainties caused by the imperfect CSIT. Furthermore, to substantially reduce the computing complexity, an online learning algorithm is proposed to estimate the per-queue postdecision value functions and update the Lagrange multipliers. The simulation results demonstrate performance gains of the proposed stochastic gradient algorithms and confirm the asymptotical convergence of the proposed online learning algorithm.

Incremental methods for computing bounds in partially observable Markov decision processes

Abstract: Partially observable Markov decision processes (POMDPs) allow one to model complex dynamic decision or control problems that include both action outcome uncertainty and imperfect observability. The control problem is formulated as a dynamic optimization problem with a value function combining costs or rewards from multiple steps. In this paper we propose, analyse and test various incremental methods for computing bounds on the value function for control problems with infinite discounted horizon criteria. The methods described and tested include novel incremental versions of grid-based linear interpolation method and simple lower bound method with Sondik's updates. Both of these can work with arbitrary points of the belief space and can be enhanced by various heuristic point selection strategies. Also introduced is a new method for computing an initial upper bound - the fast informed bound method. This method is able to improve significantly on the standard and commonly used upper bound computed by the MDP-based method. The quality of resulting bounds are tested on a maze navigation problem with 20 states, 6 actions and 8 observations.