Bellman equation

About: Bellman equation is a research topic. Over the lifetime, 5884 publications have been published within this topic receiving 135589 citations.

Polynomial-Time Algorithms for Stochastic Uncapacitated Lot-Sizing Problems

Abstract: In 1958, Wagner and Whitin published a seminal paper on the deterministic uncapacitated lot-sizing problem, a fundamental model that is embedded in many practical production planning problems. In this paper, we consider a basic version of this model in which problem parameters are stochastic: the stochastic uncapacitated lot-sizing problem. We define the production-path property of an optimal solution for our model and use this property to develop a backward dynamic programming recursion. This approach allows us to show that the value function is piecewise linear and right continuous. We then use these results to show that a full characterization of the optimal value function can be obtained by a dynamic programming algorithm in polynomial time for the case that each nonleaf node contains at least two children. Moreover, we show that our approach leads to a polynomial-time algorithm to obtain an optimal solution to any instance of the stochastic uncapacitated lot-sizing problem, regardless of the structur...

A variational formulation for higher order macroscopic traffic flow models of the GSOM family

Abstract: The GSOM (Generic second order modelling) family of traffic flow models combines the LWR model with dynamics of driver-specific attributes and can be expressed as a system of conservation laws. The object of the paper is to show that a proper Lagrangian formulation of the GSOM model can be recast as a Hamilton-Jacobi equation, the solution of which can be expressed as the value function of an optimal control problem. This value function is interpreted as the position of vehicles, and the optimal trajectories of the optimal control formulation can be identified with the characteristics. Further the paper analyzes the initial and boundary conditions, proposes a generalization of the inf-morphism and the Lax-Hopf formulas to the GSOM model, and considers numerical aspects.

Portfolio optimization and stochastic volatility asymptotics

Abstract: We study the Merton portfolio optimization problem in the presence of stochastic volatility using asymptotic approximations when the volatility process is characterized by its time scales of fluctuation. This approach is tractable because it treats the incomplete markets problem as a perturbation around the complete market constant volatility problem for the value function, which is well-understood. When volatility is fast mean-reverting, this is a singular perturbation problem for a nonlinear Hamilton-JacobiBellman PDE, while when volatility is slowly varying, it is a regular perturbation. These analyses can be combined for multifactor multiscale stochastic volatility models. The asymptotics shares remarkable similarities with the linear option pricing problem, which follows from some new properties of the Merton risk-tolerance function. We give examples in the family of mixture of power utilities and also we use our asymptotic analysis to suggest a “practical” strategy which does not require tracking the fast-moving volatility. In this paper, we present formal derivations of asymptotic approximations, and we provide a convergence proof in the case of power utility and single factor stochastic volatility. We assess our approximation in a particular case where there is an explicit solution.

A Connection Between the Maximum Principle and Dynamic Programming for Constrained Control Problems

Abstract: We consider the Mayer optimal control problem with dynamics given by a nonconvex differential inclusion, whose trajectories are constrained to a closed set and obtain necessary optimality conditions in the form of the maximum principle together with a relation between the costate and the value function. This additional relation is applied in turn to show that the maximum principle is nondegenerate. We also provide a sufficient condition for the normality of the maximum\break principle. To derive these results we use convex linearizations of differential inclusions and convex linearizations of constraints along optimal trajectories. Then duality theory of convex analysis is applied to derive necessary conditions for optimality. In this way we extend the known relations between the maximum principle and dynamic programming from the unconstrained problems to the constrained case.

Existence of Value and Randomized Strategies in Zero-Sum Discrete-Time Stochastic Dynamic Games

Abstract: Two players with conflicting objectives are simultaneously controlling a discrete-time stochastic system. The goal of this paper is to analyze such zero-sum, discrete-time, stochastic systems when the two players are allowed to use randomized strategies.Previous results have been restricted to systems with finite or compact state spaces. Such restrictions are usually untenable from the point of view of applications, since many applications frequently use either the integers or $\mathbb{R}^n $ as a state space. Our results are proved for complete, separable, metric spaces which are very useful for applications.All previously known results emerge as special cases of our results. In addition, a variety of conjectures and open problems are resolved regarding the existence of a value function, its properties such as Borel measurability or continuity, and the existence for either or both players of optimal or $\varepsilon $-optimal stationary strategies.