Bellman equation

About: Bellman equation is a research topic. Over the lifetime, 5884 publications have been published within this topic receiving 135589 citations.

Reflected solutions of backward SDE's, and related obstacle problems for PDE's

Abstract: We study reflected solutions of one-dimensional backward stochastic differential equations. The “reflection” keeps the solution above a given stochastic process. We prove uniqueness and existence both by a fixed point argument and by approximation via penalization. We show that when the coefficient has a special form, then the solution of our problem is the value function of a mixed optimal stopping–optimal stochastic control problem. We finally show that, when put in a Markovian framework, the solution of our reflected BSDE provides a probabilistic formula for the unique viscosity solution of an obstacle problem for a parabolic partial differential equation.

Regression methods for pricing complex American-style options

Abstract: We introduce and analyze a simulation-based approximate dynamic programming method for pricing complex American-style options, with a possibly high-dimensional underlying state space. We work within a finitely parameterized family of approximate value functions, and introduce a variant of value iteration, adapted to this parametric setting. We also introduce a related method which uses a single (parameterized) value function, which is a function of the time-state pair, as opposed to using a separate (independently parameterized) value function for each time. Our methods involve the evaluation of value functions at a finite set, consisting of "representative" elements of the state space. We show that with an arbitrary choice of this set, the approximation error can grow exponentially with the time horizon (time to expiration). On the other hand, if representative states are chosen by simulating the state process using the underlying risk-neutral probability distribution, then the approximation error remains bounded.

Optimal control of production rate in a failure prone manufacturing system

Abstract: We address the problem of controlling the production rate of a failure prone manufacturing system so as to minimize the discounted inventory, cost, where certain cost rates are specified for both positive and negative inventories, and there is a constant demand rate for the commodity produced. The underlying theoretical problem is the optimal control of a continuous-time system with jump Markov disturbances, with an infinite horizon discounted cost criterion. We use two complementary approaches. First, proceeding informally, and using a combination of stochastic coupling, linear system arguments, stable and unstable eigenspaces, renewal theory, parametric optimization, etc., we arrive at a conjecture for the optimal policy. Then we address the previously ignored mathematical difficulties associated with differential equations with discontinuous right-hand sides, singularity of the optimal control problem, smoothness, and validity of the dynamic programming equation, etc., to give a rigorous proof of optimality of the conjectured policy. It is hoped that both approaches will find uses in other such problems also. We obtain the complete solution and show that the optimal solution is simply characterized by a certain critical number, which we call the optimal inventory level. If the current inventory level exceeds the optimal, one should not produce at all; if less, one should produce at the maximum rate; while if exactly equal, one should produce exactly enough to meet demand. We also give a simple explicit formula for the optimal inventory level.

A quartet of semigroups for model specification, robustness, prices of risk, and model detection

Abstract: A representative agent fears that his model, a continuous time Markov process with jump and diffusion components, is misspecified and therefore uses robust control theory to make decisions. Under the decision maker’s approximating model, cautious behavior puts adjustments for model misspecification into market prices for risk factors. We use a statistical theory of detection to quantify how much model misspecification the decision maker should fear, given his historical data record. A semigroup is a collection of objects connected by something like the law of iterated expectations. The law of iterated expectations defines the semigroup for a Markov process, while similar laws define other semigroups. Related semigroups describe (1) an approximating model; (2) a model misspecification adjustment to the continuation value in the decision maker’s Bellman equation; (3) asset prices; and (4) the behavior of the model detection statistics that we use to calibrate how much robustness the decision maker prefers. Semigroups 2, 3, and 4 establish a tight link between the market price of uncertainty and a bound on the error in statistically discriminating between an approximating and a worst case model. (JEL: C00, D51, D81, E1, G12)