Showing papers on "Bellman equation published in 1987"

Dynamic programming : applications to agriculture and natural resources

Abstract: I Introduction.- 1 The Management of Agricultural and Natural Resource Systems.- 1.1 The Nature of Agricultural and Natural Resource Problems.- 1.2 Management Techniques Applied to Resource Problems.- 1.2.1 Farm management.- 1.2.2 Forestry management.- 1.2.3 Fisheries management.- 1.3 Control Variables in Resource Management.- 1.3.1 Input decisions.- 1.3.2 Output decisions.- 1.3.3 Timing and replacement decisions.- 1.4 A Simple Derivation of the Conditions for Intertemporal Optimality.- 1.4.1 The general resource problem without replacement.- 1.4.2 The general resource problem with replacement.- 1.5 Numerical Dynamic Programming.- 1.5.1 Types of resource problem.- 1.5.2 Links with simulation.- 1.5.3 Solution procedures.- 1.5.4 Types of dynamic programming problem.- 1.6 References.- 1.A Appendix: A Lagrangian Derivation of the Discrete Maximum Principle.- 1.B Appendix: A Note on the Hamiltonian Used in Control Theory.- II The Methods of Dynamic Programming.- 2 Introduction to Dynamic Programming.- 2.1 Backward Recursion Applied to the General Resource Problem.- 2.2 The Principle of Optimality.- 2.3 The Structure of Dynamic Programming Problems.- 2.4 A Numerical Example.- 2.5 Forward Recursion and Stage Numbering.- 2.6 A Simple Crop-irrigation Problem.- 2.6.1 The formulation of the problem.- 2.6.2 The solution procedure.- 2.7 A General-Purpose Computer Program for Solving Dynamic Programming Problems.- 2.7.1 An introduction to the GPDP programs.- 2.7.2 Data entry using DPD.- 2.7.3 Using GPDP to solve the least-cost network problem.- 2.7.4 Using GPDP to solve the crop-irrigation problem.- 2.8 References.- 3 Stochastic and Infinite-Stage Dynamic Programming.- 3.1 Stochastic Dynamic Programming.- 3.1.1 Formulation of the stochastic problem.- 3.1.2 A stochastic crop-irrigation problem.- 3.2 Infinite-stage Dynamic Programming for Problems With Discounting.- 3.2.1 Formulation of the problem.- 3.2.2 Solution by value iteration.- 3.2.3 Solution by policy iteration.- 3.3 Infinite-stage Dynamic Programming for Problems Without Discounting.- 3.3.1 Formulation of the problem.- 3.3.2 Solution by value iteration.- 3.3.3 Solution by policy iteration.- 3.4 Solving Infinite-stage Problems in Practice.- 3.4.1 Applications to agriculture and natural resources.- 3.4.2 The infinite-stage crop-irrigation problem.- 3.4.3 Solution to the crop-irrigation problem with discounting.- 3.4.4 Solution to the crop-irrigation problem without discounting.- 3.5 Using GPDP to Solve Stochastic and Infinite-stage Problems.- 3.5.1 Stochastic problems.- 3.5.2 Infinite-stage problems.- 3.6 References.- 4 Extensions to the Basic Formulation.- 4.1 Linear Programming for Solving Stochastic, Infinite-stage Problems.- 4.1.1 Linear programming formulations of problems with discounting.- 4.1.2 Linear programming formulations of problems without discounting.- 4.2 Adaptive Dynamic Programming.- 4.3 Analytical Dynamic Programming.- 4.3.1 Deterministic, quadratic return, linear transformation problems.- 4.3.2 Stochastic, quadratic return, linear transformation problems.- 4.3.3 Other problems which can be solved analytically.- 4.4 Approximately Optimal Infinite-stage Solutions.- 4.5 Multiple Objectives.- 4.5.1 Multi-attribute utility.- 4.5.2 Risk.- 4.5.3 Problems involving players with conflicting objectives.- 4.6 Alternative Computational Methods.- 4.6.1 Approximating the value function in continuous form.- 4.6.2 Alternative dynamic programming structures.- 4.6.3 Successive approximations around a nominal control policy.- 4.6.4 Solving a sequence of problems of reduced dimension.- 4.6.5 The Lagrange multiplier method.- 4.7 Further Information on GPDP.- 4.7.1 The format for user-written data files.- 4.7.2 Redimensioning arrays in FDP and IDP.- 4.8. References.- 4.A Appendix: The Slope and Curvature of the Optimal Return Function Vi{xi}.- III Dynamic Programming Applications to Agriculture.- 5 Scheduling, Replacement and Inventory Management.- 5.1 Critical Path Analysis.- 5.1.1 A farm example.- 5.1.2 Solution using GPDP.- 5.1.3 Selected applications.- 5.2 Farm Investment Decisions.- 5.2.1 Optimal tractor replacement.- 5.2.2 Formulation of the problem without tax.- 5.2.3 Formulation of the problem with tax.- 5.2.4 Discussion.- 5.2.5 Selected applications.- 5.3 Buffer Stock Policies.- 5.3.1 Stochastic yields: planned production and demand constant.- 5.3.2 Stochastic yields and demand: planned production constant.- 5.3.3 Planned production a decision variable.- 5.3.4 Selected applications.- 5.4 References.- 6 Crop Management.- 6.1 The Crop Decision Problem.- 6.1.1 States.- 6.1.2 Stages.- 6.1.3 Returns.- 6.1.4 Decisions.- 6.2 Applications to Water Management.- 6.3 Applications to Pesticide Management.- 6.4 Applications to Crop Selection.- 6.5 Applications to Fertilizer Management.- 6.5.1 Optimal rules for single-period carryover functions.- 6.5.2 Optimal rules for a multiperiod carryover function.- 6.5.3 A numerical example.- 6.5.4 Extensions.- 6.6 References.- 7 Livestock Management.- 7.1 Livestock Decision Problems.- 7.2 Livestock Replacement Decisions.- 7.2.1 Types of problem.- 7.2.2 Applications to dairy cows.- 7.2.3 Periodic revision of estimated yield potential.- 7.3 Combined Feeding and Replacement Decisions.- 7.3.1 The optimal ration sequence: an example.- 7.3.2 Maximizing net returns per unit of time.- 7.3.3 Replacement a decision option.- 7.4 Extensions to the Combined Feeding and Replacement Problem.- 7.4.1 The number of livestock.- 7.4.2 Variable livestock prices.- 7.4.3 Stochastic livestock prices.- 7.4.4 Ration formulation systems.- 7.5 References.- 7.A Appendix: Yield Repeatability and Adaptive Dynamic Programming.- 7.A.1 The concept of yield repeatability.- 7.A.2 Repeatability of average yield.- 7.A.3 Expected yield given average individual and herd yields.- 7.A.4 Yield probabilities conditional on recorded average yields.- IV Dynamic Programming Applications to Natural Resources.- 8 Land Management.- 8.1 The Theory of Exhaustible Resources.- 8.1.1 The simple theory of the mine.- 8.1.2 Risky possession and risk aversion.- 8.1.3 Exploration.- 8.2 A Pollution Problem.- 8.2.1 Pollution as a stock variable.- 8.2.2 A numerical example.- 8.3 Rules for Making Irreversible Decisions Under Uncertainty.- 8.3.1 Irreversible decisions and quasi-option value.- 8.3.2 A numerical example.- 8.3.3 The discounting procedure.- 8.4 References.- 9 Forestry Management.- 9.1 Problems in Forestry Management.- 9.2 The Optimal Rotation Period.- 9.2.1 Deterministic problems.- 9.2.2 Stochastic problems.- 9.2.3 A numerical example of a combined rotation and protection problem.- 9.3 The Optimal Rotation and Thinning Problem.- 9.3.1 Stage intervals.- 9.3.2 State variables.- 9.3.3 Decision variables.- 9.3.4 Objective function.- 9.4 Extensions.- 9.4.1 Allowance for distributions of tree sizes and ages.- 9.4.2 Alternative objectives.- 9.5 References.- 10 Fisheries Management.- 10.1 The Management Problem.- 10.2 Modelling Approaches.- 10.2.1 Stock dynamics.- 10.2.2 Stage return.- 10.2.3 Developments in analytical modelling.- 10.3 Analytical Dynamic Programming Approaches.- 10.3.1 Deterministic results.- 10.3.2 Stochastic results.- 10.4 Numerical Dynamic Programming Applications.- 10.4.1 An application to the southern bluefin tuna fishery.- 10.4.2 A review of applications.- 10.5 References.- V Conclusion.- 11 The Scope for Dynamic Programming Applied to Resource Management.- 11.1 Dynamic Programming as a Method of Conceptualizing Resource Problems.- 11.2 Dynamic Programming as a Solution Technique.- 11.3 Applications to Date.- 11.4 Expected Developments.- 11.5 References.- Appendices.- A1 Coding Sheets for Entering Data Using DPD.- A2 Program Listings.- A2.1 Listing of DPD.- A2.2 Listing of FDP.- A2.3 Listing of IDP.- A2.4 Listing of DIM.- Author Index.

An optimal stochastic production planning problem with randomly fluctuating dem and

Abstract: This paper considers an infinite horizon stochastic production planning problem with demand assumed to be a continuous-time Markov chain. The problems with control (production) and state (inventory) constraints are treated. It is shown that a unique optimal feedback solution exists, after first showing that convex viscosity solutions to the associated dynamic programming equation are continuously differentiable.

Optimal trajectories associated to a solution of contingent Hamilton-Jacobi equation

Abstract: In this paper we study the existence of optimal trajectories associated with a generalized solution to Hamilton-Jacobi-Bellman equation arising in optimal control. In general, we cannot expect such solutions to be differentiable. But, in a way analogous to the use of distributions in PDE, we replace the usual derivatives with "contingent epiderivatives" and the Hamilton-Jacobi equation by two "contingent Hamilton-Jacobi inequalities". We show that the value function of an optimal control problem verifies these "contingent inequalities". Our approach allows the following three results: (a) The upper semicontinuous solutions to contingent inequalities are monotone along the trajectories of the dynamical system. (b) With every continuous solution V of the contingent inequalities, we can associate an optimal trajectory along which V is constant. (c) For such solutions, we can construct optimal trajectories through the corresponding optimal feedback. They are also "viscosity solutions" of a Hamilton-Jacobi equation. Finally we discuss the link of viscosity solutions with Clarke's approach to the Hamilton-Jacobi equation.

Minimizing expected makespans on uniform processor systems

Abstract: We study the problem of scheduling n given jobs on m uniform processors to minimize expected makespan (maximum finishing time). Job execution times are not known in advance, but are known to be exponentially distributed, with identical rate parameters depending solely on the executing processor. For m = 2 and 3, we show that there exist optimal scheduling rules of a certain threshold type, and we show how the required thresholds can be easily determined. We conjecture that similar threshold rules suffice for m > 3 but are unable to prove this. However, for m > 3 we do obtain a general bound on problem size that permits Bellman equations to be used to construct an optimal scheduling rule for any given set of m rate parameters, with the memory required to represent that scheduling rule being independent of the number of remaining jobs.

Singular stochastic control and optimal stopping

Abstract: The problem of controlling aBrownian motion by a process of bounded variation on a finite time-horizon, with convex, symmetric running and terminal state costs and linear cost of control, is related to a problem of optimal stopping for Brownian motion, with absorption at the origin. The connection between the two problems is established by probabilistic methods. First we establish an analogous relationship between a modified control problem, where we impose a reflecting barrier at the origin, and the aforementioned stopping problem. The remaining problem is then to show that the reflected and non-reflected control problems have the same value function. This is done by an application of the Tanaka-Meyer formula for semimartingales, which enables us to represent the absolute value of a state process in the non-reflected problem as a state process in the weak sense, we show that the converse is also true, in a certain sense, when the control law is given by a bounded Markovian drift. Fundamental to our appro...

On the robustness of optimal regulators for nonlinear discrete-time systems

Abstract: In this paper the robustness of nonlinear discrete-time systems is analyzed. The nominal plant is supposed to be controlled by means of a feedback control law which is optimal with respect to some given criterion. The robustness of the closed-loop system is studied for two different classes of perturbations in the control law, which are called gain and additive nonlinear perturbations. The results are entirely based on the existence of a stationary solution of the dynamic programming equation (DPE), which provides directly a Lyapunov function associated to the closed-loop system. The convexity of that solution and the use of the Taylor formula appear to be the key to establish the robustness properties of the nominal plant. Two examples are solved in order to show an interesting fact: the existence of a compromise between the robustness of the system subjected to the two different classes of perturbations.

Martingale problems for controlled processes

Abstract: Martingale problems provide a powerful method for characterizing Markov processes. Stochastic control problems can also be formulated naturally as martingale problems (see for example Fleming (1983)), and our goal here is to exploit this formulation to give a general existence theorem for optimal solutions of a stochastic control problem in the finite time horizon and discounted cases (Sections 1,3), to construct the Nisio semigroup (Section 2), and to give conditions in terms of the generator of the Nisio semigroup under which the optimal solution can be approximated by solutions in which the control is piecewise constant (Sections 2,3). The proof of this last result involves showing that the value function of an appropriately defined discounted control problem is in the domain of the generator of the Nisio semigroup.

Some properties of constrained viscosity solutions of Hamilton-Jacobi Bellman equations

Abstract: We consider an optimal control problem with space constraints and we show some properties of the optimal cost function. Our main result is the equality between the value functions of four optimal control problems and the constrained viscosity solution of the Hamilton–Jacobi–Bellman equation.

Sur l'Analyse Numérique des Equations de Hamilton-Jacobi-Bellman

Abstract: Convergence dans la norme L ∞ de l'approximation par elements finis de l'equation d'Hamilton-Jacobi-Bellman. Application de la methode des sous-solutions et de la notion de regularite discrete

Arbitrary state semi-Markov decision processes

Abstract: We study the problem of maximizing the expectation of the total discounted rewards in semi-Markov decision processes with unbounded rewards. Both the state space and the action space are uncountable. We give a short and simple proof of the Bellman's principle of optimality and present sufficient conditions for the existence of an optimal stationary policy.

Optimal switching for systems governed by nonlinear evolution equations

Abstract: Optimal switching for a nonlinear evolution equation is considered. A type of approximating problems with value functions {uu0

An asymptotic formula for solutions of Hamilton-Jacobi-Bellman equations

Abstract: On demontre que la solution de l'equation de Hamilton-Jacobi-Bellman converge exponentiellement vers 0 quand e→0 avec un taux de decroissance {−I(x)+0(1)}/e et on donne une formule de representation pour I(x)

Survey of input optimization 1

Abstract: Input optimization is a conceptually new level of optimization, at which the mathematical programming model, rather than a usual program, is optimized. This is achieved by optimizing the optimal value function by stable perturbations of the parameters (input). This paper is a survey of the basic ideas in finite-dimensional input optimization for both single- and multi-objective linear and convex models. The theory is general enough not to require extraneous assumptions, such as linear independence of the gradients or Slater's Condition. On the other hand, it is of a constructive nature that takes it possible to formulate numerical methods for computing an “optimal input”and the corresponding “optimal realization” of the mathematical model. Many results from the “usual” mathematical programming and sensitivity analysis follow as special cases. The paper contains many illustrative examples and lists a wide range of (potential) applications of input optimization from long-range planning for an economic syste...

Optimal locally absolutely continuous change of measure. finite set of decisions. part ii:optimization problems

Abstract: This article continues Part I. Optimality conditions are studied for a control problem, where the control choice implies locally absolutely continuous transformation of some initial measure. Attention is focused on the derivation of a stochastic differential equation for the value process which is a non-linear analogue of the inverse Girsanov transformation standing for Bellman's equation. In. particular, application of this equation gives a description of the extreme points of the space of controls. The questions of sufficiency of partial information and equivalent transformations of the optimization problem are also studied

Smoothness near the boundary of the solutions of the elliptic Bellman equations

Abstract: One considers Bellman's elliptic equation with constant coefficients and zero boundary values on a plane part of the boundary. In this case one gives a simplified proof of N. V. Krylov's result regarding the boundary estimates of the Holder constants of the second derivatives of the solutions of the Bellman equation.

Effort fluctuations in a harvest model with random prices

Abstract: The effect of a single increase in the unit price of biomass on the optimal harvest policy for an exploited population is studied numerically. The price of a unit is assumed constant until a random time, when the price increases by a given amount. The optimal policy corresponding to the expected return is computed from the Bellman equation of dynamic programming. The results are compared with a model in which prices remain constant as well as a “well-timed” model in which the price increases at the expected increase time of the random case. Both optimal expected return and optimal policy computed from the deterministic models may differ substantially from that calculated from the random model, particularly if marginal costs are large. The emphasis is on numerical computation.

Stochastic control of geometric processes

Abstract: Stochastic optimization of semimartingales which permit a dynamic description, like a stochastic differential equation, leads normally to dynamic programming procedures. The resulting Bellman equation is often of a very general nature, and analytically hard to solve. The models in the present paper are formulated in terms of the relative change, and the optimality criterion is to maximize the expected rate of growth. We show how this can be done in a simple way, where we avoid using the Bellman equation. An application is indicated.

Stochastic Task Selection and Renewable Resource Allocation

Abstract: This paper studies a class of renewable resource allocation problems for the processing of dynamically arriving tasks with deterministic deadlines. This class of problems has many applications, however, it conforms to neither the standard resource allocation model nor the standard optimal control model. A new problem formulation has to be developed and analyzed to provide a satisfactory solution. In this paper, a new formulation is presented. By augmenting the state variabls, the problem is converted into a Markovian decision problem. It can then be treated, at least in principle, by using the stochastic dynamic programming (SDP) method. However, since the system dynamics involves the evolution of sets (of tasks and resources), the implementation of the dynamic programming equation is by no means straightforward. For a problem with infinite planning horizon, the optimal strategy is shown to be stationary under mild conditions. An SDP algorithm based on a successive approximation technique is developed to obtain the optimal stationary strategy. The implementation of the algorithm employs a special coding scheme to handle set variables, and utilizes a dominance property for computational efficiency. Effects of key system parameters on optimal decisions are investigated and analyzed through numerical examples. As the computational complexity of the algorithm is of exponential increase, practical applications of the algorithm is limited to problems of moderate size. Two heuristic rules are therefore investigated and compared to the optimal policy.

On the computation of optimal control policies of energy production systems with random perturbations

Abstract: We propose a numerical procedure to compute the value function and the optimal policies of a stochastic model of an energy production system.

Average Cost Optimality in Inventory Models with Markovian Demands

Abstract: This paper is concerned with long-run average cost minimization of a stochastic inventory problem with Markovian demand, fixed ordering cost, and convex surplus cost. The states of the Markov chain represent different possible states of the environment. Using a vanishing discount approach, a dynamic programming equation and the corresponding verification theorem are established. Finally, the existence of an optimal state-dependent (s, S) policy is proved.

Computable optimal value bounds for generalized convex programs

Abstract: It has been shown by Fiacco that convexity or concavity of the optimal value of a parametric nonlinear programming problem can readily be exploited to calculate global parametric upper and lower bounds on the optimal value function. The approach is attractive because it involves manipulation of information normally required to characterize solution optimality. A procedure is briefly described for calculating and improving the bounds as well as its extensions to generalized convex and concave functions. Several areas of applications are also indicated.

An Optimal Stochastic Production Planning Problem with Randomly Fluctuating Demand

Abstract: This paper considers an infinite horizon stochastic production planning problem with demand assumed to be a continuous-time Markov chain. The problems with control (production) and state (inventory) constraints are treated. It is shown that a unique optimal feedback solution exists, after first showing that convex viscosity solutions to the associated dynamic programming equation are continuously differentiable.

Discretization of the Bellman equation and the corresponding stochastic control problem

Abstract: IXTRODUCTIOF u (x ) = I n f J x ( z ( . ) ) ( 1 . 1 1 ) Z(.) We review some o f t h e c u r r e n t d i s c r e t i z a t i o n s c h e mes u s e d i n t h e c o t e x t of Bellman equations. Our main 1 . 2 . S o t a t i o n f r d i s c r e t i z a t i o n o b j e c t i v e i s t o t r e a t c a s e s when the Hami l tonian has Le t h be a small parameter such tha t h = 1 / R , X quadra t ic g rowth and we s h a l l compare wi th the more c las in teger t ending to + * We consider the grid O f size s i c a l s i t u a t i o n when the Hami l tonian has l inear g rowth . h , i * e * the Points 2% Y j = O , * . * >s* We s h a l l d i s c u s s t h e a s s o c i a t e d s t o c h a s t i c o n t r o l p r o Let blem i n d i s c r e t e t i m e when t h e r e i s one. The a n a l y t i c H = s e t of s t e p f u n c t i o n s w h i c h a r e c o n s t a n t on par t of t h e p a p e r r e l i e s on r e s u l t s of R. GLOWINSKI and t h e A. h i n t e r v a l s ( j h , ( j + l ) h ) , j = O ,... ,5-1.