Showing papers on "Bellman equation published in 1988"

Exit Time Problems in Optimal Control and Vanishing Viscosity Method

Abstract: The authors study the connections between deterministic exit time control problems and possibly discontinuous viscosity solutions of a first-order Hamilton-Jacobi (HJ) equation up to the boundary. This equation admits a maximum and a minimum solution that are the value functions associated to stopping time problems on the boundary. When these solutions are equal, they can be obtained through the vanishing viscosity method. Finally, when the HJ equation has a continuous solution, it is proved to be the value function for the first exit time of the domain. It is also the vanishing viscosity limit arising, in particular, in some large deviations problems.

Sensitivity analysis of nonlinear programs and differentiability properties of metric projections

Abstract: This paper is concerned with a study of differentiability properties of the optimal value function and an associated optimal solution of a parametrized nonlinear program. Second order analysis is presented essentially under the Mangasarian-Fromovitz constraint qualification when the corresponding vector of Lagrange multipliers is not necessarily unique. It is shown that under certain regularity conditions the optimal value function possesses second order directional derivatives and the optimal solution mapping is directionally differentiable. The results obtained are applied to an investigation of metric projections in finite-dimensional spaces.

Optimal Control of Jump-Markov Processes and Viscosity Solutions

Abstract: We investigate the Bellman equation that arises in the optimal control of Markov processes This is a fully nonlinear integro-differential equation The notion of viscosity solutions is introduced and then existence and uniqueness results are obtained Also, the connection between the optimal control problem and the Bellman equation is developed

Directional behaviour of optimal solutions in nonlinear mathematical programming

Abstract: This paper is concerned with Holder, Lipschitz and differential properties of the optimal solutions of a nonlinear mathematical programming problem with perturbations in some fixed direction. These properties are obtained with virtually minimal regularity conditions using an approach based on duality and stability. The Holder property is used to obtain the directional derivative for the optimal value function.

Stochastic differential games and viscosity solutions of Isaacs equations

Abstract: Recently P. L. Lions has demonstrated the connection between the value function of stochastic optimal control and a viscosity solution of Hamilton-Jacobi-Bellman equation [cf. 10, 11, 12]. The purpose of this paper is to extend partially his results to stochastic differential games, where two players conflict each other. If the value function of stochatic differential game is smooth enough, then it satisfies a second order partial differential equation with max-min or min-max type nonlinearity, called Isaacs equation [cf. 5]. Since we can write a nonlinear function as min-max of appropriate affine functions, under some mild conditions, the stochastic differential game theory provides some convenient representation formulas for solutions of nonlinear partial differential equations [cf. 1, 2, 3].

New results on the relationship between dynamic programming and the maximum principle

Abstract: The dynamic programming approach to optimal control theory attempts to characterize the value functionV as a solution to the Hamilton-Jacobian-Bellman equation. Heuristic arguments have long been advanced relating the Pontryagin maximum principle and dynamic programming according to the equation (H(t, x*(t), u*(t), p(t)),−p(t))=√V(t,x*(t)), where (x*, u*) is the optimal control process under consideration,p(t), is the coextremal, andH is the Hamiltonian. The relationship has previously been verified under only very restrictive hypotheses. We prove new results, establishing the relationship, now expressed in terms of the generalized gradient ofV, for a large class of nonsmooth problems.

On the Hamilton-Jacobi-Bellman equations in Banach spaces

Abstract: This paper is concerned with a certain class of distributed parameter control problems. The value function of these problems is shown to be the unique viscosity solution of the corresponding Hamiltonian-Jacobi-Bellman equation. The main assumption is the existence of an increasing sequence of compact invariant subsets of the state space. In particular, this assumption is satisfied by a class of controlled delay equations.

Bellman inequalities in markov decision deterministic drift processes

Abstract: In the case of both continuously and impulsively controlled Markov-type stochastic processes the usual Bellman optimality equation for the value function vturns into a system of the form T1 v≤0, T2 v≤0, T1 v. T2 v= 0, where T1and T2are, respectively, a non-differential and a differential non-linear operators ‘in the controlled diffusion this system became known as "quasi variational inequalities"». We examine to what extent analogous inequalities are valid in Markov decision deterministic drift processes, in which randomness affects only jumps. As auxiliary results we prove that v is upper semianalytic, that almost everywhere E-optimal policies do exist, and also prove a defect formula for policies which is closely related to the inequalities in consideration. Our investigation is based on the concept of stochastic processes with a randomly split time parameter

Characterizing optimality in mathematical programming models

Abstract: This paper is a survey of basic results that characterize optimality in single- and multi-objective mathematical programming models. Many people believe, or want to believe, that the underlying behavioural structure of management, economic, and many other systems, generates basically ‘continuous’ processes. This belief motivates our definition and study of optimality, termed ‘structural’ optimality. Roughly speaking, we say that a feasible point of a mathematical programming model is structurally optimal if every improvement of the optimal value function, with respect to parameters, results in discontinuity of the corresponding feasible set of decision variables. This definition appears to be more suitable for many applications and it is also more general than the usual one: every optimum is a structural optimum but not necessarily vice versa. By characterizing structural optima, we obtain some new, and recover the familiar, optimality conditions in nonlinear programming. The paper is self-contained. Our approach is geometric and inductive: we develop intiution by studying finite-dimensional models before moving on to abstract situations.

Theory and applications of generalized dynamic programming: An overview

Abstract: Extensions of dynamic programming (DP) into generalized preference structures, such as exist in multicriteria optimization, have invariably assumed sufficient structure to ensure the validity of Bellman's “principle of optimality” or the presence of monotonicity, hence ensuring the validity of the functional equations of DP. Often, however, the only reasonable formulation of problem (in terms of the size of the resulting state space) is one for which neither monotonicity nor the (strong) principle of optimality is satisfied. Generalized DP, based on a weaker principle of optimality, is a generalization of DP that produces optimal solutions, even in the absence of monotonicity. It is shown both to subsume conventional DP and to provide an alternative means of implementing Mitten's preference order DP. This paper provides an overview of generalized DP and its applications to date, summarizing our computational experiences in the process.

Controlled diffusion processes on infinite horizon with the overtaking criterion

Abstract: The optimal control of diffusion processes on the infinite time interval are studied. All the costs diverge to infinity and we employ the overtaking criterion, as well as considering minimal growth rate controls. The Bellman equation for the problem is considered and its solution is given an interpretation connected with the overtaking optimality notion. Control problems with a cost including a generalized discount factor (which is not integrable on [(, ∞)) are also studied. Both cases, where the diffusion is inR n or where it is reflected from the boundary of a bounded set, are considered.

Switching control piecewise-deterministic processes

Abstract: A finite collection of piecewise-deterministic processes are controlled in order to minimize the expected value of a performance functional with continuous operating cost and discrete switching control costs. The solution of the associated dynamic programming equation is obtained by an iterative approximation using optimal stopping time problems.

Controlled Diffusions in a Random Medium.

Abstract: Using homogenization theory we treat the problem of controlled diffusions in a random medium with rapidly varying composition. This involves homogenization of a nonlinear Bellman dynamic programmin...

On the numerical approximation of an optimal correction problem

Abstract: The numerical solution of an optimal correction problem for a damped random linear oscillator is studied. A numerical algorithm for the discretized system of the associated dynamic programming equation is given. To initiate the computation, a numerical scheme derived from the deterministic version of the problem is adopted. A correction-type algorithm based on a discrete maximum principle is introduced to ensure the convergence of the iteration procedure. >

On the first boundary value problem for nonlinear degenerate elliptic equations

Abstract: This article is devoted to a proof of a general theorem on the existence of a solution of the first boundary value problem for a degenerate Bellman equation. In contrast to other papers the nonlinearity of the equation is used here and leads, for example, to a proof of solvability of the simplest Monge-Ampere equation for , in a strictly convex region of class with zero data on the boundary. Bibliography: 18 titles.

On the numerical approximation of an optimal correction problem

Abstract: The numerical solution of an optimal correction problem for a damped random linear oscillator is studied. A numerical algorithm for the discretized system of the associated dynamic programming equation is given. To initiate the computation, we adopt a numerical scheme derived from the deterministic version of the problem. Next, a correction-type algorithm based on a discrete maximum principle is introduced to ensure the convergence of the iteration procedure.

Dynamic programming, reduction of dimensionality and eigenvalue problems☆

Abstract: This paper develops an iterative algorithm, based on a priori deduction from Bellman's principle of optimality, for solving eigenvalue problems. During each iteration, the set of admissible states is restricted only to those stages that are “near” to the nominal trajectory. The algorithm is shown to use only minimal storage requirements. The significance of the method is that it provides a means of reducing Bellman's “curse of dimensionality” and broadens the scope of problems that can effectively be solved with the dynamic programming approach. The technique is then applied to evaluate the smallest eigenvalue for the differential equation arising in the mathematical modelling of the desorption from a liquid film.

Acyclicity and Stability for Intertemporal Optimization Models

Abstract: This paper provides a turnpike-like theorem for multidimensional, optimal-growth models, which holds for evey level of the discount factor. It is shown that when the short-run return fu nction of the reduced-form model satisfies a certain sufficient condi tion, then the resulting dynamics is of a simple type, i.e., it must converge to some steady state. The result is obtained in two steps: f irst it is shown that dynamical systems satisfying an acyclic binary relation must be simple; second, the value function for the optimal p roblem is used to define a binary relation on the space of feasible s tates. The necessary and sufficient conditions under which the latter binary relation is acyclic are provided, and their relation to the t echnology and preferences is outlined. Copyright 1988 by Economics Department of the University of Pennsylvania and the Osaka University Institute of Social and Economic Research Association.

Policy Improvement and the Newton-Raphson Algorithm

Abstract: We show that the calculation of the infinite-horizon value function for a linear/quadratic Markov decision process by policy improvement is exactly equivalent to solution of the equilibrium Riccati equation by the Newton-Raphson method. The assertion extends to risk-sensitive and non-Markov forinulations and thus shows, for example, that the Newton-Raphson method provides an iterative algorithm for the canonical factorization of operators which shows second-order convergence and has a variational basis.

Logarithmic transformations for discrete-time, finite-horizon stochastic control problems

Abstract: Using logarithmic transformations, we construct discrete-time stochastic control problems where the optimal value function (cost-to-go) belongs to a same parametrized class of functions that remains invariant under the dynamic programming operator. This extends a well-known property of the classical LQG problems, where the optimal value function is a quadratic. Some related questions are also discussed.

Entropy-based selection with multiple objectives

Abstract: In this article we present an approach to determine the initially unspecified weights in an additive measurable multiattribute value function We formulate and solve a series of nonlinear programming problems which (1) incorporate whatever partial information concerning the attribute weights or overall relative value of alternatives the decision maker chooses to provide, yet (2) yield a specific set of weights as a result Although each formulation is rather easily solved using the nonlinear programming software GINO (general interactive optimizer), solutions in closed form dependent on a single parameter are also provided for a number of these problems

Dynamic Duality in Resource Economics: Theory and Empirical Usefulness

Abstract: It Is Possible to Think of Numerous Economic Problems Involving Dynamic Constrained Optimization Where Dual Representation Exist and Possibly Can Be Characterized. One Example Arises in the Theory of Dynamic Factor Demand And, More Generally, Investment Decisions. Its Duality Structure Has Been Studied by Epstein in Several Papers. the Extraction of Non-Renewable Resources Is an Investment Problem And, Furthermore Specific Boundary Conditions Must Be Satisfied At the Date of Exhaustion If It Occurs in a Finite Time. This Paper Shows How the Theory of Dynamic Duality Can Be Worked Out in Such Problems. a Family of Functions Which Satisfy the Properties of the Corresponding Value Function Is Also Provided. While Natural Resource Extraction Is a Prime Candidate for Empirical Applications, Other Applications Could Be Carried Out in Such Areas As R&D, the Management of Informaiton, and Advertising.

Linear programming as a technique for optimization of queueing systems

Abstract: For a significant number of queueing models that appear in diverse, seemingly unrelated application areas, such as routing, resource allocation and flow control, the optimal policy exhibits a certain switching-curve structure. The authors formulate the optimal control problem for such models in a unified way, using abstract linear programming. Using well-known facts from sensitivity analysis of linear programs, they show how certain properties of the optimal policy can be easily derived, even in cases where dynamic programming (DP) and stochastic dominance (SD) arguments fail. A structural property of the optimal value function of the linear program, namely piecewise linearity, is exploited to derive properties of the optimal cost function. The authors also consider additional problems in the realm of queueing system control in which DP or SD approaches are not applicable but linear programming can provide useful results. >