Bellman equation

About: Bellman equation is a research topic. Over the lifetime, 5884 publications have been published within this topic receiving 135589 citations.

Scheduling of project networks

Abstract: The paper deals with the network optimization problem of minimizing regular project cost subject to an arbitrary precedence relation on the sets of activities and to arbitrarily many resource constraints. The treatment is done via a purely structural approach that considerably extends the disjunctive graph concept. It is based on so-called feasible posets and includes a quite deep and useful representation theorem. This theorem permits many insights concerning the analytical behaviour of the optimal value function, the description and counting of all essentially different optimization problems, the nature of Graham anomalies, connections with the on-line stochastic generalizations, and several others. In addition, it also allows the design of a quite powerful class of branch-and-bound algorithms for such problems, which is based on an iterative construction of feasible posets. Using so-called distance matrices, this approach permits the restriction of the exponential part of the algorithm to the often comparatively small set of ‘resource and cost essential’ jobs. The paper reports on computational experience with this algorithm for examples from the building industry and includes a rough comparison with the integer programming approach by Talbot and Patterson.

A stochastic control problem with delay arising in a pension fund model

Abstract: This paper deals with the optimal control of a stochastic delay differential equation arising in the management of a pension fund with surplus. The problem is approached by the tool of a representation in infinite dimension. We show the equivalence between the one-dimensional delay problem and the associated infinite-dimensional problem without delay. Then we prove that the value function is continuous in this infinite-dimensional setting. These results represent a starting point for the investigation of the associated infinite-dimensional Hamilton–Jacobi–Bellman equation in the viscosity sense and for approaching the problem by numerical algorithms. Also an example with complete solution of a simpler but similar problem is provided.

Scheduling control for queueing systems with many servers: Asymptotic optimality in heavy traffic

Abstract: A multiclass queueing system is considered, with heterogeneous service stations, each consisting of many servers with identical capabilities. An optimal control problem is formulated, where the control corresponds to scheduling and routing, and the cost is a cumulative discounted functional of the system’s state. We examine two versions of the problem: “nonpreemptive,” where service is uninterruptible, and “preemptive,” where service to a customer can be interrupted and then resumed, possibly at a different station. We study the problem in the asymptotic heavy traffic regime proposed by Halfin and Whitt, in which the arrival rates and the number of servers at each station grow without bound. The two versions of the problem are not, in general, asymptotically equivalent in this regime, with the preemptive version showing an asymptotic behavior that is, in a sense, much simpler. Under appropriate assumptions on the structure of the system we show: (i) The value function for the preemptive problem converges to V , the value of a related diffusion control problem. (ii) The two versions of the problem are asymptotically equivalent, and in particular nonpreemptive policies can be constructed that asymptotically achieve the value V . The construction of these policies is based on a Hamilton–Jacobi–Bellman equation associated with V .

Reward functionals, salvage values, and optimal stopping

Abstract: We consider the optimal stopping of a linear diffusion in a problem subject to both a cumulative term measuring the expected cumulative present value of a continuous and potentially state-dependent profit flow and an instantaneous payoff measuring the salvage or terminal value received at the optimally chosen stopping date. We derive an explicit representation of the value function in terms of the minimal r-excessive mappings for the considered diffusion, and state a set of necessary conditions for optimal stopping by applying the classical theory of linear diffusions and ordinary non-linear programming techniques. We also state a set of conditions under which our necessary conditions are also sufficient and prove that the smooth pasting principle follows directly from our approach, while the contrary is not necessarily true.

Optimal control of piecewise deterministic Markov processes.

Abstract: This thesis describes a complete theory of optimal control of piecewise deterministic Markov processes under weak assumptions. The theory consists of a description of the processes, a nonsmooth stochastic maximum principle as a necessary optimality condition, a generalized Bellman-Hamilton-Jacobi necessary and sufficient optimality condition involving the Clarke generalized gradient, existence results and regularity properties of the value function. The impulse control problem is transformed to an equivalent optimal dynamic control problem. Cost functions are subject only to growth conditions.