Showing papers on "Markov decision process published in 1983"

Long-term average cost control problems for continuous time Markov processes: A survey

Abstract: This paper addresses the long-term average cost control of continuous time Markov processes. A survey of problems and methods contained in various works is given for continuous control, optimal stopping, and impulse control.

Denumerable Undiscounted Semi-Markov Decision Processes with Unbounded Rewards

Abstract: This paper establishes the existence of a solution to the optimality equations in undis-counted semi-Markov decision models with countable state space, under conditions generalizing the hitherto obtained results. In particular, we merely require the existence of a finite set of states in which every pair of states can reach each other via some stationary policy, instead of the traditional and restrictive assumption that every stationary policy has a single irreducible set of states. A replacement model and an inventory model illustrate why this extension is essential. Our approach differs fundamentally from classical approaches; we convert the optimality equations into a form suitable for the application of a fixed point theorem.

Vector-Valued Dynamic Programming

Abstract: Dynamic programming models with vector-valued returns are investigated. The sets of (Pareto) maximal returns and (Pareto) maximal policies are defined. Monotonicity conditions are shown to be sufficient for the set of maximal policies to include a stationary policy, and for the set of maximal returns to be in the convex hull of returns of stationary policies. In particular, it is shown that these results hold for Markov decision processes.

Average optimal policies in Markov decision drift processes with applications to a queueing and a replacement model

Abstract: Recently the authors introduced the concept of Markov decision drift processes. A Markov decision drift process can be seen as a straightforward generalization of a Markov decision process with continuous time parameter. In this paper we investigate the existence of stationary average optimal policies for Markov decision drift processes. Using a well-known Abelian theorem we derive sufficient conditions, which guarantee that a 'limit point' of a sequence of discounted optimal policies with the discounting factor approaching 1 is an average optimal policy. An alternative set of sufficient conditions is obtained for the case in which the discounted optimal policies generate regenerative stochastic processes. The latter set of conditions is easier to verify in several applications. The results of this paper are also applicable to Markov decision processes with discrete or continuous time parameter and to semi-Markov decision processes. In this sense they generalize some wellknown results for Markov decision processes with finite or compact action space. Applications to an M/M/1 queueing model and a maintenance replacement model are given. It is shown that under certain conditions on the model parameters the average optimal policy for the M/M/1 queueing model is monotone non-decreasing (as a function of the number of waiting customers) with respect to the service intensity and monotone non-increasing with respect to the arrival intensity. For the maintenance replacement model we prove the average optimality of a bang-bang type policy. Special attention is paid to the computation of the optimal control parameters.

Controlled Markov Chains and Stochastic Networks

Abstract: Controlled Markov chains with average cost criterion and with special cost and transition structures are studied. Existence of optimal stationary strategies is established for the average cost criterion. Corresponding dynamic programming equations are derived. A stochastic network problem that includes interconnected queues as a special case is described and studied within this framework.

Continuous time markov decision processes with interventions

Abstract: We study Markov jump decision processes with both continuously and instantaneouslyacting decisions and with deterministic drift between jumps. Such decision processes were recentlyintroduced and studied from discrete time approximations point of view by Van der Duyn Schouten.Weobtain necessary and sufficient optimality conditions for these decision processes in terms of equations and inequalities of quasi-variational type. By means of the latter we find simple necessaryand sufficient conditions for the existence of stationary optimal policies in such processes with finite state and action spaces, both in the discounted and average per unit time reward cases.

A Markov Decision Modeling Approach to a Multi-Objective Maintenance Problem

Abstract: This article discusses a Markov decision modeling approach to the solution of a large-scale multi-objective problem--the maintenance of a statewide network of roads. This approach integrates management objectives for public safety and comfort, and preservation of the considerable investment in highways with State and Federal budgetary policies and engineering considerations. The Markov decision model captures the dynamic and probabilistic aspects of the maintenance problem and considers the influence of environmental factors, the type of roads, traffic densities and various engineering factors influencing road deterioration. The model recommends the best maintenance action for each mile of the network of highways, and specifies the minimum funds required to carry out the maintenance program.

A Markov Analysis of Fund-Raising Alternatives:

Abstract: The author explores the strategies of fund-raising for a nonprofit organization to maximize net income. The giving pattern of members is modeled as a Markov chain. The alternatives are evaluated us...

Optimal Control of Markov Processes.

Abstract: : The purpose of this article is to give an overview of some recent developments in optimal stochastic control theory. Broadly speaking, stochastic control theory deals with models of systems whose evolution is affected both by certain random influences and also by certain inputs chosen by a controller. The authors are concerned here only with state-space formulations of control problems in continuous time. Moreover, the authors consider only markovian control problems in which the state x sub t of the process being controlled is Markov provided the controller follows a Markov control policy. They mainly discuss the case of continuously acting control, in which at each time t a control u sub t is applied to the system.

On Uniformly Nearly-Optimal Markov Strategies

Abstract: In this paper the following result is proved. In any total reward countable state Markov decision process a Markov strategy IT exists which is uniformly nearly-optimal in the following sense: v(i,π,) ≥ v*(i) − e − eu*(i) for any initial state i. Here v* denotes the value function of the process and u* denotes the value of the process if all negative rewards are neglected.

Adaptive Policies in Markov Decision Processes with Uncertain Transition Matrices

Abstract: This study is concerned with Markov Decision Processes with uncertain transition matrices. In the discounted case, the Bayesian analysis of this model is studied.We define an adaptive policy and a learning policy and show that there exists, for any ???> 0 an ???-optimal and learning policy. In the average case, the non-Bayesian analysis of this model is studied and an optimal adaptive policy is constructed.

Adaptive control of M/M /1 queues—continuous-time Markov decision process approach

Abstract: By considering continuous-time Markov decision processes where decisions can be made at any time, we show in the case of M/M /1 queues with discounted costs that there exists a monotone optimal policy among all the regular policies.

Markov Decision Processes and Ship Handling: An Exercise in Aggregation

Abstract: Operational planning in a general purpose ship terminal is treated. The decisions to be taken concern the weekly manpower capacity and the assignment of manpower and equipment to ships. As a Markov decision problem the model is very big and aggregation is desirable. As a check simulation is used, which leads to an iterative aggregation-disaggregation approach.

Two competing queues with linear costs: The µc-rule is often optimal

Abstract: A discrete-time model is presented for a system of two queues that compete for the service attention of a single server with infinite buffer capacity. The arrivals are modelled by an i.i.d, random sequence of a general type while the service completions are generated by independent Bernou lli streams, and the allocation of service attention is governed by feedback policies which are based on past decisions and buffer content histories. The cost of operation per unit time is a linear function of the queue sizes. Under the model assumptions, a fixed prioritization scheme, known as the µc-rule, is shown, to be optimal when the expected long-run average criterion and the expected discounted criterion, over both finite and infinite horizons, are used. The analysis is based on the Dynamic Programming methodology for Markov decision processes and takes advantage of the sample path properties of the adopted state-space model.

Market Value Maximization and Markov Dynamic Programming

Abstract: This paper shows how an operational method for solving dynamic programs can be used, in some cases, to solve the problem of maximizing a firm's market value. The problem is formulated as a Markov decision problem that can be solved via linear programming. The paper shows how to calculate or estimate the state-contingent prices that are used to value the firm. In addition, the paper points out how states can be aggregated to make the solution technique more practical. The paper's final section contains a specific example.

Characterization of an optimal delayed resolution policy

Abstract: A class of policies called stationary delayed resolution policies have been proposed recently for sharing finite number of buffers at a store-and-forward node in a message switching network [9]. It has been shown that with respect to the total weighted throughput these policies comprise the optimal class of policies. In this paper, we present methods to obtain an optimal policy from the class of stationary delayed resolution policies for given values of the parameters.A method based upon policy iteration technique for Markov decision processes is used to obtain the optimal delayed resolution policy. It is shown that the policy iteration technique while useful in obtaining the exact optimal policy becomes intractable for practical values of buffer sizes and number of message classes. A class of policies called SRS delayed resolution policies is proposed. It is shown that the best SRS delayed resolution policies closely approximate the performance of the optimal delayed resolution policies.

Monotonically improving limit-optimal strategies in finite state decision processes

Abstract: In every finite-state leavable gambling problem and in every finite-state Markov decision process with discounted, negative or positive reward criteria there exists a Markov strategy which is monotonically improving and optimal in the limit along every history. An example is given to show that for the positive and gambling cases such strategies cannot be constructed by simply switching to a "better" action or gamble at each successive return to a state. Key words and phrases: gambling problem, Markov decision process, strategy, stationary strategy, monotonically improving strategy, limit-optimal strategy.

A Simplex-Like Algorithm to Compute a Blackwell-Optimal Policy

Abstract: In a finite Markov decision process a ρ-optimal policy (ρ being the interest rate) can be found for fixed ρ by solving a linear programming problem. We solve such problems simultaneously for all ρ near enough to 0 by considering the problem not in the field of real numbers but in that of rational functions in ρ and applying a simplex-like algorithm in that field. This finite algorithm produces both a Blackwell-optimal policy, its total discounted reward as rational function in ρ and also the interval (0, ρ 0] in which that policy is ρ-optimal.

Application of Markov programming

Abstract: The main topic of the paper is the relation between modelling and numerical analysis for Markov decision processes. It is demonstrated that the relation is very close, since numerical possibilities strongly depend on the structure of the model. This is even true for straightforward numerical techniques, but also for numerical analysis based on aggregation and/or decomposition. Examples amplify the arguments.

Extreme-point solutions in markov decision processes

Abstract: The paper presents sufficient conditions for certain functions to be convex. Functions of this type often appear in Markov decision processes, where their maximum is the solution of the problem. Since a convex function takes its maximum at an extreme point, the conditions may greatly simplify a problem. In some cases a full solution may be obtained after the reduction is made. Some illustrative examples are discussed. OPTIMAL POLICY; CONVEX FUNCTION

Learning Automaton for Finite Semi-Markov Decision Processes

Abstract: A finite semi-Markov decision process is studied to maximize the expected average reward. The semi-Markov kernel of the process depends on an unknown parameter taking values in a subset [a, b] of ℝS. A controller modelled as a learning automaton updates sequentially the probabilities of generating decisions based on the observed decisions, states, and jump times. Convergence results are stated in the form of theorems and some examples are given.