This report presents a unified approach for the study of constrained Markov decision processes with a countable state space and unbounded costs. We consider a single controller having several objectives; it is desirable to design a controller that minimize one of cost objective, subject to inequality constraints on other cost objectives. The objectives that we study are both the expected average cost, as well as the expected total cost (of which the discounted cost is a special case). We provide two frameworks: the case were costs are bounded below, as well as the contracting framework. We characterize the set of achievable expected occupation measures as well as performance vectors. This allows us to reduce the original control dynamic problem into an infinite Linear Programming. We present a Lagrangian approach that enables us to obtain sensitivity analysis. In particular, we obtain asymptotical results for the constrained control problem: convergence of both the value and the policies in the time horizon and in the discount factor. Finally, we present and several state truncation algorithms that enable to approximate the solution of the original control problem via finite linear programs.

/pdf/constrained-markov-decision-processes-3ndxrfjbtt.pdf

Constrained Markov Decision Processes

The Markov decision problem of locating a policy to maximize the long-run average reward subject to K long-run average cost constraints is considered. It is assumed that the state and action spaces are finite and the law of motion is unichain, that is, every pure policy gives rise to a Markov chain with one recurrent class. It is first proved that there exists an optimal stationary policy with a degree of randomization no greater than K; consequently, it is never necessary to randomize in more than K states. A linear program produces the optimal policy with limited randomization. For the special case of a single constraint, we also address the problem of finding optimal nonrandomized, but nonstationary, policies. We show that a round-robin type policy is optimal, and conjecture the same for a steering policy that depends on the entire past history of the process, but whose implementation requires essentially no more storage than that of a pure policy.

Randomized and Past-Dependent Policies for Markov Decision Processes with Multiple Constraints

The performance of several policies for scheduling real-time and non-real-time messages in a statistical multiplexer is examined. The performance metric for the real-time traffic is the percentage of messages not transmitted within their deadlines; the performance metric for the non-real-time traffic is the average delay. The scheduling policies are: (1) first-come first-served (FCFS); (2) head of the line priority, in which real-time packets are given priority; (3) minimum-laxity threshold (MLT) policy; and (4) queue-length threshold (QLT) policy. Under the MLT policy, priority is given to the real-time traffic when the minimum laxity is below some threshold. The QLT policy gives priority to the non-real-time traffic whenever the number of queued non-real-time packets is above some threshold. Results show that the FCFS policy causes relatively high losses for the real-time traffic while providing relatively low message delays for the non-real-time traffic; the converse holds true for the strict priority discipline. Both the MLT and QLT disciplines allow the designer to explicitly trade off the performance realized by each traffic class by using an appropriately chosen value for the threshold parameter. Little difference is observed in the performance tradeoffs available, so it is concluded that the QLT policy is more practical, as it is simpler to implement. >

Scheduling policies for real-time and non-real-time traffic in a statistical multiplexer

Discrete-time analysis of two schemes for multiplexing voice and data is presented. In each scheme voice and data are multiplexed using the movable boundary frame allocation scheme. In the first scheme, speech activity detectors (SAD's) are not used, and hence, the variations in the voice traffic are only due to the on/off characteristics of voice. In the second scheme, SAD's are employed so that talker silences can he utilized for transmission of additional voice and/or data. In this scheme, the multiplexer performs digital speech interpolation as well as movable boundary frame allocation. The performance measures considered are probability of loss for voice calls, probability of speech clipping, speech packet rejection ratio, and the expected data message delay. In the case of the multiplexer with SAD, a tradeoff exists between data message delay and speech interpolation advantage. Some numerical examples are presented which illustrate the performance of the two multiplexers.

Discrete-Time Analysis of Integrated Voice/Data Multiplexers With and Without Speech Activity Detectors

We present in this chapter a survey on applications of MDPs to communication networks. We survey both the different application areas in communication networks as well as the theoretical tools that have been developed to model and to solve the resulting control problems.

Applications of Markov Decision Processes in Communication Networks

Recently, emphasis has been placed on integrated communication facilities capable of handling both line-switching and packet-switch-ing digital traffic. The problem of dynamically allocating the bandwidth of a trunk to both types of traffic is formulated as a Markovian decision process. Line switching is modeled as a time division multiplexing loss scheme over a varying portion of a fixed time frame. Packet-switching traffic is served through the remaining portion of the frame and requires queueing at the multiplexer-concentrator. Two different cost criteria are examined involving probability of blocking for line switching and average queueing delay for packets. The corresponding optimization problems are presented under reasonable simplifying assumptions. The movable boundary scheme suggested for commercial implementation of integrated multi-plexers is shown to offer optimal or near-optimal performance.

M. Schwartz

Papers

Optimal fixed frame multiplexing in integrated line- and packet-switched communication networks