The theory of Markov Decision Processes is the theory of controlled Markov chains. Its origins can be traced back to R. Bellman and L. Shapley in the 1950’s. During the decades of the last century this theory has grown dramatically. It has found applications in various areas like e.g. computer science, engineering, operations research, biology and economics. In this article we give a short introduction to parts of this theory. We treat Markov Decision Processes with finite and infinite time horizon where we will restrict the presentation to the so-called (generalized) negative case. Solution algorithms like Howard’s policy improvement and linear programming are also explained. Various examples show the application of the theory. We treat stochastic linear-quadratic control problems, bandit problems and dividend pay-out problems.

/pdf/markov-decision-processes-3cwgsogfja.pdf

Markov Decision Processes

This paper considers constrained Markov control processes in Borel spaces, with unbounded costs. The criterion to be minimized is a long-run expected average cost, and the constraints can be imposed on similar average costs, or on average rewards, or discounted costs or rewards. We give conditions under which the constrained problem (CP) is solvable and equivalent to an equality constrained (EC) linear program. Furthermore, we show that there is no duality gap between EC and the dual program EC* and that in fact the strong duality condition holds. Finally, we introduce an explicit procedure to solve CP in some cases which is illustrated with a detailed example.

Constrained Average Cost Markov Control Processes in Borel Spaces

We consider utility-constrained Markov decision processes The expected utility of the total discounted reward is maximized subject to multiple expected utility constraints By introducing a corresponding Lagrange function, a saddle-point theorem of the utility constrained optimization is derived The existence of a constrained optimal policy is characterized by optimal action sets specified with a parametric utility

Discounted Markov decision processes with utility constraints

This paper is devoted to studying constrained continuous-time Markov decision processes (MDPs) in the class of randomized policies depending on state histories. The transition rates may be unbounded, the reward and costs are admitted to be unbounded from above and from below, and the state and action spaces are Polish spaces. The optimality criterion to be maximized is the expected discounted rewards, and the constraints can be imposed on the expected discounted costs. First, we give conditions for the nonexplosion of underlying processes and the finiteness of the expected discounted rewards/costs. Second, using a technique of occupation measures, we prove that the constrained optimality of continuous-time MDPs can be transformed to an equivalent (optimality) problem over a class of probability measures. Based on the equivalent problem and a so-called w-weak convergence of probability measures developed in this paper, we show the existence of a constrained optimal policy. Third, by providing a linear programming formulation of the equivalent problem, we show the solvability of constrained optimal policies. Finally, we use two computable examples to illustrate our main results.

/pdf/discounted-continuous-time-constrained-markov-decision-2n8qvb6vys.pdf

Discounted continuous-time constrained Markov decision processes in Polish spaces

This chapter is concerned with the Linear Programming (LP) approach to MDPs in general Borel spaces, valid for several criteria, including the finite horizon and long run expected average cost, as well as the infinite horizon expected discounted cost.

The Linear Programming Approach

In the framework of discounted Markov decision processes, we consider the case that the transition probability varies in some given domain at each time and its variation is unknown or unobservable. To this end we introduce a new model, named controlled Markov set-chains, based on Markov set-chains, and discuss its optimization under some partial order. Also, a numerical example is given to explain the theoretical results and the computation.

Controlled markov set-chains with discounting

Constrained Markov decision processes with compact state and action spaces are studied under long-run average reward or cost criteria. By introducing a corresponding Lagrange function, a saddle-point theorem is given, by which the existence of a constrained optimal pair of initial state distribution and policy is shown. Also, under the hypothesis of Doeblin, the functional characterization of a constrained optimal policy is obtained

/pdf/constrained-markov-decision-processes-with-compact-state-and-46ualncasd.pdf

Constrained markov decision processes with compact state and action spaces: the average case

Event-triggered cooperative path following control of multiple underactuated unmanned surface vehicles with complex unknowns and actuator saturation

/pdf/design-and-test-of-an-improved-active-disturbance-rejection-1e6rzwg5.pdf

Design and test of an improved active disturbance rejection control system for water sampling unmanned surface vehicle

The objective of this paper is to study the linear programming for countable state average reward Markov decision processes with multiple cost constraints A corresponding linear programming and its dual are formulated and their solvability and absence of a duality gap are proved under some reasonable conditions Also, the ergodic assumption derives the existence of a constrained optimal stationary policy

Youqiang Huang

Papers

Controlled markov set-chains with discounting

Constrained markov decision processes with compact state and action spaces: the average case

Event-triggered cooperative path following control of multiple underactuated unmanned surface vehicles with complex unknowns and actuator saturation

Design and test of an improved active disturbance rejection control system for water sampling unmanned surface vehicle

The LP approach in average reward MDPs with multiple cost constraints: The countable state case