The theory of Markov Decision Processes is the theory of controlled Markov chains. Its origins can be traced back to R. Bellman and L. Shapley in the 1950’s. During the decades of the last century this theory has grown dramatically. It has found applications in various areas like e.g. computer science, engineering, operations research, biology and economics. In this article we give a short introduction to parts of this theory. We treat Markov Decision Processes with finite and infinite time horizon where we will restrict the presentation to the so-called (generalized) negative case. Solution algorithms like Howard’s policy improvement and linear programming are also explained. Various examples show the application of the theory. We treat stochastic linear-quadratic control problems, bandit problems and dividend pay-out problems.

/pdf/markov-decision-processes-3cwgsogfja.pdf

Markov Decision Processes

Active learning reduces the amount of manually annotated sentences necessary when training state-of-the-art statistical parsers. One popular method, uncertainty sampling, selects sentences for which the parser exhibits low certainty. However, this method does not quantify confidence about the current statistical model itself. In particular, we should be less confident about selection decisions based on low frequency events. We present a novel two-stage method which first targets sentences which cannot be reliably selected using uncertainty sampling, and then applies standard uncertainty sampling to the remaining sentences. An evaluation shows that this method performs better than pure uncertainty sampling, and better than an ensemble method based on bagged ensemble members only.

/pdf/a-two-stage-method-for-active-learning-of-statistical-6cjv1ewqx3.pdf

A two-stage method for active learning of statistical grammars

In this paper we consider Markov Decision Processes with discounted cost and a random rate in Borel spaces We establish the dynamic programming algorithm in finite and infinity horizon cases We provide conditions for the existence of measurable selectors And we show an example of consumption-investment problem

Markov control processes with randomized discounted cost

Stochastic Shortest Path problems (SSPs), a subclass of Markov Decision Problems (MDPs), can be efficiently dealt with using Real-Time Dynamic Programming (RTDP). Yet, MDP models are often uncertain (obtained through statistics or guessing). The usual approach is robust planning: searching for the best policy under the worst model. This paper shows how RTDP can be made robust in the common case where transition probabilities are known to lie in a given interval.

/pdf/robust-planning-with-l-rtdp-o3ozu02fhb.pdf

Robust planning with (L)RTDP

We study finite-state, finite-action, discounted infinite-horizon Markov decision processes with uncertain correlated transition matrices in deterministic policy spaces. Existing robust dynamic programming methods cannot be extended to solving this class of general problems. In this paper, based on a robust optimality criterion, an approximate robust policy iteration using a multilayer perceptron neural network is proposed. It is proven that the proposed algorithm converges in finite iterations, and it converges to a stationary optimal or near-optimal policy in a probability sense. In addition, we point out that sometimes even a direct enumeration may not be applicable to addressing this class of problems. However, a direct enumeration based on our proposed maximum value approximation over the parameter space is a feasible approach. We provide further analysis to show that our proposed algorithm is more efficient than such an enumeration method for various scenarios.

Approximate Robust Policy Iteration Using Multilayer Perceptron Neural Networks for Discounted Infinite-Horizon Markov Decision Processes With Uncertain Correlated Transition Matrices

In the framework of discounted Markov decision processes, we consider the case that the transition probability varies in some given domain at each time and its variation is unknown or unobservable. To this end we introduce a new model, named controlled Markov set-chains, based on Markov set-chains, and discuss its optimization under some partial order. Also, a numerical example is given to explain the theoretical results and the computation.

Controlled markov set-chains with discounting

https://www.kurims.kyoto-u.ac.jp/~kyodo/kokyuroku/contents/pdf/1043-22.pdf

Masanori Hosaka

Papers

Controlled markov set-chains with discounting

Controlled Markov set-chains under average criteria

Non-discounted optimal policies in controlled markov set-chains

A Span Seminorm Approach to Controlled Markov Set-Chains(III : Natural Sciences)

A Span Seminorm Approach to Controlled Markov Set-Chains