A first course in stochastic processes

Contents: Transition Functions and Resolvents.- Existence and Uniqueness of Q-Functions.- Examples of Continuous Time Markov Chains.- More on the Uniqueness Problem.- Classification of States and Invariant Measures.- Strong and Exponential Ergodicity.- Reversibility, Monotonictity, and Other Properties.- Birth and Death Processes.- Population Processes.- Bibliography.- Symbol Index.- Author Index.- Subject Index.

Book Reviews - Continuous Time Markov Chains: An Applications-Oriented Approach

In this paper, we present a literature review, classification schemes and a simple meta-analysis for scheduling of batch processors (SBP) research in semiconductor manufacturing (SM). This review is based on a study of journals and web-based documents/articles, which include conference materials, lecture notes in computer science, working papers, etc. There are 98 articles published in various publication outlets between 1986 and October 2004. Based on the literature review carried out and the nature of SBP research observed in SM, we have introduced two classification schemes to systematically organize the published articles. The first classification scheme is based on the problem configurations of SBP research in SM and the second one is based on the solution methodology followed by the researchers. These classification schemes indicate that there is much research scope on SBP research in SM. Furthermore, a simple meta-analysis is carried out to enhance understanding on the development and evolution of SBP research in SM and to identify potential research areas for further research and for improvement. The results show that there is an increasing trend in SBP research in SM. A comprehensive list of references is presented. This study is expected to provide a source of reference for other researchers (or readers), who are interested in SBP research particularly in SM and help stimulate further interest.

A literature review, classification and simple meta-analysis on scheduling of batch processors in semiconductor

The deviation matrix of an ergodic, continuous-time Markov chain with transition probability matrix P(·) and ergodic matrix P is the matrix D ≡ ∫0∞(P(t) − P) dt. We give conditions for D to exist and discuss properties and a representation of D. The deviation matrix of a birth–death process is investigated in detail. We also describe a new application of deviation matrices by showing that a measure for the convergence to stationarity of a stochastically increasing Markov chain can be expressed in terms of the elements of the deviation matrix of the chain.

/pdf/the-deviation-matrix-of-a-continuous-time-markov-chain-3ygaaxjms8.pdf

The deviation matrix of a continuous-time markov chain

Markov Chains with Stationary Transition Probabilities. By Kai Lai Chung. Pp. x + 278. 120 S. 1960. (Springer, Berlin)

We study deviation matrices of birth–death processes. This is relevant to the control of multidimensional queueing systems. We give an algorithm for computing deviation matrices for birth–death processes. As an application, we compute them explicitly for the M/M/s/N and M/M/s/∞ queues.

/pdf/on-deviation-matrices-for-birth-death-processes-2ibctgq0p6.pdf

On deviation matrices for birth–death processes

In countable state non-explosive minimal Markov processes the Kolmogorov forward equations hold under sufficiently weak conditions. However, a precise description of the functions that one may integrate with respect to these equations seems to be absent in the literature. This problem arises for instance when studying the Poisson equation, as well as the average cost optimality equation in a Markov decision process.

Kolmogorov forward equation and explosiveness in countable state Markov processes

This paper provides a comparison of the successive lumping (SL) methodology developed in Katehakis et al. (2015) with the popular lattice path counting (Mohanty (1979)) in obtaining rate matrices for queueing models, satisfying the specific quasi birth and death structure as in Van Leeuwaarden et al. (2009) and Van Leeuwaarden and Winands (2006). The two methodologies are compared both in terms of applicability requirements and numerical complexity by analyzing their performance for the same classical queueing models considered in Van Leeuwaarden et al. (2009). The main findings are threefold. First, when both methods are applicable, the SL-based algorithms outperform the lattice path counting algorithm (LPCA). Second, there are important classes of problems (for example, models with (level) nonhomogenous rates or with finite state spaces) for which the SL methodology is applicable and for which the LPCA cannot be used. Third, another main advantage of SL algorithms over lattice path counting is that the former includes a method to compute the steady state distribution using this rate matrix.

https://www.cambridge.org/core/services/aop-cambridge-core/content/view/S0021900215000121

A comparative analysis of the successive lumping and the lattice path counting algorithms

The existence of a moment function satisfying a drift function condition is well known to guarantee non-explosiveness of the associated minimal Markov process (cf. [1,9]), under standard technical conditions. Surprisingly, the reverse is true as well for a countable space Markov process. We prove this result by showing that recurrence of an associated jump process, that we call the α-jump process, is equivalent to non-explosiveness. Non-explosiveness corresponds in a natural way to the validity of the Kolmogorov integral relation for the function identically equal to 1. In particular, we show that positive recurrence of the α-jump chain implies that all bounded functions satisfy the Kolmogorov integral relation. We present a drift function criterion characterizing positive recurrence of this α-jump chain. Suppose that to a drift function V there corresponds another drift function W, which is a moment with respect to V. Via a transformation argument, the above relations hold for the transformed process with respect to V. Transferring the results back to the original process, allows to characterize the V-bounded functions that satisfy the Kolmogorov forward equation.

Countable state markov processes: non-explosiveness and moment function

This paper defines and studies the down entrance state (DES) and the restart entrance state (RES) classes of quasi-skip free (QSF) processes specified in terms of the nonzero structure of the elements of their transition rate matrix Q. A QSF process is a Markov chain with states that can be specified by tuples of the form (m, i), where the current phase of the state, and its transition probability matrix Q does not permit one-step transitions to states that are two or more levels away from the current state in one direction of the level variable m. A QSF process is a DES process if and only if one step “down” transitions from a level m can only reach a single state in level m − 1, for all m. A QSF process is a RES process if and only if one step “up” transitions from a level m can only reach a single set of states in the highest level M2, largest of all m.We derive explicit solutions and simple truncation bounds for the steady-state probabilities of both DES and RES processes, when in addition Q insures ergodicity. DES and RES processes have applications in many areas of applied probability comprising computer science, queueing theory, inventory theory, reliability, and the theory of branching processes. To motivate their applicability we present explicit solutions for the well-known open problem of the M/Er/n queue with batch arrivals, an inventory model, and a reliability model.

/pdf/des-and-res-processes-and-their-explicit-solutions-3ju4b248hb.pdf

Floske M. Spieksma

Papers

On deviation matrices for birth–death processes

Kolmogorov forward equation and explosiveness in countable state Markov processes

A comparative analysis of the successive lumping and the lattice path counting algorithms

Countable state markov processes: non-explosiveness and moment function

Des and res processes and their explicit solutions