From the Publisher:
The past decade has seen considerable theoretical and applied research on Markov decision processes, as well as the growing use of these models in ecology, economics, communications engineering, and other fields where outcomes are uncertain and sequential decision-making processes are needed. A timely response to this increased activity, Martin L. Puterman's new work provides a uniquely up-to-date, unified, and rigorous treatment of the theoretical, computational, and applied research on Markov decision process models. It discusses all major research directions in the field, highlights many significant applications of Markov decision processes models, and explores numerous important topics that have previously been neglected or given cursory coverage in the literature. Markov Decision Processes focuses primarily on infinite horizon discrete time models and models with discrete time spaces while also examining models with arbitrary state spaces, finite horizon models, and continuous-time discrete state models. The book is organized around optimality criteria, using a common framework centered on the optimality (Bellman) equation for presenting results. The results are presented in a "theorem-proof" format and elaborated on through both discussion and examples, including results that are not available in any other book. A two-state Markov decision process model, presented in Chapter 3, is analyzed repeatedly throughout the book and demonstrates many results and algorithms. Markov Decision Processes covers recent research advances in such areas as countable state space models with average reward criterion, constrained models, and models with risk sensitive optimality criteria. It also explores several topics that have received little or no attention in other books, including modified policy iteration, multichain models with average reward criterion, and sensitive optimality. In addition, a Bibliographic Remarks section in each chapter comments on relevant historic

Markov Decision Processes: Discrete Stochastic Dynamic Programming

/pdf/convex-analysisnoer-san-nojin-zhan-nituite-5dl2rbmoyr.pdf

Convex Analysisの二,三の進展について

Publisher Summary This chapter reviews the supply chain coordination with contracts. Numerous supply chain models are discussed. In each model, the supply chain optimal actions are identified. The chapter extends the newsvendor model by allowing the retailer to choose the retail price in addition to the stocking quantity. Coordination is more complex in this setting because the incentives provided to align one action might cause distortions with the other action. The newsvendor model is also extended by allowing the retailer to exert costly effort to increase demand. Coordination is challenging because the retailer's effort is noncontractible—that is, the firms cannot write contracts based on the effort chosen. The chapter also discusses an infinite horizon stochastic demand model in which the retailer receives replenishments from a supplier after a constant lead time. Coordination requires that the retailer chooses a large basestock level.

Supply Chain Coordination with Contracts

A comparison is undertaken between scale development and index construction procedures to trace the implications of adopting a reflective versus formative perspective when creating multi-item measures for organizational research. Focusing on export coordination as an illustrative construct of interest, the results show that the choice of measurement perspective impacts on the content, parsimony and criterion validity of the derived coordination measures. Implications for practising researchers seeking to develop multi-item measures of organizational constructs are considered.

https://papers.ssrn.com/sol3/Delivery.cfm?abstractid=950400

Formative Versus Reflective Indicators in Organizational Measure Development: A Comparison and Empirical Illustration

In traditional supply chain inventory management, orders are the only information firms exchange, but information technology now allows firms to share demand and inventory data quickly and inexpensively. We study the value of sharing these data in a model with one supplier, N identical retailers, and stationary stochastic consumer demand. There are inventory holding costs and back-order penalty costs. We compare a traditional information policy that does not use shared information with a full information policy that does exploit shared information. In a numerical study we find that supply chain costs are 2.2% lower on average with the full information policy than with the traditional information policy, and the maximum difference is 12.1%. We also develop a simulation-based lower bound over all feasible policies. The cost difference between the traditional information policy and the lower bound is an upper bound on the value of information sharing: In the same study, that difference is 3.4% on average, and no more than 13.8%. We contrast the value of information sharing with two other benefits of information technology, faster and cheaper order processing, which lead to shorter lead times and smaller batch sizes, respectively. In our sample, cutting lead times nearly in half reduces costs by 21% on average, and cutting batches in half reduces costs by 22% on average. For the settings we study, we conclude that implementing information technology to accelerate and smooth the physical flow of goods through a supply chain is significantly more valuable than using information technology to expand the flow of information.

Supply Chain Inventory Management and the Value of Shared Information

We address a fundamental two-echelon distribution system in which the sales volumes of the retailers are endogenously determined on the basis of known demand functions. Specifically, this paper studies a distribution channel where a supplier distributes a single product to retailers, who in turn sell the product to consumers. The demand in each retail market arrives continuously at a constant rate that is a general decreasing function of the retail price in the market. We have characterized an optimal strategy, maximizing total systemwide profits in a centralized system. We have also shown that the same optimum level of channelwide profits can be achieved in a decentralized system, but only if coordination is achieved via periodically charged, fixed fees, and a nontraditional discount pricing scheme under which the discount given to a retailer is the sum of three discount components based on the retailer's i annual sales volume, ii order quantity, and iii order frequency, respectively. Moreover, we show that no traditional discount scheme, based on order quantities only, suffices to optimize channelwide profits when there are multiple nonidentical retailers. The paper also considers a scenario where the channel members fail to coordinate their decisions and provides numerical examples that illustrate the value of coordination. We extend our results to settings in which the retailers' holding cost rates depend on the wholesale price.

/pdf/coordination-mechanisms-for-a-distribution-system-with-one-x87hs49j35.pdf

Coordination Mechanisms for a Distribution System with One Supplier and Multiple Retailers

In this paper, a new algorithm for computing optimal (s, S) policies is derived based upon a number of new properties of the infinite horizon cost function c(s, S) as well as a new upper bound for optimal order-up-to levels S* and a new lower bound for optimal reorder levels s*. The algorithm is simple and easy to understand. Its computational complexity is only 2.4 times that required to evaluate a (specific) single (s, S) policy. The algorithm applies to both periodic review and continuous review inventory systems.

/pdf/finding-optimal-s-s-policies-is-about-as-simple-as-ttd5m5o3mr.pdf

Finding Optimal (s, S) Policies Is About As Simple As Evaluating a Single Policy

For most order quantity/reorder point inventory systems, the stochastic model, which specifies the demands as stochastic processes, is often more accurate than its deterministic counterpart—the EOQ model. However, the application of the stochastic model has been limited because of the absence of insightful analytical results on the model. This paper analyzes the stochastic order quantity/reorder point model in comparison with a corresponding deterministic EOQ model. Based on simple optimality conditions for the control variables derived in the paper, a sensitivity analysis is carried out, and a number of basic qualitative properties are established for the optimal control parameters. Our main results include the following: (1) in contrast to the deterministic EOQ model, the controllable costs of the stochastic model due to selection of the order quantity (assuming the reorder point is chosen optimally for every order quantity) are actually smaller, while the total costs are clearly larger; the optimal ord...

https://www.jstor.org/stable/info/2632586

On properties of stochastic inventory systems

We establish lower bounds on the minimum cost of managing certain production-distribution networks with setup costs at all stages and stochastic demands. These networks include serial, assembly, and one-warehouse multi-retailer systems. We obtain the bounds through novel cost-allocation schemes. We evaluate the bounds' performance for one-warehouse multi-retailer systems by comparing them with simple, heuristic policies. The bounds are quite tight for systems with a small number of retailers. We also present simplified proof of known optimality results for serial and assembly systems.

/pdf/lower-bounds-for-multi-echelon-stochastic-inventory-systems-4d5kvgmp8f.pdf

Lower Bounds for Multi-Echelon Stochastic Inventory Systems

The reorder point/reorder quantity policies, also referred to as r, Q policies, are widely used in industry and extensively studied in the literature. However, for a period of almost 30 years there has been no efficient algorithm for computing optimal control parameters for such policies. In this paper, we present a surprisingly simple and efficient algorithm for the determination of an optimal r*, Q* policy. The computational complexity of the algorithm is linear in Q*. For the most prevalent case of linear holding, backlogging and stockout penalty costs in addition to fixed order costs, the algorithm requires at most 6r* + 13Q* elementary operations additions, comparisons and multiplications, and hence, no more than 13 times the amount of work required to do a single evaluation of the long-run average cost function in the point r*, Q*.

Yu-Sheng Zheng

Papers

Coordination Mechanisms for a Distribution System with One Supplier and Multiple Retailers

Finding Optimal (s, S) Policies Is About As Simple As Evaluating a Single Policy

On properties of stochastic inventory systems

Lower Bounds for Multi-Echelon Stochastic Inventory Systems

An efficient algorithm for computing an optimal ( r,Q ) policy in continuous review stochastic inventory systems