Planning algorithms are impacting technical disciplines and industries around the world, including robotics, computer-aided design, manufacturing, computer graphics, aerospace applications, drug design, and protein folding. This coherent and comprehensive book unifies material from several sources, including robotics, control theory, artificial intelligence, and algorithms. The treatment is centered on robot motion planning but integrates material on planning in discrete spaces. A major part of the book is devoted to planning under uncertainty, including decision theory, Markov decision processes, and information spaces, which are the “configuration spaces” of all sensor-based planning problems. The last part of the book delves into planning under differential constraints that arise when automating the motions of virtually any mechanical system. Developed from courses taught by the author, the book is intended for students, engineers, and researchers in robotics, artificial intelligence, and control theory as well as computer graphics, algorithms, and computational biology.

Planning Algorithms: Introductory Material

Convergence of Probability Measures. By P. Billingsley. Chichester, Sussex, Wiley, 1968. xii, 253 p. 9 1/4“. 117s.

Convergence of Probability Measures

I. THE RECURSIVE APPROACH 1. Introduction 2. An Overview 2.1 A Deterministic Model of Optimal Growth 2.2 A Stochastic Model of Optimal Growth 2.3 Competitive Equilibrium Growth 2.4 Conclusions and Plans II. DETERMINISTIC MODELS 3. Mathematical Preliminaries 3.1 Metric Spaces and Normed Vector Spaces 3.2 The Contraction Mapping Theorem 3.3 The Theorem of the Maximum 4. Dynamic Programming under Certainty 4.1 The Principle of Optimality 4.2 Bounded Returns 4.3 Constant Returns to Scale 4.4 Unbounded Returns 4.5 Euler Equations 5. Applications of Dynamic Programming under Certainty 5.1 The One-Sector Model of Optimal Growth 5.2 A "Cake-Eating" Problem 5.3 Optimal Growth with Linear Utility 5.4 Growth with Technical Progress 5.5 A Tree-Cutting Problem 5.6 Learning by Doing 5.7 Human Capital Accumulation 5.8 Growth with Human Capital 5.9 Investment with Convex Costs 5.10 Investment with Constant Returns 5.11 Recursive Preferences 5.12 Theory of the Consumer with Recursive Preferences 5.13 A Pareto Problem with Recursive Preferences 5.14 An (s, S) Inventory Problem 5.15 The Inventory Problem in Continuous Time 5.16 A Seller with Unknown Demand 5.17 A Consumption-Savings Problem 6. Deterministic Dynamics 6.1 One-Dimensional Examples 6.2 Global Stability: Liapounov Functions 6.3 Linear Systems and Linear Approximations 6.4 Euler Equations 6.5 Applications III. STOCHASTIC MODELS 7. Measure Theory and Integration 7.1 Measurable Spaces 7.2 Measures 7.3 Measurable Functions 7.4 Integration 7.5 Product Spaces 7.6 The Monotone Class Lemma

Recursive methods in economic dynamics

This report presents a unified approach for the study of constrained Markov decision processes with a countable state space and unbounded costs. We consider a single controller having several objectives; it is desirable to design a controller that minimize one of cost objective, subject to inequality constraints on other cost objectives. The objectives that we study are both the expected average cost, as well as the expected total cost (of which the discounted cost is a special case). We provide two frameworks: the case were costs are bounded below, as well as the contracting framework. We characterize the set of achievable expected occupation measures as well as performance vectors. This allows us to reduce the original control dynamic problem into an infinite Linear Programming. We present a Lagrangian approach that enables us to obtain sensitivity analysis. In particular, we obtain asymptotical results for the constrained control problem: convergence of both the value and the policies in the time horizon and in the discount factor. Finally, we present and several state truncation algorithms that enable to approximate the solution of the original control problem via finite linear programs.

/pdf/constrained-markov-decision-processes-3ndxrfjbtt.pdf

Constrained Markov Decision Processes

Spot and Futures Markets.- An Introduction to Financial Derivatives.- Discrete-time Security Markets.- Benchmark Models in Continuous Time.- Foreign Market Derivatives.- American Options.- Exotic Options.- Volatility Risk.- Continuous-time Security Markets.- Fixed-income Markets.- Interest Rates and Related Contracts.- Short-Term Rate Models.- Models of Instantaneous Forward Rates.- Market LIBOR Models.- Alternative Market Models.- Cross-currency Derivatives.

Martingale Methods in Financial Modelling

1 Introduction and Summary.- 1.1 Introduction.- 1.2 Markov control processes.- 1.3 Preliminary examples.- 1.4 Summary of the following chapters.- 2 Markov Control Processes.- 2.1 Introduction.- 2.2 Markov control processes.- 2.3 Markov policies and the Markov property.- 3 Finite-Horizon Problems.- 3.1 Introduction.- 3.2 Dynamic programming.- 3.3 The measurable selection condition.- 3.4 Variants of the DP equation.- 3.5 LQ problems.- 3.6 A consumption-investment problem.- 3.7 An inventory-production system.- 4 Infinite-Horizon Discounted-Cost Problems.- 4.1 Introduction.- 4.2 The discounted-cost optimality equation.- 4.3 Complements to the DCOE.- 4.4 Policy iteration and other approximations.- 4.5 Further optimality criteria.- 4.6 Asymptotic discount optimality.- 4.7 The discounted LQ problem.- 4.8 Concluding remarks.- 5 Long-Run Average-Cost Problems.- 5.1 Introduction.- 5.2 Canonical triplets.- 5.3 The vanishing discount approach.- 5.4 The average-cost optimality inequality.- 5.5 The average-cost optimality equation.- 5.6 Value iteration.- 5.7 Other optimality results.- 5.8 Concluding remarks.- 6 The Linear Programming Formulation.- 6.1 Introduction.- 6.2 Infinite-dimensional linear programming.- 6.3 Discounted cost.- 6.4 Average cost: preliminaries.- 6.5 Average cost: solvability.- 6.6 Further remarks.- Appendix A Miscellaneous Results.- Appendix B Conditional Expectation.- Appendix C Stochastic Kernels.- Appendix D Multifunctions and Selectors.- Appendix E Convergence of Probability Measures.- References.

Discrete-time Markov control processes

7 Ergodicity and Poisson's Equation.- 7.1 Introduction.- 7.2 Weighted norms and signed kernels.- A. Weighted-norm spaces.- B. Signed kernels.- C. Contraction maps.- 7.3 Recurrence concepts.- A. Irreducibility and recurrence.- B. Invariant measures.- C. Conditions for irreducibility and recurrence.- D. w-Geometric ergodicity.- 7.4 Examples on w-geometric ergodicity.- 7.5 Poisson's equation.- A. The multichain case.- B. The unichain P.E.- C. Examples.- 8 Discounted Dynamic Programming with Weighted Norms.- 8.1 Introduction.- 8.2 The control model and control policies.- 8.3 The optimality equation.- A. Assumptions.- B. The discounted-cost optimality equation.- C. The dynamic programming operator.- D. Proof of Theorem 8.3.6.- 8.4 Further analysis of value iteration.- A. Asymptotic discount optimality.- B. Estimates of VI convergence.- C. Rolling horizon procedures.- D. Forecast horizons and elimination of non-optimal actions.- 8.5 The weakly continuous case.- 8.6 Examples.- 8.7 Further remarks.- 9 The Expected Total Cost Criterion.- 9.1 Introduction.- 9.2 Preliminaries.- A. Extended real numbers.- B. Integrability.- 9.3 The expected total cost.- 9.4 Occupation measures.- A. Expected occupation measures.- B. The sufficiency problem.- 9.5 The optimality equation.- A. The optimality equation.- B. Optimality criteria.- C. Deterministic stationary policies.- 9.6 The transient case.- A. Transient models.- B. Optimality conditions.- C. Reduction to deterministic policies.- D. The policy iteration algorithm.- 10 Undiscounted Cost Criteria.- 10.1 Introduction.- A. Undiscounted criteria.- B. AC criteria.- C. Outline of the chapter.- 10.2 Preliminaries.- A. Assumptions.- B. Corollaries.- C. Discussion.- 10.3 From AC-optimality to undiscounted criteria.- A. The AC optimality inequality.- B. The AC optimality equation.- C. Uniqueness of the ACOE.- D. Bias-optimal policies.- E. Undiscounted criteria.- 10.4 Proof of Theorem 10.3.1.- A. Preliminary lemmas.- B. Completion of the proof.- 10.5 Proof of Theorem 10.3.6.- A. Proof of part (a).- B. Proof of part (b).- C. Policy iteration.- 10.6 Proof of Theorem 10.3.7.- 10.7 Proof of Theorem 10.3.10.- 10.8 Proof of Theorem 10.3.11.- 10.9 Examples.- 11 Sample Path Average Cost.- 11.1 Introduction.- A. Definitions.- B. Outline of the chapter.- 11.2 Preliminaries.- A. Positive Harris recurrence.- B. Limiting average variance.- 11.3 The w-geometrically ergodic case.- A. Optimality in IIDS.- B. Optimality in II.- C. Variance minimization.- D. Proof of Theorem 11.3.5.- E. Proof of Theorem 11.3.8.- 11.4 Strictly unbounded costs.- 11.5 Examples.- 12 The Linear Programming Approach.- 12.1 Introduction.- A. Outline of the chapter.- 12.2 Preliminaries.- A. Dual pairs of vector spaces.- B. Infinite linear programming.- C. Approximation of linear programs.- D. Tightness and invariant measures.- 12.3 Linear programs for the AC problem.- A. The linear programs.- B. Solvability of (P).- C. Absence of duality gap.- D. The Farkas alternative.- 12.4 Approximating sequences and strong duality.- A. Minimizing sequences for (P).- B. Maximizing sequences for (P*).- 12.5 Finite LP approximations.- A. Aggregation.- B. Aggregation-relaxation.- C. Aggregation-relaxion-inner approximations.- 12.6 Proof of Theorems 12.5.3, 12.5.5, 12.5.7.- References.- Abbreviations.- Glossary of notation.

Further Topics on Discrete-Time Markov Control Processes

The objective of this chapter is to introduce the stochastic control processes we are interested in; these are the so-called (discrete-time) controlled Markov processes (CMP’s), also known as Markov decision processes or Markov dynamic programs. The main part is Section 1.2. It contains some basic definitions and the statement of the optimal and the adaptive control problems studied in this book. In Section 1.3 we present several examples; the idea is to illustrate the main concepts and provide sources for possible applications. Also in Section 1.3 we discuss (briefly) more general control systems, such as non-stationary CMP’s and semi-Markov control models. The chapter is concluded in Section 1.4 with some comments on related references.

Controlled Markov Processes

1 Controlled Markov Processes.- 1.1 Introduction.- 1.2 Stochastic Control Problems.- Control Models.- Policies.- Performance Criteria.- Control Problems.- 1.3 Examples.- An Inventory/Production System.- Control of Water Reservoirs.- Fisheries Management.- Nonstationary MCM's.- Semi-Markov Control Models.- 1.4 Further Comments.- 2 Discounted Reward Criterion.- 2.1 Introduction.- Summary.- 2.2 Optimality Conditions.- Continuity of ?*.- 2.3 Asymptotic Discount Optimality.- 2.4 Approximation of MCM's.- Nonstationary Value-Iteration.- Finite-State Approximations.- 2.5 Adaptive Control Models.- Preliminaries.- Nonstationary Value-Iteration.- The Principle of Estimation and Control.- Adaptive Policies.- 2.6 Nonparametric Adaptive Control.- The Parametric Approach.- New Setting.- The Empirical Distribution Process.- Nonparametric Adaptive Policies.- 2.7 Comments and References.- 3 Average Reward Criterion.- 3.1 Introduction.- Summary.- 3.2 The Optimality Equation.- 3.3 Ergodicity Conditions.- 3.4 Value Iteration.- Uniform Approximations.- Successive Averagings.- 3.5 Approximating Models.- 3.6 Nonstationary Value Iteration.- Nonstationary Successive Averagings.- Discounted-Like NVI.- 3.7 Adaptive Control Models.- Preliminaries.- The Principle of Estimation and Control (PEC).- Nonstationary Value Iteration (NVI).- 3.8 Comments and References.- 4 Partially Observable Control Models.- 4.1 Introduction.- Summary.- 4.2 PO-CM: Case of Known Parameters.- The PO Control Problem.- 4.3 Transformation into a CO Control Problem.- I-Policies.- The New Control Model.- 4.4 Optimal I-Policies.- 4.5 PO-CM's with Unknown Parameters.- PEC and NVI I-Policies.- 4.6 Comments and References.- 5 Parameter Estimation in MCM's.- 5.1 Introduction.- Summary.- 5.2 Contrast Functions.- 5.3 Minimum Contrast Estimators.- 5.4 Comments and References.- 6 Discretization Procedures.- 6.1 Introduction.- Summary.- 6.2 Preliminaries.- 6.3 The Non-Adaptive Case.- A Non-Recursive Procedure.- A Recursive Procedure.- 6.4 Adaptive Control Problems.- Preliminaries.- Discretization of the PEC Adaptive Policy.- Discretization of the NVI Adaptive Policy.- 6.5 Proofs.- The Non-Adaptive Case.- The Adaptive Case.- 6.6 Comments and References.- Appendix A. Contraction Operators.- Appendix B. Probability Measures.- Total Variation Norm.- Weak Convergence.- Appendix C. Stochastic Kernels.- Appendix D. Multifunctions and Measurable Selectors.- The Hausdorff Metric.- Multifunctions.- References.- Author Index.

Adaptive Markov control processes

1 Preliminaries.- 1.1 Introduction.- 1.2 Measures and Functions.- 1.3 Weak Topologies.- 1.4 Convergence of Measures.- 1.5 Complements.- 1.6 Notes.- I Markov Chains and Ergodicity.- 2 Markov Chains and Ergodic Theorems.- 2.1 Introduction.- 2.2 Basic Notation and Definitions.- 2.3 Ergodic Theorems.- 2.4 The Ergodicity Property.- 2.5 Pathwise Results.- 2.6 Notes.- 3 Countable Markov Chains.- 3.1 Introduction.- 3.2 Classification of States and Class Properties.- 3.3 Limit Theorems.- 3.4 Notes.- 4 Harris Markov Chains.- 4.1 Introduction.- 4.2 Basic Definitions and Properties.- 4.3 Characterization of Harris recurrence.- 4.4 Sufficient Conditions for P.H.R.- 4.5 Harris and Doeblin Decompositions.- 4.6 Notes.- 5 Markov Chains in Metric Spaces.- 5.1 Introduction.- 5.2 The Limit in Ergodic Theorems.- 5.3 Yosida's Ergodic Decomposition.- 5.4 Pathwise Results.- 5.5 Proofs.- 5.6 Notes.- 6 Classification of Markov Chains via Occupation Measures.- 6.1 Introduction.- 6.2 A Classification.- 6.3 On the Birkhoff Individual Ergodic Theorem.- 6.4 Notes.- II Further Ergodicity Properties.- 7 Feller Markov Chains.- 7.1 Introduction.- 7.2 Weak-and Strong-Feller Markov Chains.- 7.3 Quasi Feller Chains.- 7.4 Notes.- 8 The Poisson Equation.- 8.1 Introduction.- 8.2 The Poisson Equation.- 8.3 Canonical Pairs.- 8.4 The Cesaro-Averages Approach.- 8.5 The Abelian Approach.- 8.6 Notes.- 9 Strong and Uniform Ergodicity.- 9.1 Introduction.- 9.2 Strong and Uniform Ergodicity.- 9.3 Weak and Weak Uniform Ergodicity.- 9.4 Notes.- III Existence and Approximation of Invariant Probability Measures.- 10 Existence of Invariant Probability Measures.- 10.1 Introduction and Statement of the Problems.- 10.2 Notation and Definitions.- 10.3 Existence Results.- 10.4 Markov Chains in Locally Compact Separable Metric Spaces.- 10.5 Other Existence Results in Locally Compact Separable Metric Spaces.- 10.6 Technical Preliminaries.- 10.7 Proofs.- 10.8 Notes.- 11 Existence and Uniqueness of Fixed Points for Markov Operators.- 11.1 Introduction and Statement of the Problems.- 11.2 Notation and Definitions.- 11.3 Existence Results.- 11.4 Proofs.- 11.5 Notes.- 12 Approximation Procedures for Invariant Probability Measures.- 12.1 Introduction.- 12.2 Statement of the Problem and Preliminaries.- 12.3 An Approximation Scheme.- 12.4 A Moment Approach for a Special Class of Markov Chains.- 12.5 Notes.

Onésimo Hernández-Lerma

Papers

Discrete-time Markov control processes

Further Topics on Discrete-Time Markov Control Processes

Controlled Markov Processes

Adaptive Markov control processes

Markov chains and invariant probabilities