Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning.

/pdf/reinforcement-learning-an-introduction-rzxgej9p17.pdf

Reinforcement Learning: An Introduction

Machine Learning is the study of methods for programming computers to learn. Computers are applied to a wide range of tasks, and for most of these it is relatively easy for programmers to design and implement the necessary software. However, there are many tasks for which this is difficult or impossible. These can be divided into four general categories. First, there are problems for which there exist no human experts. For example, in modern automated manufacturing facilities, there is a need to predict machine failures before they occur by analyzing sensor readings. Because the machines are new, there are no human experts who can be interviewed by a programmer to provide the knowledge necessary to build a computer system. A machine learning system can study recorded data and subsequent machine failures and learn prediction rules. Second, there are problems where human experts exist, but where they are unable to explain their expertise. This is the case in many perceptual tasks, such as speech recognition, hand-writing recognition, and natural language understanding. Virtually all humans exhibit expert-level abilities on these tasks, but none of them can describe the detailed steps that they follow as they perform them. Fortunately, humans can provide machines with examples of the inputs and correct outputs for these tasks, so machine learning algorithms can learn to map the inputs to the outputs. Third, there are problems where phenomena are changing rapidly. In finance, for example, people would like to predict the future behavior of the stock market, of consumer purchases, or of exchange rates. These behaviors change frequently, so that even if a programmer could construct a good predictive computer program, it would need to be rewritten frequently. A learning program can relieve the programmer of this burden by constantly modifying and tuning a set of learned prediction rules. Fourth, there are applications that need to be customized for each computer user separately. Consider, for example, a program to filter unwanted electronic mail messages. Different users will need different filters. It is unreasonable to expect each user to program his or her own rules, and it is infeasible to provide every user with a software engineer to keep the rules up-to-date. A machine learning system can learn which mail messages the user rejects and maintain the filtering rules automatically. Machine learning addresses many of the same research questions as the fields of statistics, data mining, and psychology, but with differences of emphasis. Statistics focuses on understanding the phenomena that have generated the data, often with the goal of testing different hypotheses about those phenomena. Data mining seeks to find patterns in the data that are understandable by people. Psychological studies of human learning aspire to understand the mechanisms underlying the various learning behaviors exhibited by people (concept learning, skill acquisition, strategy change, etc.).

Machine learning

The leading and most up-to-date textbook on the far-ranging algorithmic methododogy of Dynamic Programming, which can be used for optimal control, Markovian decision problems, planning and sequential decision making under uncertainty, and discrete/combinatorial optimization. The treatment focuses on basic unifying themes, and conceptual foundations. It illustrates the versatility, power, and generality of the method with many examples and applications from engineering, operations research, and other fields. It also addresses extensively the practical application of the methodology, possibly through the use of approximations, and provides an extensive treatment of the far-reaching methodology of Neuro-Dynamic Programming/Reinforcement Learning.

/pdf/dynamic-programming-and-optimal-control-21dopm2oo5.pdf

Dynamic Programming and Optimal Control

/pdf/convex-analysisnoer-san-nojin-zhan-nituite-5dl2rbmoyr.pdf

Convex Analysisの二,三の進展について

Function approximation is essential to reinforcement learning, but the standard approach of approximating a value function and determining a policy from it has so far proven theoretically intractable. In this paper we explore an alternative approach in which the policy is explicitly represented by its own function approximator, independent of the value function, and is updated according to the gradient of expected reward with respect to the policy parameters. Williams's REINFORCE method and actor-critic methods are examples of this approach. Our main new result is to show that the gradient can be written in a form suitable for estimation from experience aided by an approximate action-value or advantage function. Using this result, we prove for the first time that a version of policy iteration with arbitrary differentiable function approximation is convergent to a locally optimal policy.

/pdf/policy-gradient-methods-for-reinforcement-learning-with-4dw6389kcb.pdf

Policy Gradient Methods for Reinforcement Learning with Function Approximation

1. Introduction to Discrete Event Dynamic Systems.- 1.1 Introduction.- 1.2 Models of DEDS.- 2. Introduction to Perturbation Analysis.- 2.1 Notations.- 2.2 A Short History of the Perturbation Analysis Development.- 3. Informal Treatment of the Infinitesimal Perturbation Analysis (IPA) 2.- 3.1 The Basic Idea.- 3.2 Single Class Queueing Networks.- 3.3 A GSMP Formalism for IPA.- 3.4 Realization Ratios.- 3.5 The GI/G/1 Queue.- 3.6 Load Dependent Server and IPA.- 3.7 Remarks about PA.- 4. Foundation of Infinitesimal Perturbation Analysis.- 4.1 Sample Derivative and Interchangeability.- 4.2 Perturbation Analysis for Closed Jackson Network.- 1. Sample Performance Functions.- 2. Sample Derivative of Performance in Transient Periods.- 3. Derivative of Steady State Performance.- 4.3 Realization and Sensitivity Analysis.- 1. Realization Probability.- 2. Realization Index and The Convergence Theorems.- 3. Sensitivity of Throughputs.- 4. Calculation of Other Sensitivities.- 4.4 Sensitivity Analysis of Networks with General Service Distributions.- 4.5 IPA Estimates of the M/G/1 and GI/G/1 Queue.- 1. M/G/1 Queue: A Direct Approach.- 2. M/G/1 Queue: The Approach Based on Stochastic Convexity.- 4.6 Some Technical Proofs.- 5. Extensions of IPA.- 5.1 System Representation and PA.- 5.2 Another Sufficient Condition for Interchangeability.- 1. The Basic Idea.- 2. A sufficient condition based on the generalized semi Markov process model.- 3 Queueing network examples.- 5.3 Routing Probability Sensitivity.- 5.4 The Multiclass M/G/1 Queue and Networks.- 1. The Two Class M/M/1 Queue.- 2. The Rescheduling Approach.- 3. Approximate MultiClass Network Analysis and IPA.- 5.5 Smoothed Perturbation Analysis (SPA).- 6. Finite Perturbation Analysis.- 6.1 The Idea of "Cut-and-Paste".- 1. A Simple Example of "Cut-and-Paste".- 2. State vs. Event Matching.- 6.2 The State Matching vs. Event Matching Algorithms.- 1. Analytical Comparison of State Matching Algorithms vs. Event Matching Algorithms for a Simple System.- 2. Empirical Validation.- 6.3 "Cut-and-Paste" as a Generalization of the Rejection Method.- 6.4 First Order Propagation Rules and Approximate EPA.- 7. General Sensitivity Analysis.- 7.1 Efficient Sample Path Generation.- 1. The Standard Clock and the M/M/1 Example.- 2. IPA via the Standard Clock.- 3. The Alias Method of Choosing Event Types.- 7.2 Trajectory Projection and State Augmentation.- 1. Trajectory Projection of the Automata Model.- 2. State Augmentation Method for Markov Chains.- 7.3 The Likelihood Ratio Approach.- 1. A Markov Chain Example and Variance Reduction.- 2. Basic Analysis.- 3. An Example of the Variances of PA and LR Estimates.- 4. Comparison of the PA and Likelihood Ratio Estimates.- 5. DEDS models Revisited.- 8. Aggregation, Decomposition, and Equivalence.- 8.1 Equivalent Server and Aggregation.- 1. Product-Form Networks.- 2. General Networks.- 8.2 Perturbation Analysis on Aggregated Systems and Aggregated PA.- 1. Perturbation Analysis of Aggregated Systems.- 2. Aggregated Perturbation Analysis.- 3. Aggregation of Multiclass Queueing Networks -An Example.- 8.3 Aggregation and Decomposition of Markov Chains.- 1. The Case of v'(s) = 0.- 2. The Case Where v'(s) Can Be Calculated in Closed Form.- 8.4 Decomposition via the A-Segment Algorithm for very Large Markov Chains.- Appendix A. Elements of Queueing Theory.- Appendix B. elements of Discrete Event Simulation.- Appendix C. Elements of Optimization and Control Theory 3.- Appendix D. A Simple Illustrative Example of the Different Models of DEDS.- Appendix E. A Sample Program of Perturbation Analysis.- References.

Perturbation Analysis of Discrete Event Dynamic Systems

A new time-domain-based approach is developed in this paper for the perturbation analysis of queueing networks. We show that, by observing a single sample path realization of the network trajectory, we can derive sensitivity information of the throughput of the system with respect to various parameters. This information can then be used for the optimization of queueing networks. Numerous experiments as well as analytical results demonstrating the validity of this new approach are given and discussed.

Perturbation analysis and optimization of queueing networks

This paper identifies and studies two major issues in the blind source separation problem: separability and separation principles. We show that separability is an intrinsic property of the measured signals and can be described by the concept of m-row decomposability introduced in this paper; we also show that separation principles can be developed by using the structure characterization theory of random variables. In particular, we show that these principles can be derived concisely and intuitively by applying the Darmois-Skitovich theorem, which is well known in statistical inference theory and psychology. Some new insights are gained for designing blind source separation filters.

General approach to blind source separation

Performance optimization is vital in the design and operation of modern engineering systems, including communications, manufacturing, robotics, and logistics. Most engineering systems are too complicated to model, or the system parameters cannot be easily identified, so learning techniques have to be applied. This book provides a unified framework based on a sensitivity point of view. It also introduces new approaches and proposes new research topics within this sensitivity-based framework. This new perspective on a popular topic is presented by a well respected expert in the field.

Stochastic Learning and Optimization - A Sensitivity-Based Approach

Two fundamental concepts and quantities, realization factors and performance potentials, are introduced for Markov processes. The relations among these two quantities and the group inverse of the infinitesimal generator are studied. It is shown that the sensitivity of the steady-state performance with respect to the change of the infinitesimal generator can be easily calculated by using either of these three quantities and that these quantities can be estimated by analyzing a single sample path of a Markov process. Based on these results, algorithms for estimating performance sensitivities on a single sample path of a Markov process can be proposed. The potentials in this paper are defined through realization factors and are shown to be the same as those defined by Poisson equations. The results provide a uniform framework of perturbation realization for infinitesimal perturbation analysis (IPA) and non-IPA approaches to the sensitivity analysis of steady-state performance; they also provide a theoretical background for the PA algorithms developed in recent years.

/pdf/perturbation-realization-potentials-and-sensitivity-analysis-2es3dc9t48.pdf

Xi-Ren Cao

Papers

Perturbation Analysis of Discrete Event Dynamic Systems

Perturbation analysis and optimization of queueing networks

General approach to blind source separation

Stochastic Learning and Optimization - A Sensitivity-Based Approach

Perturbation realization, potentials, and sensitivity analysis of Markov processes