Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning.

/pdf/reinforcement-learning-an-introduction-rzxgej9p17.pdf

Reinforcement Learning: An Introduction

Reinforcement learning policies face the exploration versus exploitation dilemma, i.e. the search for a balance between exploring the environment to find profitable actions while taking the empirically best action as often as possible. A popular measure of a policy's success in addressing this dilemma is the regret, that is the loss due to the fact that the globally optimal policy is not followed all the times. One of the simplest examples of the exploration/exploitation dilemma is the multi-armed bandit problem. Lai and Robbins were the first ones to show that the regret for this problem has to grow at least logarithmically in the number of plays. Since then, policies which asymptotically achieve this regret have been devised by Lai and Robbins and many others. In this work we show that the optimal logarithmic regret is also achievable uniformly over time, with simple and efficient policies, and for all reward distributions with bounded support.

/pdf/finite-time-analysis-of-the-multiarmed-bandit-problem-4ydrjmqvvu.pdf

Finite-time Analysis of the Multiarmed Bandit Problem

/pdf/convex-analysisnoer-san-nojin-zhan-nituite-5dl2rbmoyr.pdf

Convex Analysisの二,三の進展について

Convergence of Probability Measures. By P. Billingsley. Chichester, Sussex, Wiley, 1968. xii, 253 p. 9 1/4“. 117s.

Convergence of Probability Measures

In 57 normal subjects (age 20-60 years), we analyzed the spontaneous beat-to-beat oscillation in R-R interval during control recumbent position, 90 degrees upright tilt, controlled respiration (n = 16) and acute (n = 10) and chronic (n = 12) beta-adrenergic receptor blockade. Automatic computer analysis provided the autoregressive power spectral density, as well as the number and relative power of the individual components. The power spectral density of R-R interval variability contained two major components in power, a high frequency at approximately 0.25 Hz and a low frequency at approximately 0.1 Hz, with a normalized low frequency:high frequency ratio of 3.6 +/- 0.7. With tilt, the low-frequency component became largely predominant (90 +/- 1%) with a low frequency:high frequency ratio of 21 +/- 4. Acute beta-adrenergic receptor blockade (0.2 mg/kg IV propranolol) increased variance at rest and markedly blunted the increase in low frequency and low frequency:high frequency ratio induced by tilt. Chronic beta-adrenergic receptor blockade (0.6 mg/kg p.o. propranolol, t.i.d.), in addition, reduced low frequency and increased high frequency at rest, while limiting the low frequency:high frequency ratio increase produced by tilt. Controlled respiration produced at rest a marked increase in the high-frequency component, with a reduction of the low-frequency component and of the low frequency:high frequency ratio (0.7 +/- 0.1); during tilt, the increase in the low frequency:high frequency ratio (8.3 +/- 1.6) was significantly smaller. In seven additional subjects in whom direct high-fidelity arterial pressure was recorded, simultaneous R-R interval and arterial pressure variabilities were examined at rest and during tilt. Also, the power spectral density of arterial pressure variability contained two major components, with a relative low frequency:high frequency ratio at rest of 2.8 +/- 0.7, which became 17 +/- 5 with tilt. These power spectral density components were numerically similar to those observed in R-R variability. Thus, invasive and noninvasive studies provided similar results. More direct information on the role of cardiac sympathetic nerves on R-R and arterial pressure variabilities was derived from a group of experiments in conscious dogs before and after bilateral stellectomy. Under control conditions, high frequency was predominant and low frequency was very small or absent, owing to a predominant vagal tone. During a 9% decrease in arterial pressure obtained with IV nitroglycerin, there was a marked increase in low frequency, as a result of reflex sympathetic activation.(ABSTRACT TRUNCATED AT 400 WORDS)

/pdf/power-spectral-analysis-of-heart-rate-and-arterial-pressure-4km9looazn.pdf

Power spectral analysis of heart rate and arterial pressure variabilities as a marker of sympatho-vagal interaction in man and conscious dog.

Asymptotically efficient adaptive allocation rules

Strong consistency and asymptotic normality of least squares estimates in stochastic regression models are established under certain weak assumptions on the stochastic regressors and errors. We discuss applications of these results to interval estimation of the regression parameters and to recursive on-line identification and control schemes for linear dynamic systems.

/pdf/least-squares-estimates-in-stochastic-regression-models-with-3llvr7lggu.pdf

Least Squares Estimates in Stochastic Regression Models with Applications to Identification and Control of Dynamic Systems

After a brief survey of a large variety of sequential detection procedures that are widely scattered in statistical references on quality control and engineering references on fault detection and signal processing, we study some open problems concerning these procedures and introduce a unified theory of sequential changepoint detection. This theory leads to a class of sequential detection rules which are not too demanding in computational and memory requirements for on-line implementation and yet are nearly optimal under several performance criteria.

https://rss.onlinelibrary.wiley.com/doi/pdf/10.1111/j.2517-6161.1995.tb02052.x

Sequential changepoint detection in quality control and dynamical systems

By using information-theoretic bounds and sequential hypothesis testing theory, this paper provides a new approach to optimal detection of abrupt changes in stochastic systems. This approach not only generalizes previous work in the literature on optimal detection far beyond the relatively simple models treated but also suggests alternative performance criteria which are more tractable and more appropriate for general stochastic systems. In addition, it leads to detection rules which have manageable computational complexity for on-line implementation and yet are nearly optimal under the different performance criteria considered.

Information bounds and quick detection of parameter changes in stochastic systems

We give a brief review of the developments in several classical problems of sequential analysis and their applications to biomedicine, economics and engi- neering. Even though it can only focus on a limited number of topics, the review shows that sequential analysis is still a vibrant subject after six decades of contin- ual development, with fresh ideas brought in from various fields of application and through interactions with other branches of statistics and probability. We conclude with some remarks on the opportunities and challenges ahead.

/pdf/sequential-analysis-some-classical-problems-and-new-4nma4mdg86.pdf

Tze Leung Lai

Papers

Asymptotically efficient adaptive allocation rules

Least Squares Estimates in Stochastic Regression Models with Applications to Identification and Control of Dynamic Systems

Sequential changepoint detection in quality control and dynamical systems

Information bounds and quick detection of parameter changes in stochastic systems

Sequential analysis: some classical problems and new challenges