Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning.

/pdf/reinforcement-learning-an-introduction-rzxgej9p17.pdf

Reinforcement Learning: An Introduction

The authors present random early detection (RED) gateways for congestion avoidance in packet-switched networks. The gateway detects incipient congestion by computing the average queue size. The gateway could notify connections of congestion either by dropping packets arriving at the gateway or by setting a bit in packet headers. When the average queue size exceeds a present threshold, the gateway drops or marks each arriving packet with a certain probability, where the exact probability is a function of the average queue size. RED gateways keep the average queue size low while allowing occasional bursts of packets in the queue. During congestion, the probability that the gateway notifies a particular connection to reduce its window is roughly proportional to that connection's share of the bandwidth through the gateway. RED gateways are designed to accompany a transport-layer congestion control protocol such as TCP. The RED gateway has no bias against bursty traffic and avoids the global synchronization of many connections decreasing their window at the same time. Simulations of a TCP/IP network are used to illustrate the performance of RED gateways. >

/pdf/random-early-detection-gateways-for-congestion-avoidance-34o7r9dqub.pdf

Random early detection gateways for congestion avoidance

In this book, Andrew Harvey sets out to provide a unified and comprehensive theory of structural time series models. Unlike the traditional ARIMA models, structural time series models consist explicitly of unobserved components, such as trends and seasonals, which have a direct interpretation. As a result the model selection methodology associated with structural models is much closer to econometric methodology. The link with econometrics is made even closer by the natural way in which the models can be extended to include explanatory variables and to cope with multivariate time series. From the technical point of view, state space models and the Kalman filter play a key role in the statistical treatment of structural time series models. The book includes a detailed treatment of the Kalman filter. This technique was originally developed in control engineering, but is becoming increasingly important in fields such as economics and operations research. This book is concerned primarily with modelling economic and social time series, and with addressing the special problems which the treatment of such series poses. The properties of the models and the methodological techniques used to select them are illustrated with various applications. These range from the modellling of trends and cycles in US macroeconomic time series to to an evaluation of the effects of seat belt legislation in the UK.

Forecasting, Structural Time Series Models and the Kalman Filter

The End of History and the Last Man

A tutorial account of variable structure control with sliding mode is presented. The purpose is to introduce in a concise manner the fundamental theory, main results, and practical applications of this powerful control system design approach. This approach is particularly attractive for the control of nonlinear systems. Prominent characteristics such as invariance, robustness, order reduction, and control chattering are discussed in detail. Methods for coping with chattering are presented. Both linear and nonlinear systems are considered. Future research areas are suggested and an extensive list of references is included. >

Variable structure control: a survey

Recursive Estimation and Time Series Analysis

Parameter estimation for continuous-time models-A survey

This is a revised version of the 1984 book of the same name but considerably modified and enlarged to accommodate the developments in recursive estimation and time series analysis that have occurred over the last quarter century. Also over this time, the CAPTAIN Toolbox for recursive estimation and time series analysis has been developed at Lancaster, for use in the MatlabTM software environment (see Appendix G). Consequently, the present version of the book is able to exploit the many computational routines that are contained in this widely available Toolbox, as well as some of the other routines in MatlabTM and its other toolboxes.The book is an introductory one on the topic of recursive estimation and it demonstrates how this approach to estimation, in its various forms, can be an impressive aid to the modelling of stochastic, dynamic systems. It is intended for undergraduate or Masters students who wish to obtain a grounding in this subject; or for practitioners in industry who may have heard of topics dealt with in this book and, while they want to know more about them, may have been deterred by the rather esoteric nature of some books in this challenging area of study.

Recursive Estimation and Time-Series Analysis: An Introduction for the Student and Practitioner

An instrumental variable method for real-time identification of a noisy process

This paper describes in detail a flexible approach to nonstationary time series analysis based on a Dynamic Harmonic Regression (DHR) model of the Unobserved Components (UC) type, formulated within a stochastic state space setting. The model is particularly useful for adaptive seasonal adjustment, signal extraction and interpolation over gaps, as well as forecasting or backcasting. The Kalman Filter and Fixed Interval Smoothing algorithms are exploited for estimating the various components, with the Noise Variance Ratio and other hyperparameters in the stochastic state space model estimated by a novel optimization method in the frequency domain. Unlike other approaches of this general type, which normally exploit Maximum Likelihood methods, this optimization procedure is based on a cost function defined in terms of the difference between the logarithmic pseudo-spectrum of the DHR model and the logarithmic autoregressive spectrum of the time series. The cost function not only seems to yield improved convergence characteristics when compared with the alternative ML cost function, but it also has much reduced numerical requirements.

/pdf/dynamic-harmonic-regression-18bjc839ts.pdf

Peter C. Young

Papers

Recursive Estimation and Time Series Analysis

Parameter estimation for continuous-time models-A survey

Recursive Estimation and Time-Series Analysis: An Introduction for the Student and Practitioner

An instrumental variable method for real-time identification of a noisy process

Dynamic harmonic regression.