Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning.

/pdf/reinforcement-learning-an-introduction-rzxgej9p17.pdf

Reinforcement Learning: An Introduction

The purpose of this paper is to provide a comprehensive presentation and interpretation of the Ensemble Kalman Filter (EnKF) and its numerical implementation. The EnKF has a large user group, and numerous publications have discussed applications and theoretical aspects of it. This paper reviews the important results from these studies and also presents new ideas and alternative interpretations which further explain the success of the EnKF. In addition to providing the theoretical framework needed for using the EnKF, there is also a focus on the algorithmic formulation and optimal numerical implementation. A program listing is given for some of the key subroutines. The paper also touches upon specific issues such as the use of nonlinear measurements, in situ profiles of temperature and salinity, and data which are available with high frequency in time. An ensemble based optimal interpolation (EnOI) scheme is presented as a cost-effective approach which may serve as an alternative to the EnKF in some applications. A fairly extensive discussion is devoted to the use of time correlated model errors and the estimation of model bias.

The Ensemble Kalman Filter: Theoretical formulation and practical implementation

Introduction to linear and nonlinear programming

Combinatorial Group Theory: COMBINATORIAL GROUP THEORY

Changes in monsoon rainfall are caused by human-produced aerosols slowing the tropical atmospheric circulation. Observations show that South Asia underwent a widespread summertime drying during the second half of the 20th century, but it is unclear whether this trend was due to natural variations or human activities. We used a series of climate model experiments to investigate the South Asian monsoon response to natural and anthropogenic forcings. We find that the observed precipitation decrease can be attributed mainly to human-influenced aerosol emissions. The drying is a robust outcome of a slowdown of the tropical meridional overturning circulation, which compensates for the aerosol-induced energy imbalance between the Northern and Southern Hemispheres. These results provide compelling evidence of the prominent role of aerosols in shaping regional climate change over South Asia.

Anthropogenic aerosols and the weakening of the South Asian summer monsoon

Dynamic data assimilation is the assessment, combination and synthesis of observational data, scientific laws and mathematical models to determine the state of a complex physical system, for instance as a preliminary step in making predictions about the system's behaviour. The topic has assumed increasing importance in fields such as numerical weather prediction where conscientious efforts are being made to extend the term of reliable weather forecasts beyond the few days that are presently feasible. This book is designed to be a basic one-stop reference for graduate students and researchers. It is based on graduate courses taught over a decade to mathematicians, scientists, and engineers, and its modular structure accommodates the various audience requirements. Thus Part I is a broad introduction to the history, development and philosophy of data assimilation, illustrated by examples; Part II considers the classical, static approaches, both linear and nonlinear; and Part III describes computational techniques. Parts IV to VII are concerned with how statistical and dynamic ideas can be incorporated into the classical framework. Key themes covered here include estimation theory, stochastic and dynamic models, and sequential filtering. The final part addresses the predictability of dynamical systems. Chapters end with a section that provides pointers to the literature, and a set of exercises with instructive hints.

Dynamic Data Assimilation: A Least Squares Approach

This survey provides a comprehensive and unified analysis of symmetry in a wide variety of Cayley graphs of permutation groups. These include the star graph, bubble-sort graph, modified bubble-sort graph, complete-transposition graph, prefix-reversal graph, alternating-group graph, binary and base-b (b ≥ 3) hypercube, cube connected cycles, bisectional graph, folded hypercube and binary orthogonal graph. In addition, we also define a variety of generalizations of the hypercube and orthogonal graphs. The types of symmetry analyzed include vertex and edge transitivity, distance regularity and distance transitivity. Since these notions of symmetry depend on the shortest paths, as a by product we also describe the shortest path routing algorithms for these graphs. We present a number of open problems related to the networks described in this paper.

Symmetry in interconnection networks based on Cayley graphs of permutation groups: a survey

Publisher Summary This chapter presents a survey on various parallel sorting algorithms. Sorting is a nontrivial problem and has widespread commercial and business applications. Serial algorithms for sorting have been available since the days of punched-card machines. At present, there is a considerable body of literature on serial sorting algorithms. Parallel algorithms for sorting are of a recent origin and came into existence over the past decade. The chapter presents a unified treatment of various parallel sorting algorithms by bringing out clearly the relation between the architecture of parallel computers and the structure of algorithms. In the design of parallel algorithms in general, and of parallel sorting algorithms in particular, two models have been widely used: (1) models based on fixed interconnection networks such as the same or single instruction on multiple data (SIMD) machine mesh-connected network and (2) models based on a global memory, which is shared by various processors. The special-purpose network-sorting algorithms are described. Algorithms for SIMD machines are given.

Parallel Sorting Algorithms

This paper introduces a new class of interconnection scheme based on the Cayley graph of the alternating group. It is shown that this class of graphs are edge symmetric and 2-transitive. We then describe an algorithm for (a) packet routing based on the shortest path analysis, (b) finding a Hamiltonian cycle, (c) ranking and unranking along the chosen Hamiltonian cycle, (d) unit expansion and dilation three embedding of a class of two-dimensional grids, (e) unit dilation embedding of a variety of cycles, and (f) algorithm for broadcasting messages. The paper concludes with a short analysis of contention resulting from a typical communication scheme. Although this class of graphs does not possess many of the symmetry properties of the binary hypercube, with respect to the one source broadcasting, these graphs perform better than does a hypercube, and with respect to the contention problem, these graphs perform better than do the star graphs and are close to the hypercube. © 1993 by John Wiley & Sons, Inc.

A new class of interconnection networks based on the alternating group

The use of the star graph as a viable interconnection scheme for parallel computers has been examined by a number of authors in recent times An attractive feature of this class of graphs is that it has sublogarithmic diameter and has a great deal of symmetry akin to the binary hypercube In this paper we describe a new class of algorithms for embedding (a) Hamiltonian cycle (b) the set of all even cycles and (c) a variety of two- and multi-dimensional grids in a star graph In addition, we also derive an algorithm for the ranking and the unranking problem with respect to the Hamiltonian cycle

Sivaramakrishnan Lakshmivarahan

Papers

Dynamic Data Assimilation: A Least Squares Approach

Symmetry in interconnection networks based on Cayley graphs of permutation groups: a survey

Parallel Sorting Algorithms

A new class of interconnection networks based on the alternating group

Embedding of cycles and grids in star graphs