Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning.

/pdf/reinforcement-learning-an-introduction-rzxgej9p17.pdf

Reinforcement Learning: An Introduction

Learning to act using real-time dynamic programming

Recently developd Newton and quasi-Newton methods for nonlinear programming possess only local convergence properties. Adopting the concept of the damped Newton method in unconstrained optimization, we propose a stepsize procedure to maintain monotone decrease of an exact penalty function. In so doing, the convergence of the method is globalized. Keywords: nonlinear programming, global convergence, exact penalty function.

/pdf/a-globally-convergent-method-for-nonlinear-programming-4onk7d29pe.pdf

A Globally Convergent Method for Nonlinear Programming

This paper considers generalized equations, which are convenient tools for formulating problems in complementarity and in mathematical programming, as well as variational inequalities. We introduce a regularity condition for such problems and, with its help, prove existence, uniqueness and Lipschitz continuity of solutions to generalized equations with parametric data. Applications to nonlinear programming and to other areas are discussed, and for important classes of such applications the regularity condition given here is shown to be in a certain sense the weakest possible condition under which the stated properties will hold.

https://www.jstor.org/stable/pdfplus/3689393.pdf

Strongly Regular Generalized Equations

https://hal.inria.fr/hal-00840470/document

Natural actor-critic algorithms

This paper studies how the solution of the problem of minimizingQ(x) = 1/2x

 T
 
Kx − k

 T
 
x subject toGx ≦ g andDx = d behaves whenK, k, G, g, D andd are perturbed, say by terms of size∈, assuming thatK is positive definite. It is shown that in general the solution moves by roughly∈ ifG, g, D andd are not perturbed; whenG, g, D andd are in fact perturbed, much stronger hypotheses allow one to show that the solution moves by roughly∈. Many of these results can be extended to more general, nonquadratic, functionals.

Stability of the solution of definite quadratic programs

A Newton-type algorithm has been presented elsewhere for solving non-linear inequalities of the formf(x)?0,g(x)=0, and quadratic convergence has been proved under very strong hypotheses. In this paper we show that the same results hold under a considerable weakening of the hypotheses.

Newton's method for nonlinear inequalities

Splines and efficiency in dynamic programming

The iterated deferred correction of the standard fourth-order equally spaced finite-difference (Numerov’s or Cowell’s) method for second-order two-point boundary-value problems is implemented by an improved version of the codes developed by Pereyra. The improvements are of a general nature so as to be applicable to the deferred correction of other basic schemes. Results of numerical experiments are summarized.

James W. Daniel

Papers

Stability of the solution of definite quadratic programs

Newton's method for nonlinear inequalities

Splines and efficiency in dynamic programming

Numerov's Method with Deferred Corrections for Two-Point Boundary-Value Problems

The continuity of metric projections as functions of the data