https://is.tuebingen.mpg.de/fileadmin/user_upload/files/publications/Neural-Netw-2008-21-682_4867[0].pdf

2008 Special Issue: Reinforcement learning of motor skills with policy gradients

Policy-gradient-based actor-critic algorithms are amongst the most popular algorithms in the reinforcement learning framework. Their advantage of being able to search for optimal policies using low-variance gradient estimates has made them useful in several real-life applications, such as robotics, power control, and finance. Although general surveys on reinforcement learning techniques already exist, no survey is specifically dedicated to actor-critic algorithms in particular. This paper, therefore, describes the state of the art of actor-critic algorithms, with a focus on methods that can work in an online setting and use function approximation in order to deal with continuous state and action spaces. After starting with a discussion on the concepts of reinforcement learning and the origins of actor-critic algorithms, this paper describes the workings of the natural gradient, which has made its way into many actor-critic algorithms over the past few years. A review of several standard and natural actor-critic algorithms is given, and the paper concludes with an overview of application areas and a discussion on open issues.

/pdf/a-survey-of-actor-critic-reinforcement-learning-standard-and-ya29rk9vmw.pdf

A Survey of Actor-Critic Reinforcement Learning: Standard and Natural Policy Gradients

The acquisition and improvement of motor skills and control policies for robotics from trial and error is of essential importance if robots should ever leave precisely pre-structured environments. However, to date only few existing reinforcement learning methods have been scaled into the domains of high-dimensional robots such as manipulator, legged or humanoid robots. Policy gradient methods remain one of the few exceptions and have found a variety of applications. Nevertheless, the application of such methods is not without peril if done in an uninformed manner. In this paper, we give an overview on learning with policy gradient methods for robotics with a strong focus on recent advances in the field. We outline previous applications to robotics and show how the most recently developed methods can significantly improve learning performance. Finally, we evaluate our most promising algorithm in the application of hitting a baseball with an anthropomorphic arm

/pdf/policy-gradient-methods-for-robotics-2y9601e3d7.pdf

Policy Gradient Methods for Robotics

https://courses.nus.edu.sg/course/cheld/internet/CN5111/Reference/Retrospective%20on%20optimization.pdf

Retrospective on optimization

A formalism is presented for fitting dynamic forecast models to asynoptic data. Because of the importance of wind stress forcing in oceanic models and of the inadequacies of wind stress observations, the formalism allows an oceanic model to be fit to both Oceanographic and meteorological data. Within the context of this formalism the important question of whether an asynoptic data set contains sufficient information to determine the model state completely and unambiguously is discussed. Because the information travels along wave characteristics, it is clear that for the data to be sufficient to determine the model state, they must be distributed so that every feature of the flow is seen at some time or another. Such widespread coverage of the oceans requires a data collection system that relies heavily on satellites. The formalism is illustrated using a highly truncated model of the wind-driven equatorial ocean and computational examples demonstrate how surface elevation and wind stress observations might be used to recover the model state.

Fitting dynamics to data

The book represents an introduction to computation in control by an iterative, gradient, numerical method, where linearity is not assumed. The general language and approach used are those of elementary functional analysis. The particular gradient method that is emphasized and used is conjugate gradient descent, a well known method exhibiting quadratic convergence while requiring very little more computation than simple steepest descent. Constraints are not dealt with directly, but rather the approach is to introduce them as penalty terms in the criterion. General conjugate gradient descent methods are developed and applied to problems in control.

Lawrence Hasdorff

Papers

Gradient Optimization and Nonlinear Control