Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning.

/pdf/reinforcement-learning-an-introduction-rzxgej9p17.pdf

Reinforcement Learning: An Introduction

https://is.tuebingen.mpg.de/fileadmin/user_upload/files/publications/Neural-Netw-2008-21-682_4867[0].pdf

2008 Special Issue: Reinforcement learning of motor skills with policy gradients

Policy-gradient-based actor-critic algorithms are amongst the most popular algorithms in the reinforcement learning framework. Their advantage of being able to search for optimal policies using low-variance gradient estimates has made them useful in several real-life applications, such as robotics, power control, and finance. Although general surveys on reinforcement learning techniques already exist, no survey is specifically dedicated to actor-critic algorithms in particular. This paper, therefore, describes the state of the art of actor-critic algorithms, with a focus on methods that can work in an online setting and use function approximation in order to deal with continuous state and action spaces. After starting with a discussion on the concepts of reinforcement learning and the origins of actor-critic algorithms, this paper describes the workings of the natural gradient, which has made its way into many actor-critic algorithms over the past few years. A review of several standard and natural actor-critic algorithms is given, and the paper concludes with an overview of application areas and a discussion on open issues.

/pdf/a-survey-of-actor-critic-reinforcement-learning-standard-and-ya29rk9vmw.pdf

A Survey of Actor-Critic Reinforcement Learning: Standard and Natural Policy Gradients

This paper investigates a novel model-free reinforcement learning architecture, the Natural Actor-Critic. The actor updates are based on stochastic policy gradients employing Amari's natural gradient approach, while the critic obtains both the natural policy gradient and additional parameters of a value function simultaneously by linear regression. We show that actor improvements with natural policy gradients are particularly appealing as these are independent of coordinate frame of the chosen policy representation, and can be estimated more efficiently than regular policy gradients. The critic makes use of a special basis function parameterization motivated by the policy-gradient compatible function approximation. We show that several well-known reinforcement learning methods such as the original Actor-Critic and Bradtke's Linear Quadratic Q-Learning are in fact Natural Actor-Critic algorithms. Empirical evaluations illustrate the effectiveness of our techniques in comparison to previous methods, and also demonstrate their applicability for learning control on an anthropomorphic robot arm.

Natural Actor-Critic

https://homes.cs.washington.edu/~todorov/courses/amath579/reading/NaturalActorCritic.pdf

Recently, actor-critic methods have drawn much interests in the area of reinforcement learning, and several algorithms have been studied along the line of the actor-critic strategy. This paper studies an actor-critic type algorithm utilizing the RLS(recursive least-squares) method, which is one of the most efficient techniques for adaptive signal processing, together with natural policy gradient. In the actor part of the studied algorithm, we follow the strategy of performing parameter update via the natural gradient method, while in its update for the critic part, the recursive least-squares method is employed in order to make the parameter estimation for the value functions more efficient. The studied algorithm was applied to locomotion of a two-linked robot arm, and showed better performance compared to the conventional stochastic gradient ascent algorithm.

/pdf/an-rls-based-natural-actor-critic-algorithm-for-locomotion-2xc1v693mr.pdf

An RLS-based natural actor-critic algorithm for locomotion of a two-linked robot arm

The SVDD (support vector data description) is one of the most well-known one-class support vector learning methods, in which one tries the strategy of utilizing balls defined on the feature space in order to distinguish a set of normal data from all other possible abnormal objects. In this paper, we consider the problem of reconstructing facial images from the partially damaged ones, and propose to use the SVDD-based de-noising for the reconstruction. In the proposed method, we deal with the shape and texture information separately. We first solve the SVDD problem for the data belonging to the given prototype facial images, and model the data region for the normal faces as the ball resulting from the SVDD problem. Next, for each damaged input facial image, we project its feature vector onto the decision boundary of the SVDD ball so that it can be tailored enough to belong to the normal region. Finally, we obtain the image of the reconstructed face by obtaining the pre-image of the projection, and then further processing with its shape and texture information. The applicability of the proposed method is illustrated via some experiments dealing with damaged facial images.

Facial image reconstruction by SVDD-based pattern de-noising

/pdf/facial-image-reconstruction-by-svdd-based-pattern-de-noising-36hzha0tug.pdf

Daesung Kang

Papers

An RLS-based natural actor-critic algorithm for locomotion of a two-linked robot arm

Facial image reconstruction by SVDD-based pattern de-noising

Facial image reconstruction by SVDD-Based pattern de-noising