Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning.

/pdf/reinforcement-learning-an-introduction-rzxgej9p17.pdf

Reinforcement Learning: An Introduction

https://www2.econ.iastate.edu/tesfatsi/DeepLearningInNeuralNetworksOverview.JSchmidhuber2015.pdf

Deep learning in neural networks

This article presents a general class of associative reinforcement learning algorithms for connectionist networks containing stochastic units. These algorithms, called REINFORCE algorithms, are shown to make weight adjustments in a direction that lies along the gradient of expected reinforcement in both immediate-reinforcement tasks and certain limited forms of delayed-reinforcement tasks, and they do this without explicitly computing gradient estimates or even storing information from which such estimates could be computed. Specific examples of such algorithms are presented, some of which bear a close relationship to certain existing algorithms while others are novel but potentially interesting in their own right. Also given are results that show how such algorithms can be naturally integrated with backpropagation. We close with a brief discussion of a number of additional issues surrounding the use of such algorithms, including what is known about their limiting behaviors as well as further considerations that might be used to help develop similar but potentially more powerful reinforcement learning algorithms.

/pdf/simple-statistical-gradient-following-algorithms-for-39sidawxep.pdf

Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning

This paper surveys the field of reinforcement learning from a computer-science perspective. It is written to be accessible to researchers familiar with machine learning. Both the historical basis of the field and a broad selection of current work are summarized. Reinforcement learning is the problem faced by an agent that learns behavior through trial-and-error interactions with a dynamic environment. The work described here has a resemblance to work in psychology, but differs considerably in the details and in the use of the word "reinforcement." The paper discusses central issues of reinforcement learning, including trading off exploration and exploitation, establishing the foundations of the field via Markov decision theory, learning from delayed reinforcement, constructing empirical models to accelerate learning, making use of generalization and hierarchy, and coping with hidden state. It concludes with a survey of some implemented systems and an assessment of the practical utility of current methods for reinforcement learning.

/pdf/reinforcement-learning-a-survey-38lmyylg2y.pdf

Reinforcement learning: a survey

This paper surveys the field of reinforcement learning from a computer-science perspective. It is written to be accessible to researchers familiar with machine learning. Both the historical basis of the field and a broad selection of current work are summarized. Reinforcement learning is the problem faced by an agent that learns behavior through trial-and-error interactions with a dynamic environment. The work described here has a resemblance to work in psychology, but differs considerably in the details and in the use of the word ``reinforcement.'' The paper discusses central issues of reinforcement learning, including trading off exploration and exploitation, establishing the foundations of the field via Markov decision theory, learning from delayed reinforcement, constructing empirical models to accelerate learning, making use of generalization and hierarchy, and coping with hidden state. It concludes with a survey of some implemented systems and an assessment of the practical utility of current methods for reinforcement learning.

Reinforcement Learning: A Survey

The problem of learning correct decision rules to minimize the probability of misclassification is a long-standing problem of supervised learning in pattern recognition. The problem of learning such optimal discriminant functions is considered for the class of problems where the statistical properties of the pattern classes are completely unknown. The problem is posed as a game with common payoff played by a team of mutually cooperating learning automata. This essentially results in a probabilistic search through the space of classifiers. The approach is inherently capable of learning discriminant functions that are nonlinear in their parameters also. A learning algorithm is presented for the team and convergence is established. It is proved that the team can obtain the optimal classifier to an arbitrary approximation. Simulation results with a few examples are presented where the team learns the optimal classifier.

Learning Optimal Discriminant Functions through a Cooperative Game of Automata

Relaxation labeling processes are a class of mechanisms that solve the problem of assigning labels to objects in a manner that is consistent with respect to some domain-specific constraints. We reformulate this using the model of a team of learning automata interacting with an environment or a high-level critic that gives noisy responses as to the consistency of a tentative labeling selected by the automata. This results in an iterative linear algorithm that is itself probabilistic. Using an explicit definition of consistency we give a complete analysis of this probabilistic relaxation process using weak convergence results for stochastic algorithms. Our model can accommodate a range of uncertainties in the compatibility functions. We prove a local convergence result and show that the point of convergence depends both on the initial labeling and the constraints. The algorithm is implementable in a highly parallel fashion.

Relaxation Labeling with Learning Automata

In vector quantization (VQ), minimization of mean square error (MSE) between code book vectors and training vectors is a non-linear problem. Traditional Linde-Buzo-Gray (1980) type of algorithms converge to a local minimum, which depends on the initial code book. While most of the efforts in VQ have been directed towards designing effcient search algorithms for the code book, little has been done in evolving a procedure to obtain an optimum code book. This paper addresses the problem of designing a globally optimum code book using genetic algorithms (GAs). A hybrid GA called the genetic K-means algorithm (GKA) that combines the advantages of gradient descent algorithms and GAs have been proposed previously. In this algorithm, the gradient descent part helps to speed up the algorithm whereas the GA features help to obtain a global optimum. The standard K-means algorithm is used instead of crossover and a distance based mutation is defined specifically for this problem to effectively perturb the solutions. The GKA has been proved to converge to a global optimum with probability one. We applied GKA to VQ for image compression. Since most of the code book design algorithms are based on the LBG we compared the performance of the GKA with that of the LBG algorithm.

/pdf/vector-quantization-using-genetic-k-means-algorithm-for-4rjusfr9l8.pdf

Vector quantization using genetic K-means algorithm for image compression

New criteria in the multiplier form are presented for the input-output stability in the L 2 -space of a linear system with a time-varying element k(t) in a feedback loop. These are sufficient conditions for the system stability and involve conditions on the shifted imaginary-axis behavior of the multipliers. The criteria permit the use of noncausal multipliers, and it is shown that this necessitates dk/dt to be bounded from above as well as from below. The method of derivation draws on the theory of positivity of compositions of operators and time-varying gains, and the results are shown to be more general than the existing criteria.

L_{2} -stability of linear time-varying systems--Conditions involving noncausal multipliers

This paper considers the problem of asymptotic stability in the large of an autonomous system containing a single nonlinearity. The nonlinear function is assumed to belong to several subclasses of monotonically increasing functions in the sector (0, K) , and the stability criterion is shown to be of the form Re Z(j\omega) [G(j\omega) +\frac{1}{K}] -\frac{\delta'}{K} \geq 0 where the constant \delta' is equal to Z(\infty) - Z_{p}(\infty) and Z_{p}(s) is a Popov multiplier. The multiplier Z(s) can, in general, have complex conjugate poles and zeros and is thus more general than the type of multipliers obtained in previous results. The nonlinear functions considered are odd monotonic functions, functions with a power law restriction, and a new class of functions with restricted asymmetry having the property |\frac{f{\theta}{f{\-theta}|\leq c for all \theta . Unlike in certain earlier publications, no upper bound is placed on the derivative of the nonlinearity here. The results obtained can be used to establish stability in some cases even when the Nyquist plot of the linear part transfer function lies in all four quadrants and the nonlinearity is not necessarily odd. Furthermore, it is shown that the conditions on the multiplier and, consequently, those on the linear part can be relaxed as the feedback function approaches linearity.

M. A. L. Thathachar

Papers

Learning Optimal Discriminant Functions through a Cooperative Game of Automata

Relaxation Labeling with Learning Automata

Vector quantization using genetic K-means algorithm for image compression

L_{2} -stability of linear time-varying systems--Conditions involving noncausal multipliers

On a modified Lur'e problem