The reinforcement learning paradigm is a popular way to address problems that have only limited environmental feedback, rather than correctly labeled examples, as is common in other machine learning contexts. While significant progress has been made to improve learning in a single task, the idea of transfer learning has only recently been applied to reinforcement learning tasks. The core idea of transfer is that experience gained in learning to perform one task can help improve learning performance in a related, but different, task. In this article we present a framework that classifies transfer learning methods in terms of their capabilities and goals, and then use it to survey the existing literature, as well as to suggest future directions for transfer learning work.

/pdf/transfer-learning-for-reinforcement-learning-domains-a-4d00ui6nyo.pdf

Transfer Learning for Reinforcement Learning Domains: A Survey

The theory of Markov Decision Processes is the theory of controlled Markov chains. Its origins can be traced back to R. Bellman and L. Shapley in the 1950’s. During the decades of the last century this theory has grown dramatically. It has found applications in various areas like e.g. computer science, engineering, operations research, biology and economics. In this article we give a short introduction to parts of this theory. We treat Markov Decision Processes with finite and infinite time horizon where we will restrict the presentation to the so-called (generalized) negative case. Solution algorithms like Howard’s policy improvement and linear programming are also explained. Various examples show the application of the theory. We treat stochastic linear-quadratic control problems, bandit problems and dividend pay-out problems.

/pdf/markov-decision-processes-3cwgsogfja.pdf

Markov Decision Processes

When I started out as a newly hatched PhD student, back in the day, one of the first articles I read and understood (or at least thought that I understood) was Ray Reiter’s classic article on default logic (Reiter, 1980).This was some years after the famous ‘non-monotonic logic’ issue of Artificial Intelligence in which that article appeared, but default logic was still one of the leading approaches, a tribute to the simplicity and power of the theory. As a result of reading the article, I became fascinated by both default logic and, more generally, non-monotonic logics. However, despite my fascination, these approaches never seemed terribly useful for the kinds of problem that I was supposed to be studying—problems like those in medical decision making—and so I eventually lost interest. In fact non-monotonic logics seemed to me, and to many people at the time I think, not to be terribly useful for anything. They were interesting, and clearly relevant to the long-term goals of Artificial Intelligence as a discipline, but not of any immediate practical importance. This verdict, delivered at the end of the 1980s, continued, I think, to be true for the next few years while researchers working in non-monotonic logics studied problems that to outsiders seemed to be ever more obscure. However, by the end of the 1990s, it was becoming clear, even to folk as short-sighted as I, that non-monotonic logics were getting to the point at which they could be used to solve practical problems. Knowledge in action shows quite how far these techniques have come. The reason that non-monotonic logics were invented was, of course, in order to use logic to reason about the world. Our knowledge of the world is typically incomplete, and so, in order to reason about it, one has to make assumptions about things one does not know. This, in turn, requires mechanisms for both making assumptions and then retracting them if and when they turn out not to be true. Non-monotonic logics are intended to handle this kind of assumption making and retracting, providing a mechanism that has the clean semantics of logic, but which has a non-monotonic set of conclusions. Much of the early work on non-monotonic logics was concerned with theoretical reasoning, that is reasoning about the beliefs of an agent—what the agent believes to be true. Theoretical reasoning is the domain of all those famous examples like ‘Typically birds fly. Tweety is a bird, so does Tweety fly?’, and the fact that so much of non-monotonic reasoning seemed to focus on theoretical reasoning was why I lost interest in it. I became much more concerned with practical reasoning—that is reasoning about what an agent should do—and non-monotonic reasoning seemed to me to have nothing interesting to say about practical reasoning. Of course I was wrong. When one tries to formulate any kind of description of the world as the basis for planning, one immediately runs into applications of non-monotonic logics, for example in keeping track of the state of a changing world. It is this use of non-monotonic logic that is at the heart of Knowledge in action. Building on the McCarthy’s situation calculus, Knowledge in action constructs a theory of action that encompasses a very large part of what an agent requires to reason about the world. As Reiter says in the final chapter,

Knowledge in Action: Logical Foundations for Specifying and Implementing Dynamical Systems

Learning goal-directed behavior in environments with sparse feedback is a major challenge for reinforcement learning algorithms. The primary difficulty arises due to insufficient exploration, resulting in an agent being unable to learn robust value functions. Intrinsically motivated agents can explore new behavior for its own sake rather than to directly solve problems. Such intrinsic behaviors could eventually help the agent solve tasks posed by the environment. We present hierarchical-DQN (h-DQN), a framework to integrate hierarchical value functions, operating at different temporal scales, with intrinsically motivated deep reinforcement learning. A top-level value function learns a policy over intrinsic goals, and a lower-level function learns a policy over atomic actions to satisfy the given goals. h-DQN allows for flexible goal specifications, such as functions over entities and relations. This provides an efficient space for exploration in complicated environments. We demonstrate the strength of our approach on two problems with very sparse, delayed feedback: (1) a complex discrete stochastic decision process, and (2) the classic ATARI game `Montezuma's Revenge'.

Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation

Learning goal-directed behavior in environments with sparse feedback is a major challenge for reinforcement learning algorithms. One of the key difficulties is insufficient exploration, resulting in an agent being unable to learn robust policies. Intrinsically motivated agents can explore new behavior for their own sake rather than to directly solve external goals. Such intrinsic behaviors could eventually help the agent solve tasks posed by the environment. We present hierarchical-DQN (h-DQN), a framework to integrate hierarchical action-value functions, operating at different temporal scales, with goal-driven intrinsically motivated deep reinforcement learning. A top-level q-value function learns a policy over intrinsic goals, while a lower-level function learns a policy over atomic actions to satisfy the given goals. h-DQN allows for flexible goal specifications, such as functions over entities and relations. This provides an efficient space for exploration in complicated environments. We demonstrate the strength of our approach on two problems with very sparse and delayed feedback: (1) a complex discrete stochastic decision process with stochastic transitions, and (2) the classic ATARI game -'Montezuma's Revenge'.

/pdf/hierarchical-deep-reinforcement-learning-integrating-1sijc68el0.pdf

Hierarchical deep reinforcement learning: integrating temporal abstraction and intrinsic motivation

A longstanding goal in planning research is the ability to generalize plans developed for some set of environments to a new but similar environment, with minimal or no replanning. Such generalization can both reduce planning time and allow us to tackle larger domains than the ones tractable for direct planning. In this paper, we present an approach to the generalization problem based on a new framework of relational Markov Decision Processes (RMDPs). An RMDP can model a set of similar environments by representing objects as instances of different classes. In order to generalize plans to multiple environments, we define an approximate value function specified in terms of classes of objects and, in a multiagent setting, by classes of agents. This class-based approximate value function is optimized relative to a sampled subset of environments, and computed using an efficient linear programming method. We prove that a polynomial number of sampled environments suffices to achieve performance close to the performance achievable when optimizing over the entire space. Our experimental results show that our method generalizes plans successfully to new, significantly larger, environments, with minimal loss of performance relative to environment-specific planning. We demonstrate our approach on a real strategic computer war game.

/pdf/generalizing-plans-to-new-environments-in-relational-mdps-280i89oicg.pdf

Neal Kanodia

Papers

Generalizing plans to new environments in relational MDPs