On April 13th, 2019, OpenAI Five became the first AI system to defeat the world champions at an esports game. The game of Dota 2 presents novel challenges for AI systems such as long time horizons, imperfect information, and complex, continuous state-action spaces, all challenges which will become increasingly central to more capable AI systems. OpenAI Five leveraged existing reinforcement learning techniques, scaled to learn from batches of approximately 2 million frames every 2 seconds. We developed a distributed training system and tools for continual training which allowed us to train OpenAI Five for 10 months. By defeating the Dota 2 world champion (Team OG), OpenAI Five demonstrates that self-play reinforcement learning can achieve superhuman performance on a difficult task.

Dota 2 with Large Scale Deep Reinforcement Learning

The field of meta-learning, or learning-to-learn, has seen a dramatic rise in interest in recent years. Contrary to conventional approaches to AI where tasks are solved from scratch using a fixed learning algorithm, meta-learning aims to improve the learning algorithm itself, given the experience of multiple learning episodes. This paradigm provides an opportunity to tackle many conventional challenges of deep learning, including data and computation bottlenecks, as well as generalization. This survey describes the contemporary meta-learning landscape. We first discuss definitions of meta-learning and position it with respect to related fields, such as transfer learning and hyperparameter optimization. We then propose a new taxonomy that provides a more comprehensive breakdown of the space of meta-learning methods today. We survey promising applications and successes of meta-learning such as few-shot learning and reinforcement learning. Finally, we discuss outstanding challenges and promising areas for future research.

Meta-Learning in Neural Networks: A Survey

Recent advances in artificial intelligence have been driven by the presence of increasingly realistic and complex simulated environments However, many of the existing environments provide either unrealistic visuals, inaccurate physics, low task complexity, restricted agent perspective, or a limited capacity for interaction among artificial agents Furthermore, many platforms lack the ability to flexibly configure the simulation, making the simulated environment a black-box from the perspective of the learning system In this work, we propose a novel taxonomy of existing simulation platforms and discuss the highest level class of general platforms which enable the development of learning environments that are rich in visual, physical, task, and social complexity We argue that modern game engines are uniquely suited to act as general platforms and as a case study examine the Unity engine and open source Unity ML-Agents Toolkit We then survey the research enabled by Unity and the Unity ML-Agents Toolkit, discussing the kinds of research a flexible, interactive and easily configurable general platform can facilitate

Unity: A General Platform for Intelligent Agents

/pdf/meta-world-a-benchmark-and-evaluation-for-multi-task-and-wom9oupayh.pdf

Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning

We propose a simple data augmentation technique that can be applied to standard model-free reinforcement learning algorithms, enabling robust learning directly from pixels without the need for auxiliary losses or pre-training. The approach leverages input perturbations commonly used in computer vision tasks to regularize the value function. Existing model-free approaches, such as Soft Actor-Critic (SAC), are not able to train deep networks effectively from image pixels. However, the addition of our augmentation method dramatically improves SAC's performance, enabling it to reach state-of-the-art performance on the DeepMind control suite, surpassing model-based (Dreamer, PlaNet, and SLAC) methods and recently proposed contrastive learning (CURL). Our approach can be combined with any model-free reinforcement learning algorithm, requiring only minor modifications. An implementation can be found at this https URL.

Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels

In this paper, we investigate the problem of overfitting in deep reinforcement learning. Among the most common benchmarks in RL, it is customary to use the same environments for both training and testing. This practice offers relatively little insight into an agent's ability to generalize. We address this issue by using procedurally generated environments to construct distinct training and test sets. Most notably, we introduce a new environment called CoinRun, designed as a benchmark for generalization in RL. Using CoinRun, we find that agents overfit to surprisingly large training sets. We then show that deeper convolutional architectures improve generalization, as do methods traditionally found in supervised learning, including L2 regularization, dropout, data augmentation and batch normalization.

/pdf/quantifying-generalization-in-reinforcement-learning-qiqztnodaw.pdf

Quantifying Generalization in Reinforcement Learning

We study how to leverage off-the-shelf visual and linguistic data to cope with out-of-vocabulary answers in visual question answering task. Existing large-scale visual datasets with annotations such as image class labels, bounding boxes and region descriptions are good sources for learning rich and diverse visual concepts. However, it is not straightforward how the visual concepts can be captured and transferred to visual question answering models due to missing link between question dependent answering models and visual data without question. We tackle this problem in two steps: 1) learning a task conditional visual classifier, which is capable of solving diverse question-specific visual recognition tasks, based on unsupervised task discovery and 2) transferring the task conditional visual classifier to visual question answering models. Specifically, we employ linguistic knowledge sources such as structured lexical database (e.g. WordNet) and visual descriptions for unsupervised task discovery, and transfer a learned task conditional visual classifier as an answering unit in a visual question answering model. We empirically show that the proposed algorithm generalizes to out-of-vocabulary answers successfully using the knowledge transferred from the visual dataset.

/pdf/transfer-learning-via-unsupervised-task-discovery-for-visual-wt3wzby6wi.pdf

Transfer Learning via Unsupervised Task Discovery for Visual Question Answering

Whether epigenetic factors such as DNA methylation and microRNAs interact to control adult hippocampal neurogenesis is not fully understood. Here, we show that Down syndrome critical region 1 (DSCR1) protein plays a key role in adult hippocampal neurogenesis by modulating two epigenetic factors: TET1 and miR-124. We find that DSCR1 mutant mice have impaired adult hippocampal neurogenesis. DSCR1 binds to TET1 introns to regulate splicing of TET1, thereby modulating TET1 level. Furthermore, TET1 controls the demethylation of the miRNA-124 promoter to modulate miR-124 expression. Correcting the level of TET1 in DSCR1 knockout mice is sufficient to prevent defective adult neurogenesis. Importantly, restoring DSCR1 level in a Down syndrome mouse model effectively rescued adult neurogenesis and learning and memory deficits. Our study reveals that DSCR1 plays a critical upstream role in epigenetic regulation of adult neurogenesis and provides insights into potential therapeutic strategy for treating cognitive defects in Down syndrome.

/pdf/dscr1-mediated-tet1-splicing-regulates-mir-124-expression-to-4ddwle3zod.pdf

DSCR1-mediated TET1 splicing regulates miR-124 expression to control adult hippocampal neurogenesis.

To understand how specific interventions affect a process observed over time, we need to control for the other factors that influence outcomes. Such a model that captures all factors other than the one of interest is generally known as a baseline. In our study of how different pricing schemes affect residential electricity consumption, the baseline would need to capture the impact of outdoor temperature along with many other factors. In this work, we examine a number of different data mining techniques and demonstrate Gradient Tree Boosting (GTB) to be an effective method to build the baseline. We train GTB on data prior to the introduction of new pricing schemes, and apply the known temperature following the introduction of new pricing schemes to predict electricity usage with the expected temperature correction. Our experiments and analyses show that the baseline models generated by GTB capture the core characteristics over the two years with the new pricing schemes. In contrast to the majority of regression based techniques which fail to capture the lag between the peak of daily temperature and the peak of electricity usage, the GTB generated baselines are able to correctly capture the delay between the temperature peak and the electricity peak. Furthermore, subtracting this temperature-adjusted baseline from the observed electricity usage, we find that the resulting values are more amenable to interpretation, which demonstrates that the temperature-adjusted baseline is indeed effective.

Taehoon Kim

Papers

Quantifying Generalization in Reinforcement Learning

Quantifying Generalization in Reinforcement Learning

Transfer Learning via Unsupervised Task Discovery for Visual Question Answering

DSCR1-mediated TET1 splicing regulates miR-124 expression to control adult hippocampal neurogenesis.

Extracting Baseline Electricity Usage Using Gradient Tree Boosting