Rapid progress in machine learning and artificial intelligence (AI) has brought increasing attention to the potential impacts of AI technologies on society. In this paper we discuss one such potential impact: the problem of accidents in machine learning systems, defined as unintended and harmful behavior that may emerge from poor design of real-world AI systems. We present a list of five practical research problems related to accident risk, categorized according to whether the problem originates from having the wrong objective function ("avoiding side effects" and "avoiding reward hacking"), an objective function that is too expensive to evaluate frequently ("scalable supervision"), or undesirable behavior during the learning process ("safe exploration" and "distributional shift"). We review previous work in these areas as well as suggesting research directions with a focus on relevance to cutting-edge AI systems. Finally, we consider the high-level question of how to think most productively about the safety of forward-looking applications of AI.

Concrete Problems in AI Safety

* uschian groups of the first kind * Automorphic forms and functions * Hecke operators and the zeta-functions associated with modular forms * Elliptic curves * Abelian extensions of imaginary quadratic fields and complex multiplication of elliptic curves * Modular functions of higher level * Zeta-functions of algebraic curves and abelian varieties * The cohomology group assoicated with cusp forms * Arithmetic Fuschian groups

Introduction to the arithmetic theory of automorphic functions

In this tutorial article, we aim to provide the reader with the conceptual tools needed to get started on research on offline reinforcement learning algorithms: reinforcement learning algorithms that utilize previously collected data, without additional online data collection. Offline reinforcement learning algorithms hold tremendous promise for making it possible to turn large datasets into powerful decision making engines. Effective offline reinforcement learning methods would be able to extract policies with the maximum possible utility out of the available data, thereby allowing automation of a wide range of decision-making domains, from healthcare and education to robotics. However, the limitations of current algorithms make this difficult. We will aim to provide the reader with an understanding of these challenges, particularly in the context of modern deep reinforcement learning methods, and describe some potential solutions that have been explored in recent work to mitigate these challenges, along with recent applications, and a discussion of perspectives on open problems in the field.

Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems

Journal of the ACM

Machine learning offers a fantastically powerful toolkit for building useful complex prediction systems quickly. This paper argues it is dangerous to think of these quick wins as coming for free. Using the software engineering framework of technical debt, we find it is common to incur massive ongoing maintenance costs in real-world ML systems. We explore several ML-specific risk factors to account for in system design. These include boundary erosion, entanglement, hidden feedback loops, undeclared consumers, data dependencies, configuration issues, changes in the external world, and a variety of system-level anti-patterns.

/pdf/hidden-technical-debt-in-machine-learning-systems-2zp0mptk96.pdf

Hidden technical debt in Machine learning systems

This work shows how to leverage causal inference to understand the behavior of complex learning systems interacting with their environment and predict the consequences of changes to the system. Such predictions allow both humans and algorithms to select the changes that would have improved the system performance. This work is illustrated by experiments on the ad placement system associated with the Bing search engine.

/pdf/counterfactual-reasoning-and-learning-systems-the-example-of-3b5s0ib85y.pdf

Counterfactual reasoning and learning systems: the example of computational advertising

We propose constructing provable collision resistant hash functions from expander graphs in which finding cycles is hard. As examples, we investigate two specific families of optimal expander graphs for provable collision resistant hash function constructions: the families of Ramanujan graphs constructed by Lubotzky-Phillips-Sarnak and Pizer respectively. When the hash function is constructed from one of Pizers Ramanujan graphs, (the set of supersingular elliptic curves over with l-isogenies, l a prime different from p), then collision resistance follows from hardness of computing isogenies between supersingular elliptic curves. For the LPS graphs, the underlying hard problem is a representation problem in group theory. Constructing our hash functions from optimal expander graphs implies that the outputs closely approximate the uniform distribution. This property is useful for arguing that the output is indistinguishable from random sequences of bits. We estimate the cost per bit to compute these hash functions, and we implement our hash function for several members of the Pizer and LPS graph families and give actual timings.

/pdf/cryptographic-hash-functions-from-expander-graphs-4qzkjdgj1b.pdf

Cryptographic Hash Functions from Expander Graphs

This paper presents a practical digital signature scheme to be used in conjunction with network coding. Our scheme simultaneously provides authentication and detects malicious nodes that intentionally corrupt content on the network.

/pdf/signatures-for-network-coding-2ic5oao0sx.pdf

Signatures for Network Coding

This work shows how to leverage causal inference to understand the behavior of complex learning systems interacting with their environment and predict the consequences of changes to the system. Such predictions allow both humans and algorithms to select the changes that would have improved the system performance. This work is illustrated by experiments carried out on the ad placement system associated with the Bing search engine.

/pdf/counterfactual-reasoning-and-learning-systems-3qn3avekvc.pdf

Counterfactual Reasoning and Learning Systems

Labeling data is a seemingly simple task required for training many machine learning systems, but is actually fraught with problems. This paper introduces the notion of concept evolution, the changing nature of a person's underlying concept (the abstract notion of the target class a person is labeling for, e.g., spam email, travel related web pages) which can result in inconsistent labels and thus be detrimental to machine learning. We introduce two structured labeling solutions, a novel technique we propose for helping people define and refine their concept in a consistent manner as they label. Through a series of five experiments, including a controlled lab study, we illustrate the impact and dynamics of concept evolution in practice and show that structured labeling helps people label more consistently in the presence of concept evolution than traditional labeling.

https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/p3075.pdf

Denis X. Charles

Papers

Counterfactual reasoning and learning systems: the example of computational advertising

Cryptographic Hash Functions from Expander Graphs

Signatures for Network Coding

Counterfactual Reasoning and Learning Systems

Structured labeling for facilitating concept evolution in machine learning