In this article, we describe a method for optimizing control policies, with guaranteed monotonic improvement. By making several approximations to the theoretically-justified scheme, we develop a practical algorithm, called Trust Region Policy Optimization (TRPO). This algorithm is effective for optimizing large nonlinear policies such as neural networks. Our experiments demonstrate its robust performance on a wide variety of tasks: learning simulated robotic swimming, hopping, and walking gaits; and playing Atari games using images of the screen as input. Despite its approximations that deviate from the theory, TRPO tends to give monotonic improvement, with little tuning of hyperparameters.

/pdf/trust-region-policy-optimization-3bqnki35vf.pdf

Trust Region Policy Optimization

We describe an iterative procedure for optimizing policies, with guaranteed monotonic improvement. By making several approximations to the theoretically-justified procedure, we develop a practical algorithm, called Trust Region Policy Optimization (TRPO). This algorithm is similar to natural policy gradient methods and is effective for optimizing large nonlinear policies such as neural networks. Our experiments demonstrate its robust performance on a wide variety of tasks: learning simulated robotic swimming, hopping, and walking gaits; and playing Atari games using images of the screen as input. Despite its approximations that deviate from the theory, TRPO tends to give monotonic improvement, with little tuning of hyperparameters.

The goal of decentralized optimization over a network is to optimize a global objective formed by a sum of local (possibly nonsmooth) convex functions using only local computation and communication. It arises in various application domains, including distributed tracking and localization, multi-agent coordination, estimation in sensor networks, and large-scale machine learning. We develop and analyze distributed algorithms based on dual subgradient averaging, and we provide sharp bounds on their convergence rates as a function of the network size and topology. Our analysis allows us to clearly separate the convergence of the optimization algorithm itself and the effects of communication dependent on the network structure. We show that the number of iterations required by our algorithm scales inversely in the spectral gap of the network, and confirm this prediction's sharpness both by theoretical lower bounds and simulations for various networks. Our approach includes the cases of deterministic optimization and communication, as well as problems with stochastic optimization and/or communication.

Dual Averaging for Distributed Optimization: Convergence Analysis and Network Scaling

We provide a framework for studying the relationship between the financial network architecture and the likelihood of systemic failures due to contagion of counterparty risk. We show that financial contagion exhibits a form of phase transition as interbank connections increase: as long as the magnitude and the number of negative shocks affecting financial institutions are sufficiently small, more "complete" interbank claims enhance the stability of the system. However, beyond a certain point, such interconnections start to serve as a mechanism for propagation of shocks and lead to a more fragile financial system. We also show that, under natural contracting assumptions, financial networks that emerge in equilibrium may be socially inefficient due to the presence of a network externality: even though banks take the effects of their lending, risk-taking and failure on their immediate creditors into account, they do not internalize the consequences of their actions on the rest of the network.

https://papers.ssrn.com/sol3/Delivery.cfm/SSRN_ID2500504_code1428438.pdf?abstractid=2207439&mirid=5&type=2

Systemic Risk and Stability in Financial Networks

This paper argues that the extent of financial contagion exhibits a form of phase transition: as long as the magnitude of negative shocks affecting financial institutions are sufficiently small, a more densely connected financial network (corresponding to a more diversified pattern of interbank liabilities) enhances financial stability. However, beyond a certain point, dense interconnections serve as a mechanism for the propagation of shocks, leading to a more fragile financial system. Our results thus highlight that the same factors that contribute to resilience under certain conditions may function as significant sources of systemic risk under others. (JEL D85, E44, G21, G28, L14)

/pdf/systemic-risk-and-stability-in-financial-networks-ecudzsz6lj.pdf

Systemic risk and stability in financial networks

This book is an introduction to the modern approach to the theory of Markov chains. The main goal of this approach is to determine the rate of convergence of a Markov chain to the stationary distribution as a function of the size and geometry of the state space. The authors develop the key tools for estimating convergence times, including coupling, strong stationary times, and spectral methods. Whenever possible, probabilistic methods are emphasized. The book includes many examples and provides brief introductions to some central models of statistical mechanics. Also provided are accounts of random walks on networks, including hitting and cover times, and analyses of several methods of shuffling cards. As a prerequisite, the authors assume a modest understanding of probability theory and linear algebra at an undergraduate level. ""Markov Chains and Mixing Times"" is meant to bring the excitement of this active area of research to a wide audience.

Markov Chains and Mixing Times

Extending results of Christie and Irving, we examine the action of reversals and transpositions on finite strings over an alphabet of size k. We show that determining reversal, transposition, or signed reversal distance between two strings over a finite alphabet is NP-hard, while for "dense" instances we give a polynomial-time approximation scheme. We also give a number of extremal results, as well as investigating the distance between random strings and the problem of sorting a string over a finite alphabet.

/pdf/reversals-and-transpositions-over-finite-alphabets-3m1hevkael.pdf

Reversals and Transpositions Over Finite Alphabets

The Poincaré and Cheeger bounds are two useful bounds for the second largest eigenvalue of a reversible Markov chain. Diaconis and Stroock [1991] and Jerrum and Sinclair [1989] develop versions of these bounds which involve choosing paths. This paper studies these path-related bounds and shows that the Poincaré bound is superior to the Cheeger bound for simple random walk on a tree and random walk on a finite group with any symmetric generating set. This partially resolves a question posed by Diaconis and Stroock [1991].

Elizabeth Wilmer

Papers

Markov Chains and Mixing Times

Reversals and Transpositions Over Finite Alphabets

Comparing eigenvalue bounds for markov chains: when does poincare beat cheeger?

Graphs induced by Gray codes

Combinatorics in the exterior algebra and the Bollobás Two Families Theorem