In this paper we argue for the fundamental importance of the value distribution: the distribution of the random return received by a reinforcement learning agent. This is in contrast to the common approach to reinforcement learning which models the expectation of this return, or value. Although there is an established body of literature studying the value distribution, thus far it has always been used for a specific purpose such as implementing risk-aware behaviour. We begin with theoretical results in both the policy evaluation and control settings, exposing a significant distributional instability in the latter. We then use the distributional perspective to design a new algorithm which applies Bellman's equation to the learning of approximate value distributions. We evaluate our algorithm using the suite of games from the Arcade Learning Environment. We obtain both state-of-the-art results and anecdotal evidence demonstrating the importance of the value distribution in approximate reinforcement learning. Finally, we combine theoretical and empirical evidence to highlight the ways in which the value distribution impacts learning in the approximate setting.

/pdf/a-distributional-perspective-on-reinforcement-learning-7jsqb7htw2.pdf

A Distributional Perspective on Reinforcement Learning

This paper develops a model with asymmetrically informed agents and costly monitoring of loan contracts, where an equilibrium can exhibit credit rationing. Borrowers are identical ex ante, but some receive loans and others do not. In contrast to existing credit rationing theories, rationing does not occur here due to inflexible prices, adverse selection or moral hazard. Optimizing behaviour produces a standard debt contract in equilibrium. The aggregate quantity of loans and equilibrium interest rates respond differently depending on whether there is rationing in equilibrium.

https://www.jstor.org/stable/info/1884684

Costly Monitoring, Loan Contracts, and Equilibrium Credit Rationing

We consider differentially private algorithms for convex empirical risk minimization (ERM). Differential privacy (Dwork et al., 2006b) is a recently introduced notion of privacy which guarantees that an algorithm’s output does not depend on the data of any individual in the dataset. This is crucial in fields that handle sensitive data, such as genomics, collaborative filtering, and economics. Our motivation is the design of private algorithms for sparse learning problems, in which one aims to find solutions (e.g., regression parameters) with few non-zero coefficients. To this end: (a) We significantly extend the analysis of the “objective perturbation” algorithm of Chaudhuri et al. (2011) for convex ERM problems. We show that their method can be modified to use less noise (be more accurate), and to apply to problems with hard constraints and non-differentiable regularizers. We also give a tighter, data-dependent analysis of the additional error introduced by their method. A key tool in our analysis is a new nontrivial limit theorem for differential privacy which is of independent interest: if a sequence of differentially private algorithms converges, in a weak sense, then the limit algorithm is also differentially private. In particular, our methods give the best known algorithms for differentially private linear regression. These methods work in settings where the number of parametersp is less than the number of samplesn. (b) We give the first two private algorithms for sparse regression problems in high-dimensional settings, where p is much larger than n. We analyze their performance for linear regression: under standard assumptions on the data, our algorithms have vanishing empirical risk for n = poly(s; logp) when there exists a good regression vector with s nonzero coefficients. Our algorithms demonstrate that randomized algorithms for sparse regression problems can be both stable and accurate ‐ a combination which is impossible for deterministic algorithms.

http://www.cse.psu.edu/~duk17/papers/PrivateERM.pdf

Private Convex Empirical Risk Minimization and High-dimensional Regression

Learning-based anomaly detection systems build models of the expected behavior of applications by analyzing events that are generated during their normal operation. Once these models have been established, subsequent events are analyzed to identify deviations, given the assumption that anomalies usually represent evidence of an attack.

/pdf/on-the-detection-of-anomalous-system-call-arguments-2u7e2ji3qf.pdf

On the detection of anomalous system call arguments

Multistage stochastic programming - in contrast to stochastic control - has found wideapplication in the formulation and solution of financial problems characterized by a largenumber of state variables and a generally low number of possible decision stages. Theliterature on the use of multistage recourse modelling to formalize complex portfolio optimizationproblems dates back to the early seventies, when the technique was first adopted tosolve a fixed income security portfolio problem. We present here the CALM model, whichhas been designed to deal with uncertainty affecting both assets (in either the portfolio orthe market) and liabilities (in the form of scenario dependent payments or borrowing costs).We consider as an instance a pension fund problem in which portfolio rebalancing is allowedover a long-term horizon at discrete time points and where liabilities refer to five differentclasses of pension contracts. The portfolio manager, given an initial wealth, seeks the maximizationof terminal wealth at the horizon, with investment returns modelled as discretestate random vectors. Decision vectors represent possible investments in the market andholding or selling assets in the portfolio, as well as borrowing decisions from a credit lineor deposits with a bank. Computational results are presented for a set of 10-stage portfolioproblems using different solution methods and libraries (OSL, CPLEX, OB1). The portfolioproblem, with an underlying vector data process which allows up to 2688 realizations at the10-year horizon, is solved on an IBM RS6000y590 for a set of twenty-four large-scale testproblems using the simplex and barrier methods provided by CPLEX (the latter for eitherlinear or quadratic objective), the predictorycorrector interior point method provided in OB1,the simplex method of OSL, the MSLiP-OSL code instantiating nested Benders decompositionwith subproblem solution using OSL simplex, and the current version of MSLiP.

L.E. Clarke

Papers

Probability and measure, by Patrick Billingsley. Pp 515. £15·20. 1979. SBN 0 471 03173 9 (Wiley)