In this article, we describe an automatic differentiation module of PyTorch — a library designed to enable rapid research on machine learning models. It builds upon a few projects, most notably Lua Torch, Chainer, and HIPS Autograd [4], and provides a high performance environment with easy access to automatic differentiation of models executed on different devices (CPU and GPU). To make prototyping easier, PyTorch does not follow the symbolic approach used in many other deep learning frameworks, but focuses on differentiation of purely imperative programs, with a focus on extensibility and low overhead. Note that this preprint is a draft of certain sections from an upcoming paper covering all PyTorch features.

/pdf/automatic-differentiation-in-pytorch-gbuafkf7qo.pdf

Automatic differentiation in PyTorch

Deposits of clastic carbonate-dominated (calciclastic) sedimentary slope systems in the rock record have been identified mostly as linearly-consistent carbonate apron deposits, even though most ancient clastic carbonate slope deposits fit the submarine fan systems better. Calciclastic submarine fans are consequently rarely described and are poorly understood. Subsequently, very little is known especially in mud-dominated calciclastic submarine fan systems. Presented in this study are a sedimentological core and petrographic characterisation of samples from eleven boreholes from the Lower Carboniferous of Bowland Basin (Northwest England) that reveals a >250 m thick calciturbidite complex deposited in a calciclastic submarine fan setting. Seven facies are recognised from core and thin section characterisation and are grouped into three carbonate turbidite sequences. They include: 1) Calciturbidites, comprising mostly of highto low-density, wavy-laminated bioclast-rich facies; 2) low-density densite mudstones which are characterised by planar laminated and unlaminated muddominated facies; and 3) Calcidebrites which are muddy or hyper-concentrated debrisflow deposits occurring as poorly-sorted, chaotic, mud-supported floatstones. These

Phd by thesis

The brms package implements Bayesian multilevel models in R using the probabilistic programming language Stan. A wide range of distributions and link functions are supported, allowing users to fit - among others - linear, robust linear, binomial, Poisson, survival, ordinal, zero-inflated, hurdle, and even non-linear models all in a multilevel context. Further modeling options include autocorrelation of the response variable, user defined covariance structures, censored data, as well as meta-analytic standard errors. Prior specifications are flexible and explicitly encourage users to apply prior distributions that actually reflect their beliefs. In addition, model fit can easily be assessed and compared with the Watanabe-Akaike information criterion and leave-one-out cross-validation.

brms: An R Package for Bayesian Multilevel Models Using Stan

Structured additive regression models are perhaps the most commonly used class of models in statistical applications. It includes, among others, (generalized) linear models, (generalized) additive models, smoothing spline models, state space models, semiparametric regression, spatial and spatiotemporal models, log-Gaussian Cox processes and geostatistical and geoadditive models. We consider approximate Bayesian inference in a popular subset of structured additive regression models, latent Gaussian models, where the latent field is Gaussian, controlled by a few hyperparameters and with non-Gaussian response variables. The posterior marginals are not available in closed form owing to the non-Gaussian response variables. For such models, Markov chain Monte Carlo methods can be implemented, but they are not without problems, in terms of both convergence and computational time. In some practical applications, the extent of these problems is such that Markov chain Monte Carlo sampling is simply not an appropriate tool for routine analysis. We show that, by using an integrated nested Laplace approximation and its simplified version, we can directly compute very accurate approximations to the posterior marginals. The main benefit of these approximations is computational: where Markov chain Monte Carlo algorithms need hours or days to run, our approximations provide more precise estimates in seconds or minutes. Another advantage with our approach is its generality, which makes it possible to perform Bayesian analysis in an automatic, streamlined way, and to compute model comparison criteria and various predictive measures so that models can be compared and the model under study can be challenged.

/pdf/approximate-bayesian-inference-for-latent-gaussian-models-by-2m3x22eic2.pdf

Approximate Bayesian inference for latent Gaussian models by using integrated nested Laplace approximations

We consider prediction and uncertainty analysis for systems which are approximated using complex mathematical models. Such models, implemented as computer codes, are often generic in the sense that by a suitable choice of some of the model's input parameters the code can be used to predict the behaviour of the system in a variety of specific applications. However, in any specific application the values of necessary parameters may be unknown. In this case, physical observations of the system in the specific context are used to learn about the unknown parameters. The process of fitting the model to the observed data by adjusting the parameters is known as calibration. Calibration is typically effected by ad hoc fitting, and after calibration the model is used, with the fitted input values, to predict the future behaviour of the system. We present a Bayesian calibration technique which improves on this traditional approach in two respects. First, the predictions allow for all sources of uncertainty, including the remaining uncertainty over the fitted parameters. Second, they attempt to correct for any inadequacy of the model which is revealed by a discrepancy between the observed data and the model predictions from even the best-fitting parameter values. The method is illustrated by using data from a nuclear radiation release at Tomsk, and from a more complex simulated nuclear accident exercise.

https://www.jstor.org/stable/pdfplus/2680584.pdf

Bayesian Calibration of computer models

This paper presents a minimization method for Lipschitz continuous, piecewise smooth objective functions based on algorithmic differentiation (AD). We assume that all nondifferentiabilities are cau...

Algorithmic differentiation for piecewise smooth functions: a case study for robust optimization

We consider the inviscid Burgers’ equation augmented by a control term and its adjoint equation. For several discretization schemes of the inviscid Burgers’ equation the adjoint finite difference methods are derived. Applying these discretization methods and the checkpointing routine treeverse, approximations of the solution of both differential equations are calculated and compared.

Applying the Checkpointing Routine treeverse to Discretizations of Burgers’ Equation*

We present the adaptation and implementation of a composite-step trust region algorithm, developed in (Walther, SIAM J. Optim. 19(1):307---325, 2008), that incorporates the approximation of the Jacobian of the equality constraints with a specialized quasi-Newton method. The forming and/or factoring of the exact Jacobian in each optimization step is avoided. Hence, the presented approach is especially well suited for equality constrained optimization problems where the Jacobian of the constraints is dense.

In this study, we discuss the calculation of the normal and tangential steps and how the trust region radius is adapted to take the inaccurate first-order information into account. Results are presented for several examples from the CUTE test set and a simple periodic adsorption process.

Numerical experiments with an inexact Jacobian trust-region algorithm

For design optimization tasks, quite often a so-called one-shot approach is used. It augments the solution of the state equation with a suitable adjoint solver yielding approximate reduced derivatives that can be used in an optimization iteration to change the design. The coordination of these three iterative processes is well established when only the state equation is considered as equality constraint. However, numerous applications require also additional equality constraints. Therefore, we propose a modified augmented Lagrangian function, that defines a simultaneous change of all variables for this extended setting. It is shown that the augmented Lagrangian function proposed in this paper can be used in a gradient-based optimization approach to solve the original design task.

http://www.optimization-online.org/DB_FILE/2015/03/4802.pdf

On an extension of one-shot methods to incorporate additional constraints

The reversal of a program execution can be helpful for several purposes. The reverse mode of automatic differentiation, parameter estimation, and debugging represent some of the tasks that may require the intermediate values of a calculation in reverse order. The recording of a complete execution log forms the simplest approach to providing the data required for the reversal. However, this “store-everything” method causes an enormous demand for memory. The generation of the execution log piece by piece offers the possibility to reduce the storage requirement. To that end the forward calculation is restarted from suitably placed checkpoints leading to a more or less complicated reversal schedule.This paper deals with the reversal of evaluation procedures that consist of a sequence of time steps with varying computational complexity. We present a new search algorithm for determining a reversal schedule that minimizes the runtime of the reversal process for a given number of checkpoints. By exploiting a certain monotonicity property, the search algorithm based on dynamic programming can be made to grow only quadratically with the number of time steps to be reverted. We report the runtime savings that can be achieved performing the reversal with an optimal, i.e. time-minimal, reversal schedule for a test problem based on Burgers’ equation.

Andrea Walther

Papers

Algorithmic differentiation for piecewise smooth functions: a case study for robust optimization

Applying the Checkpointing Routine treeverse to Discretizations of Burgers’ Equation*

Numerical experiments with an inexact Jacobian trust-region algorithm

On an extension of one-shot methods to incorporate additional constraints

Program reversals for evolutions with non-uniform step costs