In this article, we describe an automatic differentiation module of PyTorch — a library designed to enable rapid research on machine learning models. It builds upon a few projects, most notably Lua Torch, Chainer, and HIPS Autograd [4], and provides a high performance environment with easy access to automatic differentiation of models executed on different devices (CPU and GPU). To make prototyping easier, PyTorch does not follow the symbolic approach used in many other deep learning frameworks, but focuses on differentiation of purely imperative programs, with a focus on extensibility and low overhead. Note that this preprint is a draft of certain sections from an upcoming paper covering all PyTorch features.

/pdf/automatic-differentiation-in-pytorch-gbuafkf7qo.pdf

Automatic differentiation in PyTorch

Deposits of clastic carbonate-dominated (calciclastic) sedimentary slope systems in the rock record have been identified mostly as linearly-consistent carbonate apron deposits, even though most ancient clastic carbonate slope deposits fit the submarine fan systems better. Calciclastic submarine fans are consequently rarely described and are poorly understood. Subsequently, very little is known especially in mud-dominated calciclastic submarine fan systems. Presented in this study are a sedimentological core and petrographic characterisation of samples from eleven boreholes from the Lower Carboniferous of Bowland Basin (Northwest England) that reveals a >250 m thick calciturbidite complex deposited in a calciclastic submarine fan setting. Seven facies are recognised from core and thin section characterisation and are grouped into three carbonate turbidite sequences. They include: 1) Calciturbidites, comprising mostly of highto low-density, wavy-laminated bioclast-rich facies; 2) low-density densite mudstones which are characterised by planar laminated and unlaminated muddominated facies; and 3) Calcidebrites which are muddy or hyper-concentrated debrisflow deposits occurring as poorly-sorted, chaotic, mud-supported floatstones. These

Phd by thesis

The brms package implements Bayesian multilevel models in R using the probabilistic programming language Stan. A wide range of distributions and link functions are supported, allowing users to fit - among others - linear, robust linear, binomial, Poisson, survival, ordinal, zero-inflated, hurdle, and even non-linear models all in a multilevel context. Further modeling options include autocorrelation of the response variable, user defined covariance structures, censored data, as well as meta-analytic standard errors. Prior specifications are flexible and explicitly encourage users to apply prior distributions that actually reflect their beliefs. In addition, model fit can easily be assessed and compared with the Watanabe-Akaike information criterion and leave-one-out cross-validation.

brms: An R Package for Bayesian Multilevel Models Using Stan

Structured additive regression models are perhaps the most commonly used class of models in statistical applications. It includes, among others, (generalized) linear models, (generalized) additive models, smoothing spline models, state space models, semiparametric regression, spatial and spatiotemporal models, log-Gaussian Cox processes and geostatistical and geoadditive models. We consider approximate Bayesian inference in a popular subset of structured additive regression models, latent Gaussian models, where the latent field is Gaussian, controlled by a few hyperparameters and with non-Gaussian response variables. The posterior marginals are not available in closed form owing to the non-Gaussian response variables. For such models, Markov chain Monte Carlo methods can be implemented, but they are not without problems, in terms of both convergence and computational time. In some practical applications, the extent of these problems is such that Markov chain Monte Carlo sampling is simply not an appropriate tool for routine analysis. We show that, by using an integrated nested Laplace approximation and its simplified version, we can directly compute very accurate approximations to the posterior marginals. The main benefit of these approximations is computational: where Markov chain Monte Carlo algorithms need hours or days to run, our approximations provide more precise estimates in seconds or minutes. Another advantage with our approach is its generality, which makes it possible to perform Bayesian analysis in an automatic, streamlined way, and to compute model comparison criteria and various predictive measures so that models can be compared and the model under study can be challenged.

/pdf/approximate-bayesian-inference-for-latent-gaussian-models-by-2m3x22eic2.pdf

Approximate Bayesian inference for latent Gaussian models by using integrated nested Laplace approximations

We consider prediction and uncertainty analysis for systems which are approximated using complex mathematical models. Such models, implemented as computer codes, are often generic in the sense that by a suitable choice of some of the model's input parameters the code can be used to predict the behaviour of the system in a variety of specific applications. However, in any specific application the values of necessary parameters may be unknown. In this case, physical observations of the system in the specific context are used to learn about the unknown parameters. The process of fitting the model to the observed data by adjusting the parameters is known as calibration. Calibration is typically effected by ad hoc fitting, and after calibration the model is used, with the fitted input values, to predict the future behaviour of the system. We present a Bayesian calibration technique which improves on this traditional approach in two respects. First, the predictions allow for all sources of uncertainty, including the remaining uncertainty over the fitted parameters. Second, they attempt to correct for any inadequacy of the model which is revealed by a discrepancy between the observed data and the model predictions from even the best-fitting parameter values. The method is illustrated by using data from a nuclear radiation release at Tomsk, and from a more complex simulated nuclear accident exercise.

https://www.jstor.org/stable/pdfplus/2680584.pdf

Bayesian Calibration of computer models

Advanced process optimization technologies play a critical role in improving economics of carbon capture technologies. Pressure swing adsorption (PSA) has received recent attention as a potential process for economically removing CO 2 from flue and/or shifted syngas for carbon capture and storage. In this work, we present state-of-the-art methods for PSA optimization using a superstructure-based approach; this allows simultaneous selection of PSA cycle steps and optimization of operating parameters (feed flow rate, recycle fractions, bed pressures, etc.). The partial differential equation bed models are discretized with finite volumes in space and two flux limiters are compared. Sparse linear solvers are implemented to accelerate the integration of bed models and direct sensitivity equations, which are interfaced to MATLAB. The PSA optimization approach is demonstrated on a CO 2 /H 2 separation case study for an integrated gasification combine cycle power plant, and solved by sequential quadratic programming solvers.

Pressure Swing Adsorption Optimization Strategies for CO2 Capture

Systems of stiff ordinary differential equations (ODEs) can be integrated properly only by implicit methods. For that purpose, one usually has to solve a system of nonlinear equations at each time step. This system of equations may be solved by variants of Newton’s method. The main computing effort lies in forming and factoring the Jacobian or a suitable approximation to it. We examine a new approach of constructing an appropriate quasi-Newton approximation for solving stiff ODEs. The method makes explicit use of tangent and adjoint information that can be obtained using the forward and the reverse modes of algorithmic differentiation (AD). We elaborate the conditions for invariance with respect to linear transformations of the state space and thus similarity transformations of the Jacobian. We present one new updating variant that yields such an invariant method. Numerical results for Runge-Kutta methods and linear multi-step methods are discussed.

Application of AD-based Quasi-Newton Methods to Stiff ODEs

For adjoint calculations, parameter estimation, and similar purposes, one may need to produce all qualities calculated during the execution of a computer program in reverse order. The simplest possible approach is to record a complete execution log and then to read it backwards. This may require massive amounts of storage. Instead one may generate the execution log piecewise by restarting the "forward" calculation repeatedly from suitably placed checkpoints. For such-program execution reversals we present parallel reversal schedules that are probably optimal with regards to the number of concurrent processes and the total amount of memory required.

New results on program reversals

Derivative computation using Automatic Differentiation (AD) is often considered to operate purely serial. Performing the differentiation task in parallel may require the applied AD-tool to extract parallelization information from the user function, transform it, and apply this new strategy in the differentiation process. Furthermore, when using the reverse mode of AD, it must be ensured that no data races are introduced due to the reversed data access scheme. Considering an operator overloading based AD-tool, an additional challenge is to be met: Parallelization statements are typically not recognized. In this paper, we present and discuss the parallelization approach that we have integrated into ADOL-C, an operator overloading based AD-tool for the differentiation of C/C++ programs. The advantages of the approach are clarified by means of the parallel differentiation of a function that handles the time evolution of a 1D-quantum plasma.

Parallel Derivative Computation using ADOL-C.

A class of trust-region algorithms is developed and analyzed for the solution of minimization problems with nonlinear inequality constraints. Based on composite-step trust-region methods with barrier functions, the resulting algorithm also does not require the computation of exact Jacobians; only Jacobian vector products are used along with approximate Jacobian matrices. Therefore, the proposed method is targeted on small or medium size problems with dense Jacobians of the constraints. As demonstrated on small numerical examples, this feature has significant potential benefits for problems where Jacobian calculations are expensive.

Andrea Walther

Papers

Pressure Swing Adsorption Optimization Strategies for CO2 Capture

Application of AD-based Quasi-Newton Methods to Stiff ODEs

New results on program reversals

Parallel Derivative Computation using ADOL-C.

A first-order convergence analysis of trust-region methods with inexact Jacobians and inequality constraints