Automatic, or algorithmic, differentiation (AD) is a chain rule-based technique for evaluating derivatives of functions given as computer programs for their elimination. We review the main characteristics and application of AD and illustrate the methodology on a simple example.

https://onlinelibrary.wiley.com/doi/pdf/10.1002/pamm.200310012

Introduction to Automatic Differentiation

This article presents a low-storage and low-run-time approach for calculating numerical approximations of adjoint equations for the instationary Navier–Stokes equations with adaptive evaluation of the discretization step. It utilizes adaptive checkpointing. The amounts of memory reduction due to checkpointing and of the increase in run-time caused by repeated forward integration of the Navier–Stokes equations are reported. It is one result that memory reduction of two orders of magnitude only causes a slow down factor of 2–3 in run-time. It is a further result that the adaptive checkpointing for the instationary Navier–Stokes equations causes only a slight increase of the run-time compared to the static optimal checkpointing. The amounts of run-time increase caused by the adaptive checkpoint allocations compared with the static optimal checkpointing are reported as well.

A-revolve: an adaptive memory-reduced procedure for calculating adjoints; with an application to computing adjoints of the instationary Navier–Stokes system

We discuss software tool support for the algorithmic differentiation (AD), also known as automatic differentiation, of numerical simulation programs that contain calls to solvers for parameterized systems of n nonlinear equations. The local computational overhead and the additional memory requirement for the computation of directional derivatives or adjoints of the solution of the nonlinear system with respect to the parameters can quickly become prohibitive for large values of n. Both are reduced drastically by analytical (and symbolic) approaches to differentiation of the underlying numerical methods. Following the discussion of the proposed terminology, we develop the algorithmic formalism building on prior work by other colleagues and present an implementation based on the AD software dco/c++. A representative case study supports the theoretically obtained computational complexity results with practical runtime measurements.

Algorithmic Differentiation of Numerical Methods: Tangent and Adjoint Solvers for Parameterized Systems of Nonlinear Equations

We demonstrate the flexibility and ease of use of C++ algorithmic differentiation (AD) tools based on overloading to numerical patterns (kernels) arising in computational finance. While adjoint methods and AD have been known in the finance literature for some time, there are few tools capable of handling and integrating with the C++ codes found in production. Adjoint methods are also known to be very powerful but to potentially have infeasible memory requirements. We present several techniques for dealing with this problem and demonstrate them on numerical kernels which occur frequently in finance. We build the discussion around our own AD tool dco/c++ which is designed to handle arbitrary C++ codes and to be highly flexible, however the sketched concepts can certainly be transferred to other AD solutions including in-house tools. An archive of the source code for the numerical kernels as well as all the AD solutions discussed can be downloaded from an accompanying website. This includes documentation for the code and dco/c++. Trial licences for dco/c++ are available from NAG.

Adjoint algorithmic differentiation tool support for typical numerical patterns in computational finance

For computational purposes such as the computation of adjoint, applying the reverse mode of automatic differentiation, or debugging one may require the values computed during the evaluation of a function in reverse order. The naive approach is to store all information needed for the reversal and to read this information backwards during the reversal. This technique leads to an enormous memory requirement, which is proportional to the computing time. The paper presents an approach to reducing the memory requirement without increasing the wall clock time by using parallel computers. During the parallel computation, only a fixed and small number of memory pads called checkpoints is stored. The data needed for the reversal is recomputed piecewise by starting the evaluation procedure from the checkpoints. We explain how this technique can be used on a parallel computer with distributed memory. Different implementation strategies will be shown, and some details with respect to resource-optimality are discussed.

/pdf/the-implementation-and-testing-of-time-minimal-and-resource-4wqk1clr01.pdf

The Implementation and Testing of Time-Minimal and Resource-Optimal Parallel Reversal Schedules

For computational purposes such as debugging, derivative computations using the reverse mode of automatic differentiation, or optimal control by Newton’s method, one may need to reverse the execution of a program The simplest option is to record a complete execution log and then to read it backwards As a result, massive amounts of storage are normally required This paper proposes a new approach to reversing program executions The presented technique runs the forward simulation and the reversal process at the same speed For that purpose, one only employs a fixed and usually small amount of memory pads called checkpoints to store intermediate states and a certain number of processors The execution log is generated piecewise by restarting the evaluation repeatedly and concurrently from suitably placed checkpoints The paper illustrates the principle structure of time-minimal parallel reversal schedules and quotes the required resources Furthermore, some specific aspects of adjoint calculations are discussed Initial results for the steering of a Formula 1 car are shown

/pdf/adjoint-calculation-using-time-minimal-program-reversals-for-58wbu3ytyt.pdf

Uwe Lehmann

Papers

The Implementation and Testing of Time-Minimal and Resource-Optimal Parallel Reversal Schedules

Adjoint calculation using time minimal program reversals for multi-processor machines

Uwe Lehmann

Papers

The Implementation and Testing of Time-Minimal and Resource-Optimal Parallel Reversal Schedules

Adjoint calculation using time­ minimal program reversals for multi-processor machines

Adjoint calculation using time minimal program reversals for multi-processor machines