Open AccessPosted Content
CHAD: Combinatory Homomorphic Automatic Differentiation.
TLDR
The Combinatory Homomorphic Automatic Differentiation (CHAD) as mentioned in this paper is a principled, pure, provably correct method for performing forward and reverse-mode automatic differentiation (AD) on programming languages with expressive features.Abstract:
We introduce Combinatory Homomorphic Automatic Differentiation (CHAD), a principled, pure, provably correct method for performing forward- and reverse-mode automatic differentiation (AD) on programming languages with expressive features. It implements AD as a compositional, type-respecting source-code transformation that generates purely functional code. This code transformation is principled in the sense that it is the unique homomorphic (structure preserving) extension to expressive languages of the well-known and unambiguous definitions of automatic differentiation for a first-order functional language. Correctness of the method follows by a (compositional) logical relations argument that shows that the semantics of the syntactic derivative is the usual calculus derivative of the semantics of the original program. In their most elegant formulation, the transformations generate code with linear types. However, the transformations can be implemented in a standard functional language without sacrificing correctness. This implementation can be achieved by making use of abstract data types to represent the required linear types, e.g. through the use of a basic module system.
In this paper, we detail the method when applied to a simple higher-order language for manipulating statically sized arrays. However, we explain how the methodology applies, more generally, to functional languages with other expressive features.
Finally, we discuss how the scope of CHAD extends beyond applications in automatic differentiation to other dynamic program analyses that accumulate data in a commutative monoid.read more
Citations
More filters
Journal ArticleDOI
Efficient Dual-Numbers Reverse AD via Well-Known Program Transformations
T. Smeding,Matthijs V'ak'ar +1 more
TL;DR: This work takes inspiration from the use of linear factoring to optimise dual-numbers reverse-mode AD to an algorithm that has the correct complexity and enjoys an efficient implementation in a standard functional language with support for mutable arrays, such as Haskell.
Posted Content
Higher Order Automatic Differentiation of Higher Order Functions.
TL;DR: In this article, a forward-mode AD method on a higher-order language with algebraic data types is considered, and a rich semantics for differentiable programming, based on diffeological spaces, is described.
Journal ArticleDOI
Higher Order Automatic Differentiation of Higher Order Functions
TL;DR: In this article , a forward-mode AD method on a higher-order language with algebraic data types is considered, and a rich semantics for differentiable programming, based on diffeological spaces, is described.
Posted Content
CHAD for Expressive Total Languages
TL;DR: In this paper, the forward and reverse mode Combinatory Homomorphic Automatic Differentiation (CHAD) is applied to total functional programming languages with expressive type systems featuring the combination of tuple types; sum types; inductive types; coinductive types; and function types.
Journal ArticleDOI
Monoidal reverse differential categories
TL;DR: In this article , a monoidal reverse differential category (MRSD) is defined, which is a related structure to CRDCs and can be used to model the reverse differentiation operations in supervised learning.
References
More filters
Automatic differentiation in PyTorch
Adam Paszke,Sam Gross,Soumith Chintala,Gregory Chanan,Edward Z. Yang,Zachary DeVito,Zeming Lin,Alban Desmaison,Luca Antiga,Adam Lerer +9 more
TL;DR: An automatic differentiation module of PyTorch is described — a library designed to enable rapid research on machine learning models that focuses on differentiation of purely imperative programs, with a focus on extensibility and low overhead.
Proceedings ArticleDOI
TensorFlow: a system for large-scale machine learning
Martín Abadi,Paul Barham,Jianmin Chen,Zhifeng Chen,Andy Davis,Jeffrey Dean,Matthieu Devin,Sanjay Ghemawat,Geoffrey Irving,Michael Isard,Manjunath Kudlur,Josh Levenberg,Rajat Monga,Sherry Moore,Derek G. Murray,Benoit Steiner,Paul A. Tucker,Vijay K. Vasudevan,Pete Warden,Martin Wicke,Yuan Yu,Xiaoqiang Zheng +21 more
TL;DR: TensorFlow as mentioned in this paper is a machine learning system that operates at large scale and in heterogeneous environments, using dataflow graphs to represent computation, shared state, and the operations that mutate that state.
Book
Introduction to higher order categorical logic
Joachim Lambek,Philip J. Scott +1 more
TL;DR: In this article, Cartesian closed categories and Calculus are used to represent Numerical functions in various categories and to describe the relation between categories. But they do not specify the topology of the categories.
Book
The Convenient Setting of Global Analysis
Andreas Kriegl,Peter W. Michor +1 more
TL;DR: Calculus of smooth mappings Calculus of holomorphic and real analytic mappings Partitions of unity Smoothly real compact spaces Extensions and liftings of mappings Infinite dimensional manifolds Calculus on infinite dimensional manifold, infinite dimensional differential geometry Manifolds of Mappings Further applications References as mentioned in this paper.
Book
Sketches of an Elephant: A Topos Theory Compendium
TL;DR: A1 Regular and Cartesian Closed Categories A2 Toposes - Basic Theory A3 Allegories A4 Geometric Morphisms - Basic theory B1 Fibrations and Indexed Categories B2 Internal and Locally Internal Categories B3 Toposes over a base B4 BTop/S as a 2-category as mentioned in this paper.