Normalizing Flows are generative models which produce tractable distributions where both sampling and density evaluation can be efficient and exact. The goal of this survey article is to give a coherent and comprehensive review of the literature around the construction and use of Normalizing Flows for distribution learning. We aim to provide context and explanation of the models, review current state-of-the-art literature, and identify open questions and promising future directions.

Normalizing Flows: An Introduction and Review of Current Methods

We derive generalization bounds for learning algorithms based on their robustness: the property that if a testing sample is "similar" to a training sample, then the testing error is close to the training error. This provides a novel approach, different from complexity or stability arguments, to study generalization of learning algorithms. One advantage of the robustness approach, compared to previous methods, is the geometric intuition it conveys. Consequently, robustness-based analysis is easy to extend to learning in non-standard setups such as Markovian samples or quantile loss. We further show that a weak notion of robustness is both sufficient and necessary for generalizability, which implies that robustness is a fundamental property that is required for learning algorithms to work.

Robustness and Generalization.

Plug-and-play (PnP) is a non-convex framework that integrates modern denoising priors, such as BM3D or deep learning-based denoisers, into ADMM or other proximal algorithms. An advantage of PnP is that one can use pre-trained denoisers when there is not sufficient data for end-to-end training. Although PnP has been recently studied extensively with great empirical success, theoretical analysis addressing even the most basic question of convergence has been insufficient. In this paper, we theoretically establish convergence of PnP-FBS and PnP-ADMM, without using diminishing stepsizes, under a certain Lipschitz condition on the denoisers. We then propose real spectral normalization, a technique for training deep learning-based denoisers to satisfy the proposed Lipschitz condition. Finally, we present experimental results validating the theory.

/pdf/plug-and-play-methods-provably-converge-with-properly-303lwz51yb.pdf

Plug-and-Play Methods Provably Converge with Properly Trained Denoisers

A First Course in the Numerical Analysis of Differential Equations: Gaussian elimination for sparse linear equations

Physics informed neural networks (PINNs) are deep learning based techniques for solving partial differential equations (PDEs) encounted in computational science and engineering. Guided by data and physical laws, PINNs find a neural network that approximates the solution to a system of PDEs. Such a neural network is obtained by minimizing a loss function in which any prior knowledge of PDEs and data are encoded. Despite its remarkable empirical success in one, two or three dimensional problems, there is little theoretical justification for PINNs. \nAs the number of data grows, PINNs generate a sequence of minimizers which correspond to a sequence of neural networks. We want to answer the question: Does the sequence of minimizers converge to the solution to the PDE? We consider two classes of PDEs: linear second-order elliptic and parabolic. By adapting the Schauder approach and the maximum principle, we show that the sequence of minimizers strongly converges to the PDE solution in $C^0$. Furthermore, we show that if each minimizer satisfies the initial/boundary conditions, the convergence mode becomes $H^1$. Computational examples are provided to illustrate our theoretical findings. To the best of our knowledge, this is the first theoretical work that shows the consistency of PINNs.

On the convergence of physics informed neural networks for linear second-order elliptic and parabolic type PDEs

Training neural ODEs on large datasets has not been tractable due to the necessity of allowing the adaptive numerical ODE solver to refine its step size to very small values. In practice this leads to dynamics equivalent to many hundreds or even thousands of layers. In this paper, we overcome this apparent difficulty by introducing a theoretically-grounded combination of both optimal transport and stability regularizations which encourage neural ODEs to prefer simpler dynamics out of all the dynamics that solve a problem well. Simpler dynamics lead to faster convergence and to fewer discretizations of the solver, considerably decreasing wall-clock time without loss in performance. Our approach allows us to train neural ODE-based generative models to the same performance as the unregularized dynamics, with significant reductions in training time. This brings neural ODEs closer to practical relevance in large-scale applications.

How to train your neural ODE: the world of Jacobian and kinetic regularization

/pdf/how-to-train-your-neural-ode-the-world-of-jacobian-and-54leux6rbr.pdf

How to Train Your Neural ODE: the World of Jacobian and Kinetic Regularization

In this work we study input gradient regularization of deep neural networks, and demonstrate that such regularization leads to generalization proofs and improved adversarial robustness. The proof of generalization does not overcome the curse of dimensionality, but it is independent of the number of layers in the networks. The adversarial robustness regularization combines adversarial training, which we show to be equivalent to Total Variation regularization, with Lipschitz regularization. We demonstrate empirically that the regularized models are more robust, and that gradient norms of images can be used for attack detection.

Lipschitz regularized Deep Neural Networks generalize and are adversarially robust

Mathematical models for the tumour control probability (TCP) are used to estimate the expected success of radiation treatment protocols of cancer. There are several TCP models in the literature, from the simplest (Poissonian TCP) to the well-advanced stochastic birth-death processes. Simple and complex models often make the same predictions. Hence, here, we present a systematic study where we compare six of these TCP models: the Poisson TCP, the Zaider-Minerbo TCP, a Monte Carlo TCP and their corresponding cell cycle (two-compartment) models. Several clinical non-uniform treatment protocols for prostate cancer are employed to evaluate these models. These include fractionated external beam radiotherapies, and high and low dose rate brachytherapies. We find that in realistic treatment scenarios, all one-compartment models and all two-compartment models give basically the same results. A difference occurs between one-compartment and two-compartment models due to reduced radiosensitivity of quiescent cells.We find that care must be taken for the right choice of parameters, such as the radiosensitivities α and β and the hazard function h. Typically, different hazard functions are used for fractionated treatment (fractionated survival fraction) and for brachytherapies (Lea-Catcheside protraction factor). We were able to combine these two approaches into one 'effective' hazard function. Based on our results, we can recommend the use of the Poissonian TCP for everyday treatment planning. More complicated models should only be used when absolutely necessary.

/pdf/are-more-complicated-tumour-control-probability-models-2iw32yqau1.pdf

Chris Finlay

Papers

How to train your neural ODE: the world of Jacobian and kinetic regularization

How to Train Your Neural ODE: the World of Jacobian and Kinetic Regularization

Lipschitz regularized Deep Neural Networks generalize and are adversarially robust

Are more complicated tumour control probability models better

How to train your neural ODE