scispace - formally typeset
Open AccessProceedings Article

Deep Neural Networks as Gaussian Processes

TLDR
The exact equivalence between infinitely wide deep networks and GPs is derived and it is found that test performance increases as finite-width trained networks are made wider and more similar to a GP, and thus that GP predictions typically outperform those of finite- width networks.
Abstract
It has long been known that a single-layer fully-connected neural network with an i.i.d. prior over its parameters is equivalent to a Gaussian process (GP), in the limit of infinite network width. This correspondence enables exact Bayesian inference for infinite width neural networks on regression tasks by means of evaluating the corresponding GP. Recently, kernel functions which mimic multi-layer random neural networks have been developed, but only outside of a Bayesian framework. As such, previous work has not identified that these kernels can be used as covariance functions for GPs and allow fully Bayesian prediction with a deep neural network. In this work, we derive the exact equivalence between infinitely wide deep networks and GPs. We further develop a computationally efficient pipeline to compute the covariance function for these GPs. We then use the resulting GPs to perform Bayesian inference for wide deep neural networks on MNIST and CIFAR-10. We observe that trained neural network accuracy approaches that of the corresponding GP with increasing layer width, and that the GP uncertainty is strongly correlated with trained network prediction error. We further find that test performance increases as finite-width trained networks are made wider and more similar to a GP, and thus that GP predictions typically outperform those of finite-width networks. Finally we connect the performance of these GPs to the recent theory of signal propagation in random neural networks.

read more

Content maybe subject to copyright    Report

Citations
More filters
Posted Content

The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initialization

TL;DR: In this paper, the authors study ReLU ResNets in the infinite-depth-and-width limit, where both depth and width tend to infinity as their ratio, $d/n), remains constant.
Posted Content

Neural Network Gaussian Processes by Increasing Depth

TL;DR: Wang et al. as discussed by the authors used a shortcut network to show that increasing the depth of a neural network can also give rise to a Gaussian process, which is a valuable addition to the existing theory and contributes to revealing the true picture of deep learning.
Posted Content

Gradient descent in Gaussian random fields as a toy model for high-dimensional optimisation in deep learning

TL;DR: The distribution of the improved loss function after a step of gradient descent is analyzed, analytic expressions for the moments are provided as well as asymptotic normality as the dimension of the parameter space becomes large are proved.
Posted Content

PAC-Bayesian Bounds for Deep Gaussian Processes

TL;DR: This paper introduces Probably Approximately Correct (PAC)-Bayesian risk bounds for DGPs making use of variational approximations and shows that the minimization of PAC-Bayesian generalization risk bounds maximizes the variational lower bounds belonging to the specific DGP model.
Posted Content

Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regime

TL;DR: In this article, the authors consider Kernel Ridge Regression (KRR) under the Gaussian design and provide characterization of all regimes and excess error decay rates that can be observed in terms of the interplay of noise and regularization.
References
More filters
Proceedings Article

Adam: A Method for Stochastic Optimization

TL;DR: This work introduces Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments, and provides a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework.
Book

Bayesian learning for neural networks

TL;DR: Bayesian Learning for Neural Networks shows that Bayesian methods allow complex neural network models to be used without fear of the "overfitting" that can occur with traditional neural network learning methods.
Journal ArticleDOI

A Unifying View of Sparse Approximate Gaussian Process Regression

TL;DR: A new unifying view, including all existing proper probabilistic sparse approximations for Gaussian process regression, relies on expressing the effective prior which the methods are using, and highlights the relationship between existing methods.
Journal Article

In Defense of One-Vs-All Classification

TL;DR: It is argued that a simple "one-vs-all" scheme is as accurate as any other approach, assuming that the underlying binary classifiers are well-tuned regularized classifiers such as support vector machines.
Proceedings Article

Gaussian processes for Big data

TL;DR: In this article, the authors introduce stochastic variational inference for Gaussian process models, which enables the application of Gaussian Process (GP) models to data sets containing millions of data points.