Open AccessProceedings Article
Deep Neural Networks as Gaussian Processes
Jaehoon Lee,Yasaman Bahri,Roman Novak,Samuel S. Schoenholz,Jeffrey Pennington,Jascha Sohl-Dickstein +5 more
TLDR
The exact equivalence between infinitely wide deep networks and GPs is derived and it is found that test performance increases as finite-width trained networks are made wider and more similar to a GP, and thus that GP predictions typically outperform those of finite- width networks.Abstract:
It has long been known that a single-layer fully-connected neural network with an i.i.d. prior over its parameters is equivalent to a Gaussian process (GP), in the limit of infinite network width. This correspondence enables exact Bayesian inference for infinite width neural networks on regression tasks by means of evaluating the corresponding GP. Recently, kernel functions which mimic multi-layer random neural networks have been developed, but only outside of a Bayesian framework. As such, previous work has not identified that these kernels can be used as covariance functions for GPs and allow fully Bayesian prediction with a deep neural network.
In this work, we derive the exact equivalence between infinitely wide deep networks and GPs. We further develop a computationally efficient pipeline to compute the covariance function for these GPs. We then use the resulting GPs to perform Bayesian inference for wide deep neural networks on MNIST and CIFAR-10. We observe that trained neural network accuracy approaches that of the corresponding GP with increasing layer width, and that the GP uncertainty is strongly correlated with trained network prediction error. We further find that test performance increases as finite-width trained networks are made wider and more similar to a GP, and thus that GP predictions typically outperform those of finite-width networks. Finally we connect the performance of these GPs to the recent theory of signal propagation in random neural networks.read more
Citations
More filters
Proceedings ArticleDOI
Wearing A Mask: Compressed Representations of Variable-Length Sequences Using Recurrent Neural Tangent Kernels
Sina Alemohammad,Hossein Babaei,Randall Balestriero,Matt Y. Cheung,Ahmed Imtiaz Humayun,Daniel LeJeune,Naiming Liu,Lorenzo Luzi,Jasper Tan,Zichao Wang,Richard G. Baraniuk +10 more
TL;DR: In this paper, the authors extend existing methods that rely on the use of kernels to variable-length sequences via use of the Recurrent Neural Tangent Kernel (RNTK) to handle high dimensionality.
Journal ArticleDOI
Gaussian Process States: A data-driven representation of quantum many-body physics
TL;DR: A novel, non-parametric form for compactly representing entangled many-body quantum states, which is called a `Gaussian Process State', which is found to be highly compact, systematically improvable and efficient to sample, representing a large number of known variational states within its span.
Posted Content
InClass Nets: Independent Classifier Networks for Nonparametric Estimation of Conditional Independence Mixture Models and Unsupervised Classification
TL;DR: This work introduces a novel unsupervised machine learning technique called the independent classifier networks (InClass nets) technique for the nonparameteric estimation of CIMMs, which consists of multiple independent classifiers neural networks, which are trained simultaneously using suitable cost functions.
Posted Content
Perspective: A Phase Diagram for Deep Learning unifying Jamming, Feature Learning and Lazy Training.
TL;DR: In this article, the authors present a theoretical discussion on the dimensionality paradox in deep learning and show that deep learning predicting power increases with the number of fitting parameters, even in a regime where data are perfectly fitted.
Posted Content
Disentangling the Gauss-Newton Method and Approximate Inference for Neural Networks.
TL;DR: A new disentangled understanding of recent Bayesian deep learning algorithms leads to new methods: first, the connection to Gaussian processes enables new function-space inference algorithms, and second, a marginal likelihood approximation of the underlying probabilistic model to tune neural network hyperparameters is presented.
References
More filters
Proceedings Article
Adam: A Method for Stochastic Optimization
Diederik P. Kingma,Jimmy Ba +1 more
TL;DR: This work introduces Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments, and provides a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework.
Book
Bayesian learning for neural networks
TL;DR: Bayesian Learning for Neural Networks shows that Bayesian methods allow complex neural network models to be used without fear of the "overfitting" that can occur with traditional neural network learning methods.
Journal ArticleDOI
A Unifying View of Sparse Approximate Gaussian Process Regression
TL;DR: A new unifying view, including all existing proper probabilistic sparse approximations for Gaussian process regression, relies on expressing the effective prior which the methods are using, and highlights the relationship between existing methods.
Journal Article
In Defense of One-Vs-All Classification
Ryan Rifkin,Aldebaro Klautau +1 more
TL;DR: It is argued that a simple "one-vs-all" scheme is as accurate as any other approach, assuming that the underlying binary classifiers are well-tuned regularized classifiers such as support vector machines.
Proceedings Article
Gaussian processes for Big data
TL;DR: In this article, the authors introduce stochastic variational inference for Gaussian process models, which enables the application of Gaussian Process (GP) models to data sets containing millions of data points.