Deep learning and the information bottleneck principle
Naftali Tishby,Noga Zaslavsky +1 more
- pp 1-5
TLDR
It is argued that both the optimal architecture, number of layers and features/connections at each layer, are related to the bifurcation points of the information bottleneck tradeoff, namely, relevant compression of the input layer with respect to the output layer.Abstract:
Deep Neural Networks (DNNs) are analyzed via the theoretical framework of the information bottleneck (IB) principle. We first show that any DNN can be quantified by the mutual information between the layers and the input and output variables. Using this representation we can calculate the optimal information theoretic limits of the DNN and obtain finite sample generalization bounds. The advantage of getting closer to the theoretical limit is quantifiable both by the generalization bound and by the network's simplicity. We argue that both the optimal architecture, number of layers and features/connections at each layer, are related to the bifurcation points of the information bottleneck tradeoff, namely, relevant compression of the input layer with respect to the output layer. The hierarchical representations at the layered network naturally correspond to the structural phase transitions along the information curve. We believe that this new insight can lead to new optimality bounds and deep learning algorithms.read more
Citations
More filters
Posted Content
MINE: Mutual Information Neural Estimation.
TL;DR: In this article, the estimation of mutual information between high dimensional continuous random variables can be achieved by gradient descent over neural networks, which is linearly scalable in dimensionality as well as in sample size.
Book ChapterDOI
Efficient and Accurate MRI Super-Resolution Using a Generative Adversarial Network and 3D Multi-level Densely Connected Network
Yuhua Chen,Yuhua Chen,Feng Shi,Anthony G. Christodoulou,Yibin Xie,Zhengwei Zhou,Debiao Li,Debiao Li +7 more
TL;DR: A novel 3D neural network design, namely a multi-level densely connected super-resolution network (mDCSRN) with generative adversarial network (GAN)-guided training, which beats other popular deep learning methods in recovering 4x resolution-downgraded im-ages and runs 6x faster.
Journal ArticleDOI
Convolutional Neural Networks as a Model of the Visual System: Past, Present, and Future.
TL;DR: This review highlights what, in the context of CNNs, it means to be a good model in computational neuroscience and the various ways models can provide insight.
Posted Content
Deep Semi-Supervised Anomaly Detection
Lukas Ruff,Robert A. Vandermeulen,Nico Görnitz,Alexander Binder,Emmanuel Müller,Klaus-Robert Müller,Marius Kloft +6 more
TL;DR: This work presents Deep SAD, an end-to-end deep methodology for general semi-supervised anomaly detection, and introduces an information-theoretic framework for deep anomaly detection based on the idea that the entropy of the latent distribution for normal data should be lower than the entropy the anomalous distribution, which can serve as a theoretical interpretation for the method.
Posted Content
TensorFlow Quantum: A Software Framework for Quantum Machine Learning
Michael Broughton,Guillaume Verdon,Trevor McCourt,Antonio Martinez,Jae Hyeon Yoo,Sergei V. Isakov,Philip Massey,Murphy Yuezhen Niu,Ramin Halavati,Evan Peters,Martin Leib,Andrea Skolik,Michael Streif,David Von Dollen,Jarrod R. McClean,Sergio Boixo,Dave Bacon,Alan K. Ho,Hartmut Neven,Masoud Mohseni +19 more
TL;DR: This framework offers high-level abstractions for the design and training of both discriminative and generative quantum models under TensorFlow and supports high-performance quantum circuit simulators.
References
More filters
Proceedings Article
ImageNet Classification with Deep Convolutional Neural Networks
TL;DR: The state-of-the-art performance of CNNs was achieved by Deep Convolutional Neural Networks (DCNNs) as discussed by the authors, which consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax.
Book
Elements of information theory
Thomas M. Cover,Joy A. Thomas +1 more
TL;DR: The author examines the role of entropy, inequality, and randomness in the design of codes and the construction of codes in the rapidly changing environment.
Journal ArticleDOI
Reducing the Dimensionality of Data with Neural Networks
TL;DR: In this article, an effective way of initializing the weights that allows deep autoencoder networks to learn low-dimensional codes that work much better than principal components analysis as a tool to reduce the dimensionality of data is described.
Journal ArticleDOI
Representation Learning: A Review and New Perspectives
TL;DR: Recent work in the area of unsupervised feature learning and deep learning is reviewed, covering advances in probabilistic models, autoencoders, manifold learning, and deep networks.
Book
Learning Deep Architectures for AI
TL;DR: The motivations and principles regarding learning algorithms for deep architectures, in particular those exploiting as building blocks unsupervised learning of single-layer modelssuch as Restricted Boltzmann Machines, used to construct deeper models such as Deep Belief Networks are discussed.