Deep learning and the information bottleneck principle
Naftali Tishby,Noga Zaslavsky +1 more
- pp 1-5
TLDR
It is argued that both the optimal architecture, number of layers and features/connections at each layer, are related to the bifurcation points of the information bottleneck tradeoff, namely, relevant compression of the input layer with respect to the output layer.Abstract:
Deep Neural Networks (DNNs) are analyzed via the theoretical framework of the information bottleneck (IB) principle. We first show that any DNN can be quantified by the mutual information between the layers and the input and output variables. Using this representation we can calculate the optimal information theoretic limits of the DNN and obtain finite sample generalization bounds. The advantage of getting closer to the theoretical limit is quantifiable both by the generalization bound and by the network's simplicity. We argue that both the optimal architecture, number of layers and features/connections at each layer, are related to the bifurcation points of the information bottleneck tradeoff, namely, relevant compression of the input layer with respect to the output layer. The hierarchical representations at the layered network naturally correspond to the structural phase transitions along the information curve. We believe that this new insight can lead to new optimality bounds and deep learning algorithms.read more
Citations
More filters
Journal ArticleDOI
Interpreting Convolutional Neural Networks for Device-Free Wi-Fi Fingerprinting Indoor Localization via Information Visualization
TL;DR: This paper quantifies and visualizes CNN in comparison with the fully-connected feedforward deep neural network (DNN) (or multilayer perceptron), and observes that each model can automatically identify location-specific patterns, which are however different across models and are linked to the respective performance of each model.
Posted Content
Disentangled Information Bottleneck
TL;DR: Disentangled Information Bottleneck (DisenIB) is introduced that is consistent on compressing source maximally without target prediction performance loss (maximum compression), and performs well in terms of generalization, robustness to adversarial attack, out-of-distribution detection, and supervised disentangling.
Proceedings Article
Dynamics-Aware Unsupervised Skill Discovery
TL;DR: This work proposes an unsupervised learning algorithm, Dynamics-Aware Discovery of Skills (DADS), which simultaneously discovers predictable behaviors and learns their dynamics, and demonstrates that zero-shot planning in the learned latent space significantly outperforms standard MBRL and model-free goal-conditioned RL, and substantially improves over prior hierarchical RL methods for unsuper supervised skill discovery.
Journal ArticleDOI
Deep radiomic signature with immune cell markers predicts the survival of glioma patients
Ahmad Chaddad,Ahmad Chaddad,Paul Daniel,Mingli Zhang,Saima Rathore,Paul Sargos,Christian Desrosiers,Tamim Niazi +7 more
TL;DR: The usefulness of proposed DRFs as non-invasive biomarker for predicting treatment response in patients with brain tumors is demonstrated, with a high correlation between DRFs and various markers, as well as significant differences between patients grouped based on these markers.
Journal ArticleDOI
Layer-wise Learning of Stochastic Neural Networks with Information Bottleneck
Thanh Tang Nguyen,Jaesik Choi +1 more
TL;DR: It is shown that the PIB framework can be considered as an extension of the maximum likelihood estimate (MLE) principle to every layer level and is more efficient to exploit a neural network's representation by pushing it closer to the optimal information-theoretical representation in a faster manner.
References
More filters
Proceedings Article
ImageNet Classification with Deep Convolutional Neural Networks
TL;DR: The state-of-the-art performance of CNNs was achieved by Deep Convolutional Neural Networks (DCNNs) as discussed by the authors, which consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax.
Book
Elements of information theory
Thomas M. Cover,Joy A. Thomas +1 more
TL;DR: The author examines the role of entropy, inequality, and randomness in the design of codes and the construction of codes in the rapidly changing environment.
Journal ArticleDOI
Reducing the Dimensionality of Data with Neural Networks
TL;DR: In this article, an effective way of initializing the weights that allows deep autoencoder networks to learn low-dimensional codes that work much better than principal components analysis as a tool to reduce the dimensionality of data is described.
Journal ArticleDOI
Representation Learning: A Review and New Perspectives
TL;DR: Recent work in the area of unsupervised feature learning and deep learning is reviewed, covering advances in probabilistic models, autoencoders, manifold learning, and deep networks.
Book
Learning Deep Architectures for AI
TL;DR: The motivations and principles regarding learning algorithms for deep architectures, in particular those exploiting as building blocks unsupervised learning of single-layer modelssuch as Restricted Boltzmann Machines, used to construct deeper models such as Deep Belief Networks are discussed.