Deep learning and the information bottleneck principle
Naftali Tishby,Noga Zaslavsky +1 more
- pp 1-5
TLDR
It is argued that both the optimal architecture, number of layers and features/connections at each layer, are related to the bifurcation points of the information bottleneck tradeoff, namely, relevant compression of the input layer with respect to the output layer.Abstract:
Deep Neural Networks (DNNs) are analyzed via the theoretical framework of the information bottleneck (IB) principle. We first show that any DNN can be quantified by the mutual information between the layers and the input and output variables. Using this representation we can calculate the optimal information theoretic limits of the DNN and obtain finite sample generalization bounds. The advantage of getting closer to the theoretical limit is quantifiable both by the generalization bound and by the network's simplicity. We argue that both the optimal architecture, number of layers and features/connections at each layer, are related to the bifurcation points of the information bottleneck tradeoff, namely, relevant compression of the input layer with respect to the output layer. The hierarchical representations at the layered network naturally correspond to the structural phase transitions along the information curve. We believe that this new insight can lead to new optimality bounds and deep learning algorithms.read more
Citations
More filters
Journal ArticleDOI
A Survey of Machine Unlearning
Thanh Tam Nguyen,Thanh Trung Huynh,Phi-Le Nguyen,Alan Wee-Chung Liew,Hongzhi Yin,Quoc Viet Hung Nguyen +5 more
TL;DR: This paper aspires to present a comprehensive examination of machine unlearning’s concepts, scenarios, methods, and applications as a category collection of cutting-edge studies to serve as a comprehensive resource for researchers and practitioners seeking an introduction to machine un learning.
Posted Content
Learning when to stop: a mutual information approach to fight overfitting in profiled side-channel analysis.
TL;DR: The amount of information, or, more precisely, mutual information transferred to the output layer, can be measured and used as a reference metric to determine the epoch at which the network offers optimal generalization in deep learning-based side-channel analysis.
Journal ArticleDOI
Mutual Information Driven Federated Learning
TL;DR: This article proposes a novel FL approach with resorting to mutual information (MI), in client-side, the weight update is reformulated through minimizing the MI between local and aggregated models and employing Negative Correlation Learning (NCL) strategy.
Journal ArticleDOI
Discriminative Feature Learning for Thorax Disease Classification in Chest X-ray Images
TL;DR: Wang et al. as mentioned in this paper proposed to learn discriminative features with a two-branch architecture, named ConsultNet, to achieve those two purposes simultaneously, which can automatically focus on the disease-critical regions, which usually are of small sizes; adaptively capture the intrinsic relationships among different disease features and utilize them to boost the multi-label disease recognition rates jointly.
Posted Content
Is SGD a Bayesian sampler? Well, almost
TL;DR: Estimating the probability that an overparameterised DNN, trained with stochastic gradient descent or one of its variants, converges on a function consistent with a training set, implies that strong inductive bias in the parameter-function map, rather than a special property of SGD, is the primary explanation for why DNNs generalise so well in the overparametersised regime.
References
More filters
Proceedings Article
ImageNet Classification with Deep Convolutional Neural Networks
TL;DR: The state-of-the-art performance of CNNs was achieved by Deep Convolutional Neural Networks (DCNNs) as discussed by the authors, which consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax.
Book
Elements of information theory
Thomas M. Cover,Joy A. Thomas +1 more
TL;DR: The author examines the role of entropy, inequality, and randomness in the design of codes and the construction of codes in the rapidly changing environment.
Journal ArticleDOI
Reducing the Dimensionality of Data with Neural Networks
TL;DR: In this article, an effective way of initializing the weights that allows deep autoencoder networks to learn low-dimensional codes that work much better than principal components analysis as a tool to reduce the dimensionality of data is described.
Journal ArticleDOI
Representation Learning: A Review and New Perspectives
TL;DR: Recent work in the area of unsupervised feature learning and deep learning is reviewed, covering advances in probabilistic models, autoencoders, manifold learning, and deep networks.
Book
Learning Deep Architectures for AI
TL;DR: The motivations and principles regarding learning algorithms for deep architectures, in particular those exploiting as building blocks unsupervised learning of single-layer modelssuch as Restricted Boltzmann Machines, used to construct deeper models such as Deep Belief Networks are discussed.