Deep learning and the information bottleneck principle
Naftali Tishby,Noga Zaslavsky +1 more
- pp 1-5
TLDR
It is argued that both the optimal architecture, number of layers and features/connections at each layer, are related to the bifurcation points of the information bottleneck tradeoff, namely, relevant compression of the input layer with respect to the output layer.Abstract:
Deep Neural Networks (DNNs) are analyzed via the theoretical framework of the information bottleneck (IB) principle. We first show that any DNN can be quantified by the mutual information between the layers and the input and output variables. Using this representation we can calculate the optimal information theoretic limits of the DNN and obtain finite sample generalization bounds. The advantage of getting closer to the theoretical limit is quantifiable both by the generalization bound and by the network's simplicity. We argue that both the optimal architecture, number of layers and features/connections at each layer, are related to the bifurcation points of the information bottleneck tradeoff, namely, relevant compression of the input layer with respect to the output layer. The hierarchical representations at the layered network naturally correspond to the structural phase transitions along the information curve. We believe that this new insight can lead to new optimality bounds and deep learning algorithms.read more
Citations
More filters
Posted Content
Towards Safety Verification of Direct Perception Neural Networks
TL;DR: This work approaches the specification problem by learning an input property characterizer which carefully extends a direct perception neural network at close-to-output layers, and addresses the scalability problem by a novel assume-guarantee based verification approach.
Posted Content
Unsupervised Discovery of Object Landmarks via Contrastive Learning
TL;DR: It is shown that when a deep network is trained to be invariant to geometric and photometric transformations, representations from its intermediate layers are highly predictive of object landmarks and by stacking representations across layers in a hypercolumn their effectiveness can be improved.
Proceedings ArticleDOI
FETA: Towards Specializing Foundation Models for Expert Task Applications
Amit Alfassy,Assaf Arbelle,Oshri Halimi,Sivan Harary,Roei Herzig,Eli Schwartz,Rameswar Panda,Michele Dolfi,Christoph Auer,Ksenia Saenko,Peter Staar,Rogerio Feris,Leonid Karlinsky +12 more
TL;DR: A first of its kind FETA benchmark built around the task of teaching FMs to understand technical documentation, via learning to match their graphical illustrations to corresponding language descriptions, and leads to an automated performance metric shown to be consistent with metrics computed on human-curated annotations.
Proceedings ArticleDOI
An Alphabet-Size Bound for the Information Bottleneck Function
Christoph Hirche,Andreas Winter +1 more
TL;DR: It is shown that the information bottleneck function can always be approximated up to an error with a cardinality $delta (\varepsilon,\;|\mathcal{Y}|)$ with a Cardinality bound for explicitly given functions δ and f of an approximation parameter ϵ > 0 and the cardinality of $\mathcal {Y}$.
Proceedings ArticleDOI
Dissipation in neuromorphic computing: Fundamental bounds for feedforward networks
Natesh Ganesh,Neal G. Anderson +1 more
TL;DR: The fundamental lower bound on dissipation in feedforward neural networks associated with the combined cost of the training and testing phases is presented and neural network learning algorithms that minimize the total dissipation cost of training are explored.
References
More filters
Proceedings Article
ImageNet Classification with Deep Convolutional Neural Networks
TL;DR: The state-of-the-art performance of CNNs was achieved by Deep Convolutional Neural Networks (DCNNs) as discussed by the authors, which consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax.
Book
Elements of information theory
Thomas M. Cover,Joy A. Thomas +1 more
TL;DR: The author examines the role of entropy, inequality, and randomness in the design of codes and the construction of codes in the rapidly changing environment.
Journal ArticleDOI
Reducing the Dimensionality of Data with Neural Networks
TL;DR: In this article, an effective way of initializing the weights that allows deep autoencoder networks to learn low-dimensional codes that work much better than principal components analysis as a tool to reduce the dimensionality of data is described.
Journal ArticleDOI
Representation Learning: A Review and New Perspectives
TL;DR: Recent work in the area of unsupervised feature learning and deep learning is reviewed, covering advances in probabilistic models, autoencoders, manifold learning, and deep networks.
Book
Learning Deep Architectures for AI
TL;DR: The motivations and principles regarding learning algorithms for deep architectures, in particular those exploiting as building blocks unsupervised learning of single-layer modelssuch as Restricted Boltzmann Machines, used to construct deeper models such as Deep Belief Networks are discussed.