Estimation of entropy and mutual information
Reads0
Chats0
TLDR
In this article, the authors use an exact local expansion of the entropy function to prove almost sure consistency and central limit theorems for three of the most commonly used discretized information estimators.Abstract:
We present some new results on the nonparametric estimation of entropy and mutual information. First, we use an exact local expansion of the entropy function to prove almost sure consistency and central limit theorems for three of the most commonly used discretized information estimators. The setup is related to Grenander's method of sieves and places no assumptions on the underlying probability measure generating the data. Second, we prove a converse to these consistency theorems, demonstrating that a misapplication of the most common estimation techniques leads to an arbitrarily poor estimate of the true information, even given unlimited data. This "inconsistency" theorem leads to an analytical approximation of the bias, valid in surprisingly small sample regimes and more accurate than the usual 1/N formula of Miller and Madow over a large region of parameter space. The two most practical implications of these results are negative: (1) information estimates in a certain data regime are likely contaminated by bias, even if "bias-corrected" estimators are used, and (2) confidence intervals calculated by standard techniques drastically underestimate the error of the most common estimation methods.Finally, we note a very useful connection between the bias of entropy estimators and a certain polynomial approximation problem. By casting bias calculation problems in this approximation theory framework, we obtain the best possible generalization of known asymptotic bias results. More interesting, this framework leads to an estimator with some nice properties: the estimator comes equipped with rigorous bounds on the maximum error over all possible underlying probability distributions, and this maximum error turns out to be surprisingly small. We demonstrate the application of this new estimator on both real and simulated data.read more
Citations
More filters
Journal ArticleDOI
Rarefaction and extrapolation with Hill numbers: a framework for sampling and estimation in species diversity studies
Anne Chao,Nicholas J. Gotelli,T. C. Hsieh,Elizabeth L. Sander,K. H. Ma,Robert K. Colwell,Robert K. Colwell,Aaron M. Ellison +7 more
TL;DR: In this article, the authors extended previous rarefaction and extrapolation models for species richness (Hill number q D, where q ¼ 0) to measures of taxon diversity incorporating relative abundance (i.e., for any Hill number qD, q. 0) and presented a unified approach for both individual-based (abundance) data and sample-based data.
Posted Content
Opening the Black Box of Deep Neural Networks via Information
Ravid Shwartz-Ziv,Naftali Tishby +1 more
TL;DR: This work demonstrates the effectiveness of the Information-Plane visualization of DNNs and shows that the training time is dramatically reduced when adding more hidden layers, and the main advantage of the hidden layers is computational.
Journal Article
Conditional likelihood maximisation: a unifying framework for information theoretic feature selection
TL;DR: Overall it is concluded that the JMI criterion provides the best tradeoff in terms of accuracy, stability, and flexibility with small data samples.
Proceedings Article
Mutual Information Neural Estimation.
Mohamed Ishmael Belghazi,Aristide Baratin,Sai Rajeshwar,Sherjil Ozair,Yoshua Bengio,Aaron Courville,Devon Hjelm +6 more
TL;DR: A Mutual Information Neural Estimator (MINE) is presented that is linearly scalable in dimensionality as well as in sample size, trainable through back-prop, and strongly consistent, and applied to improve adversarially trained generative models.
Journal ArticleDOI
Causality detection based on information-theoretic approaches in time series analysis
Katerina Hlaváčková-Schindler,Milan Paluš,Martin Vejmelka,Joydeep Bhattacharya,Joydeep Bhattacharya +4 more
TL;DR: The aim of this paper is to provide a detailed overview of information theoretic approaches for measuring causal influence in multivariate time series and to focus on diverse approaches to the entropy and mutual information estimation.
References
More filters
Journal ArticleDOI
A mathematical theory of communication
TL;DR: This final installment of the paper considers the case where the signals or the messages or both are continuously variable, in contrast with the discrete nature assumed until now.
Book
Elements of information theory
Thomas M. Cover,Joy A. Thomas +1 more
TL;DR: The author examines the role of entropy, inequality, and randomness in the design of codes and the construction of codes in the rapidly changing environment.
Numerical recipes in C
TL;DR: The Diskette v 2.06, 3.5''[1.44M] for IBM PC, PS/2 and compatibles [DOS] Reference Record created on 2004-09-07, modified on 2016-08-08.
Journal Article
The mathematical theory of communication
Claude E. Shannon,Warren Weaver +1 more
TL;DR: The Mathematical Theory of Communication (MTOC) as discussed by the authors was originally published as a paper on communication theory more than fifty years ago and has since gone through four hardcover and sixteen paperback printings.