Review: Divergence measures for statistical data processing-An annotated bibliography

We propose and analyse estimators for statistical functionals of one or more distributions under nonparametric assumptions. Our estimators are derived from the von Mises expansion and are based on the theory of influence functions, which appear in the semiparametric statistics literature. We show that estimators based either on data-splitting or a leave-one-out technique enjoy fast rates of convergence and other favorable theoretical properties. We apply this framework to derive estimators for several popular information theoretic quantities, and via empirical evaluation, show the advantage of this approach over existing estimators.

/pdf/nonparametric-von-mises-estimators-for-entropies-divergences-1urtcq3u6j.pdf

Nonparametric von Mises estimators for entropies, divergences and mutual informations

It was recently shown that estimating the Shannon entropy H(p) of a discrete k-symbol distribution p requires Θ(k/log k) samples, a number that grows near-linearly in the support size. In many applications H(p) can be replaced by the more general Renyi entropy of order α, Hα(p). We determine the number of samples needed to estimate Hα(p) for all α, showing that α 1 requires near-linear, roughly k samples, but integer α > 1 requires only Θ(k1-1/α) samples. In particular, estimating H2(p), which arises in security, DNA reconstruction, closeness testing, and other applications, requires only Θ([EQUATION]k) samples. The estimators achieving these bounds are simple and run in time linear in the number of samples.

http://web.eng.ucsd.edu/~htyagi/AcharyaOST14conf.pdf

The complexity of estimating Rényi entropy

It was shown recently that estimating the Shannon entropy $H(p)$ of a discrete $k$ -symbol distribution $p$ requires $\Theta (k/\log k)$ samples, a number that grows near-linearly in the support size. In many applications, $H(p)$ can be replaced by the more general Renyi entropy of order $ \alpha $ and $ H_{\alpha }(p)$ . We determine the number of samples needed to estimate $ H_{\alpha }(p)$ for all $ \alpha $ , showing that $ \alpha requires a super-linear, roughly $k^{1/ \alpha }$ samples, noninteger $ \alpha >1$ requires a near-linear $k$ samples, but, perhaps surprisingly, integer $\alpha >1$ requires only $\Theta (k^{1-1/\alpha })$ samples. Furthermore, developing on a recently established connection between polynomial approximation and estimation of additive functions of the form $\sum _{x}f (p_{x})$ , we reduce the sample complexity for noninteger values of $\alpha $ by a factor of $\log k$ compared with the empirical estimator. The estimators achieving these bounds are simple and run in time linear in the number of samples. Our lower bounds provide explicit constructions of distributions with different Renyi entropies that are hard to distinguish.

/pdf/estimating-renyi-entropy-of-discrete-distributions-1osjog8qou.pdf

Estimating Renyi Entropy of Discrete Distributions

We consider nonparametric estimation of L2, Renyi-α and Tsallis-α divergences between continuous distributions. Our approach is to construct estimators for particular integral functionals of two densities and translate them into divergence estimators. For the integral functionals, our estimators are based on corrections of a preliminary plug-in estimator. We show that these estimators achieve the parametric convergence rate of n-1/2 when the densities' smoothness, s, are both at least d/4 where d is the dimension. We also derive minimax lower bounds for this problem which confirm that s > d/4 is necessary to achieve the n-1/2 rate of convergence. We validate our theoretical guarantees with a number of simulations.

/pdf/nonparametric-estimation-of-renyi-divergence-and-friends-4w0he14oxs.pdf

Nonparametric Estimation of Renyi Divergence and Friends

Statistical inference for the ε-entropy and the quadratic Rényi entropy

Numerous entropy-type characteristics (functionals) generalizing Renyi entropy are widely used in mathematical statistics, physics, information theory, and signal processing for charac- terizing uncertainty in probability distributions and distribution identification problems. We consider estimators of some entropy (integral) functionals for discrete and continuous distribu- tions based on the number of epsilon-close vector records in the corresponding independent and identically distributed samples from two distributions. The estimators form a triangular scheme of generalized U-statistics. We show the asymptotic properties of these estimators (e.g., consis- tency and asymptotic normality). The results can be applied in various problems in computer science and mathematical statistics (e.g., approximate matching for random databases, record linkage, image matching).

Statistical Inference for Renyi Entropy Functionals

Numerous entropy-type characteristics (functionals) generalizing Renyi entropy are widely used in mathematical statistics, physics, information theory, and signal processing for characterizing uncertainty in probability distributions and distribution identification problems. We consider estimators of some entropy (integral) functionals for discrete and continuous distributions based on the number of epsilon-close vector records in the corresponding independent and identically distributed samples from two distributions. The proposed estimators are generalized U-statistics. We show the asymptotic properties of these estimators (e.g., consistency and asymptotic normality). The results can be applied in various problems in computer science and mathematical statistics (e.g., approximate matching for random databases, record linkage, image matching).

AMS 2000 subject classification: 94A15, 62G20

Statistical inference for rényi entropy functionals

Numerous entropy-type characteristics (functionals) generalizing R\'enyi entropy are widely used in mathematical statistics, physics, information theory, and signal processing for characterizing uncertainty in probability distributions and distribution identification problems. We consider estimators of some entropy (integral) functionals for discrete and continuous distributions based on the number of epsilon-close vector records in the corresponding independent and identically distributed samples from two distributions. The estimators form a triangular scheme of generalized U-statistics. We show the asymptotic properties of these estimators (e.g., consistency and asymptotic normality). The results can be applied in various problems in computer science and mathematical statistics (e.g., approximate matching for random databases, record linkage, image matching).

Statistical Inference for R\'enyi Entropy Functionals

Matching a query (reference) image to an image extracted from a database containing (possibly) transformed image copies is an important retrieval task. In this paper we present a general method based on matching densities of the corresponding image feature vectors by using the Bregman distances. We consider statistical estimators for some quadratic entropy-type characteristics. In particular, the quadratic Bregman distances can be evaluated in image matching problems whenever images are modeled by random feature vectors in large image databases. Moreover, this method can be used for average case analysis for optimization of joining large databases.

Nikolaj Leonenko

Papers

Statistical inference for the ε-entropy and the quadratic Rényi entropy

Statistical Inference for Renyi Entropy Functionals

Statistical inference for rényi entropy functionals

Statistical Inference for R\'enyi Entropy Functionals

Statistical Modeling for Image Matching in Large Image Databases