Showing papers on "Natural exponential family published in 2020"

Bayesian Hierarchical Models With Conjugate Full-Conditional Distributions for Dependent Data From the Natural Exponential Family

Abstract: We introduce a Bayesian approach for analyzing (possibly) high-dimensional dependent data that are distributed according to a member from the natural exponential family of distributions. This probl...

Characterizations of non-normalized discrete probability distributions and their application in statistics

Abstract: From the distributional characterizations that lie at the heart of Stein's method we derive explicit formulae for the mass functions of discrete probability laws that identify those distributions. These identities are applied to develop tools for the solution of statistical problems. Our characterizations, and hence the applications built on them, do not require any knowledge about normalization constants of the probability laws. We discuss several examples where this lack of feasibility of the normalization constant is a built-in feature. To demonstrate that our statistical methods are sound, we provide comparative simulation studies for the testing of fit to the Poisson distribution and for parameter estimation of the negative binomial family when both parameters are unknown. We also consider the problem of parameter estimation for discrete exponential-polynomial models which generally are non-normalized.

Calibrating the scan statistic: finite sample performance vs. asymptotics

Abstract: We consider the problem of detecting an elevated mean on an interval with unknown location and length in the univariate Gaussian sequence model. Recent results have shown that using scale-dependent critical values for the scan statistic allows to attain asymptotically optimal detection simultaneously for all signal lengths, thereby improving on the traditional scan, but this procedure has been criticized for losing too much power for short signals. We explain this discrepancy by showing that these asymptotic optimality results will necessarily be too imprecise to discern the performance of scan statistics in a practically relevant way, even in a large sample context. Instead, we propose to assess the performance with a new finite sample criterion. We then present three calibrations for scan statistics that perform well across a range of relevant signal lengths: The first calibration uses a particular adjustment to the critical values and is therefore tailored to the Gaussian case. The second calibration uses a scale-dependent adjustment to the significance levels and is therefore applicable to arbitrary known null distributions. The third calibration restricts the scan to a particular sparse subset of the scan windows and then applies a weighted Bonferroni adjustment to the corresponding test statistics. This {\sl Bonferroni scan} is also applicable to arbitrary null distributions and in addition is very simple to implement. We show how to apply these calibrations for scanning in a number of distributional settings: for normal observations with an unknown baseline and a known or unknown constant variance,for observations from a natural exponential family, for potentially heteroscadastic observations from a symmetric density by employing self-normalization in a novel way, and for exchangeable observations using tests based on permutations, ranks or signs.

A fast score test for generalized mixture models

Abstract: In biomedical studies, parametric mixture models are widely used for modeling data collected from a non-homogenous population. For instance, in genetic linkage analysis, parametric admixture models have been considered to account for the fact that linkage may exist only in a proportion of families (Liang and Rathouz, 1999; Fu et al., 2006; Di and Liang, 2011). In microarray data analysis, mixture models have been applied to test for partially differentially expressed genes (Ghosh and Chinnaiyan, 2009; Van Wieringen and Van de Viel, 2009). Generally, testing for homogeneity in mixture models is a nonregular problem because the mixture proportion parameter lies on the boundary of its parameter space and some nuisance parameters are absent under the null hypothesis of homogeneity. Therefore, theoretical development for the asymptotic distributions of tests is more difficult than that in regular statistical models. Much work has been done for testing homogeneity in parametric mixture models. For example, a score test is proposed by Liang and Rathouz (1999), and (modified) likelihood ratio tests are derived by Lemdani and Pons (1999), Chen et al. (2001) and Di and Liang (2011), among many others. For more comprehensive overview of parametric mixture models, see McLachlan and Basford (1988) and Lindsay (1995). However, in many modern applications such as the DNA methylation data analysis, the distributions of methylation values are often highly heterogeneous across different CpG sites (Wang, 2011), such that one single parametric mixture model cannot provide sufficient goodness of fit for the methylation data measured at different CpG sites. To relax the parametric assumption, Qin and Liang (2011) proposed a two-sample semiparametric exponential tilt mixture model (ETMM), where the distribution of the reference group is completely unspecified and the distribution of the other group is a mixture of two unknown distributions linked by an exponential tilt with a single parameter. Such an ETMM contains the mixture of natural exponential family (NEF) models (Morris, 2006). However, this ETMM still does not provide sufficient degrees of freedom in modeling, because it only allows one single free parameter which reduces to the non-homogeneity of means in normal models. In applications such as DNA methylation data, the two experimental groups can be different in variations but not in means, or different in both means and variations, and such difference in variation may suggest important associations between the CpG sites and diseases (Teschendorff et al., 2014; Sun et al., 2017). In such circumstances, existing methods are not sufficient to capture these key features of data and can lead to a substantial loss of power. Hong et al. (2017) proposed a pseudolikelihood EM based test (PLEMT) using a multi-parameter generalized exponential tilt mixture model (GETMM). The GETMM generalizes the ETMM by incorporating pre-specified multidimensional basis functions, which offers more flexible characterization of the non-homogeneity between two groups. Unlike the empirical likelihood method in Qin and Liang (2011), a pairwise conditioning procedure was used to eliminate the unspecified distribution function, and a penalized pseudolikelihood was constructed to handle the identifiability problem. Despite the superior power of the PLEMT over existing tests in capturing the signals in mean and variations, there are several limitations. First, it involves choosing multiple tuning parameters including the penalty parameter and number of steps for the EM algorithm. There is no universally accepted values for the tuning parameters. Second, this method is computationally expensive since EM algorithm is used to obtain the test statistic. In this paper, we propose a novel score test for homogeneity based on GETMM. We adopt the same procedure as in Hong et al. (2017) in elimination of the unspecified distribution function of the reference group. Instead of using penalization, we work on the pseudolikelihood directly and show that the parameters of interest are still identifiable. To address the challenge that some nuisance parameters disappear under the null hypothesis, we modify the score function by a novel procedure and derive the asymptotic null distribution of the proposed score test. Unlike the standard asymptotic results in Davies (1977, 1987), we unveil new Wilks phenomenon of the proposed score test, i.e., asymptotic null distributions are independent of nuisance parameters. The simple asymptotic distribution controls the Type I error well, and avoids the need of permutation based procedures. In addition, we show that, with fixed nuisance parameters, the proposed score test is locally most powerful among a class of tests. This justifies the optimality of our method in finite samples. To obtain further insight about the power of the proposed test, we derive the asymptotic power functions under local alternative hypotheses. Two types of local alternatives are studied, with respect to two different geometric paths for approximating the null hypothesis. This result is particularly useful in performing sample size calculation and experimental design. The rest of this article is organized as follows. In Section 2, we introduce the GETMM and construct the pseudolikelihood. In Section 3, we derive the score test based on the constructed pseudolikelihood and derive the asymptotic null distribution. In Section 4, we derive the local asymptotic power of the proposed tests and provide insights on experimental designs. In Section 5, we conduct simulation studies to compare the proposed tests with the existing ones. In Section 6, we apply these tests to the methylation data from an ovarian cancer study. Concluding remarks are given in Section 7.

Asymptotic normality of the test statistics for the unified relative dispersion and relative variation indexes

Abstract: Dispersion indexes with respect to the Poisson and binomial distributions are widely used to assess the conformity of the underlying distribution from an observed sample of the count with one or th...

Distributionally Robust Parametric Maximum Likelihood Estimation

Abstract: We consider the parameter estimation problem of a probabilistic generative model prescribed using a natural exponential family of distributions. For this problem, the typical maximum likelihood estimator usually overfits under limited training sample size, is sensitive to noise and may perform poorly on downstream predictive tasks. To mitigate these issues, we propose a distributionally robust maximum likelihood estimator that minimizes the worst-case expected log-loss uniformly over a parametric Kullback-Leibler ball around a parametric nominal distribution. Leveraging the analytical expression of the Kullback-Leibler divergence between two distributions in the same natural exponential family, we show that the min-max estimation problem is tractable in a broad setting, including the robust training of generalized linear models. Our novel robust estimator also enjoys statistical consistency and delivers promising empirical results in both regression and classification tasks.

Convolutional dictionary learning based auto-encoders for natural exponential-family distributions

Abstract: We introduce a class of auto-encoder neural networks tailored to data from the natural exponential family (e.g., count data). The architectures are inspired by the problem of learning the filters in a convolutional generative model with sparsity constraints, often referred to as convolutional dictionary learning (CDL). Our work is the first to combine ideas from convolutional generative models and deep learning for data that are naturally modeled with a non-Gaussian distribution (e.g., binomial and Poisson). This perspective provides us with a scalable and flexible framework that can be re-purposed for a wide range of tasks and assumptions on the generative model. Specifically, the iterative optimization procedure for solving CDL, an unsupervised task, is mapped to an unfolded and constrained neural network, with iterative adjustments to the inputs to account for the generative distribution. We also show that the framework can easily be extended for discriminative training, appropriate for a supervised task. We demonstrate 1) that fitting the generative model to learn, in an unsupervised fashion, the latent stimulus that underlies neural spiking data leads to better goodness-of-fit compared to other baselines, 2) competitive performance compared to state-of-the-art algorithms for supervised Poisson image denoising, with significantly fewer parameters, and 3) gradient dynamics of shallow binomial auto-encoder.

Compounded generalized Weibull distributions - A unified approach

Abstract: In this paper we analyze a unified approach to study a family of lifetime distributions of a system consisting of random number of components in series and in parallel proposed by Chowdhury (2014)

Distributionally Robust Parametric Maximum Likelihood Estimation

Abstract: We consider the parameter estimation problem of a probabilistic generative model prescribed using a natural exponential family of distributions. For this problem, the typical maximum likelihood estimator usually overfits under limited training sample size, is sensitive to noise and may perform poorly on downstream predictive tasks. To mitigate these issues, we propose a distributionally robust maximum likelihood estimator that minimizes the worst-case expected log-loss uniformly over a parametric Kullback-Leibler ball around a parametric nominal distribution. Leveraging the analytical expression of the Kullback-Leibler divergence between two distributions in the same natural exponential family, we show that the min-max estimation problem is tractable in a broad setting, including the robust training of generalized linear models. Our novel robust estimator also enjoys statistical consistency and delivers promising empirical results in both regression and classification tasks.

Mean Shrinkage Estimation for High-Dimensional Diagonal Natural Exponential Families

Abstract: Shrinkage estimators have been studied widely in statistics and have profound impact in many applications. In this paper, we study simultaneous estimation of the mean parameters of random observations from a diagonal multivariate natural exponential family. More broadly, we study distributions for which the diagonal entries of the covariance matrix are certain quadratic functions of the mean parameter. We propose two classes of semi-parametric shrinkage estimators for the mean vector and construct unbiased estimators of the corresponding risk. Further, we establish the asymptotic consistency and convergence rates for these shrinkage estimators under squared error loss as both $n$, the sample size, and $p$, the dimension, tend to infinity. Finally, we consider the diagonal multivariate natural exponential families, which have been classified as consisting of the normal, Poisson, gamma, multinomial, negative multinomial, and hybrid classes of distributions. We deduce consistency of our estimators in the case of the normal, gamma, and negative multinomial distributions if $p n^{-1/3}\log^{4/3}{n} \rightarrow 0$ as $n,p \rightarrow \infty$, and for Poisson and multinomial distributions if $pn^{-1/2} \rightarrow 0$ as $n,p \rightarrow \infty$.