Statistical hypothesis testing

About: Statistical hypothesis testing is a research topic. Over the lifetime, 19580 publications have been published within this topic receiving 1037815 citations. The topic is also known as: statistical hypothesis testing & confirmatory data analysis.

Hypothesis testing and information theory

Abstract: The testing of binary hypotheses is developed from an information-theoretic point of view, and the asymptotic performance of optimum hypothesis testers is developed in exact analogy to the asymptotic performance of optimum channel codes. The discrimination, introduced by Kullback, is developed in a role analogous to that of mutual information in channel coding theory. Based on the discrimination, an error-exponent function e(r) is defined. This function is found to describe the behavior of optimum hypothesis testers asymptotically with block length. Next, mutual information is introduced as a minimum of a set of discriminations. This approach has later coding significance. The channel reliability-rate function E(R) is defined in terms of discrimination, and a number of its mathematical properties developed. Sphere-packing-like bounds are developed in a relatively straightforward and intuitive manner by relating e(r) and E (R) . This ties together the aforementioned developments and gives a lower bound in terms of a hypothesis testing model. The result is valid for discrete or continuous probability distributions. The discrimination function is also used to define a source code reliability-rate function. This function allows a simpler proof of the source coding theorem and also bounds the code performance as a function of block length, thereby providing the source coding analog of E (R) .

Beyond Moran's I: Testing for Spatial Dependence Based on the Spatial Autoregressive Model

Abstract: The statistic known as Moran's I is widely used to test for the presence of spatial dependence in observations taken on a lattice. Under the null hypothesis that the data are independent and identically distributed normal random variates, the distribution of Moran's I is known, and hypothesis tests based on this statistic have been shown in the literature to have various optimality properties. Given its simplicity, Moran's I is also frequently used outside of the formal hypothesis-testing setting in exploratory analyses of spatially referenced data; however, its limitations are not very well understood. To illustrate these limitations, we show that, for data generated according to the spatial autoregressive (SAR) model, Moran's I is only a good estimator of the SAR model's spatial-dependence parameter when the parameter is close to 0. In this research, we develop an alternative closed-form measure of spatial autocorrelation, which we call APLE, because it is an approximate profile-likelihood estimator (APLE) of the SAR model's spatial-dependence parameter. We show that APLE can be used as a test statistic for, and an estimator of, the strength of spatial autocorrelation. We include both theoretical and simulation-based motivations (including comparison with the maximum-likelihood estimator), for using APLE as an estimator. In conjunction, we propose the APLE scatterplot, an exploratory graphical tool that is analogous to the Moran scatterplot, and we demonstrate that the APLE scatterplot is a better visual tool for assessing the strength of spatial autocorrelation in the data than the Moran scatterplot. In addition, Monte Carlo tests based on both APLE and Moran's I are introduced and compared. Finally, we include an analysis of the well-known Mercer and Hall wheat-yield data to illustrate the difference between APLE and Moran's I when they are used in exploratory spatial data analysis.

Robust Inference for Generalized Linear Models

Abstract: By starting from a natural class of robust estimators for generalized linear models based on the notion of quasi-likelihood, we define robust deviances that can be used for stepwise model selection as in the classical framework. We derive the asymptotic distribution of tests based on robust deviances, and we investigate the stability of their asymptotic level under contamination. The binomial and Poisson models are treated in detail. Two applications to real data and a sensitivity analysis show that the inference obtained by means of the new techniques is more reliable than that obtained by classical estimation and testing procedures.

Adjusting the outputs of a classifier to new a priori probabilities: a simple procedure

Abstract: It sometimes happens (for instance in case control studies) that a classifier is trained on a data set that does not reflect the true a priori probabilities of the target classes on real-world data. This may have a negative effect on the classification accuracy obtained on the real-world data set, especially when the classifier's decisions are based on the a posteriori probabilities of class membership. Indeed, in this case, the trained classifier provides estimates of the a posteriori probabilities that are not valid for this real-world data set (they rely on the a priori probabilities of the training set). Applying the classifier as is (without correcting its outputs with respect to these new conditions) on this new data set may thus be suboptimal. In this note, we present a simple iterative procedure for adjusting the outputs of the trained classifier with respect to these new a priori probabilities without having to refit the model, even when these probabilities are not known in advance. As a by-product, estimates of the new a priori probabilities are also obtained. This iterative algorithm is a straightforward instance of the expectation-maximization (EM) algorithm and is shown to maximize the likelihood of the new data. Thereafter, we discuss a statistical test that can be applied to decide if the a priori class probabilities have changed from the training set to the real-world data. The procedure is illustrated on different classification problems involving a multilayer neural network, and comparisons with a standard procedure for a priori probability estimation are provided. Our original method, based on the EM algorithm, is shown to be superior to the standard one for a priori probability estimation. Experimental results also indicate that the classifier with adjusted outputs always performs better than the original one in terms of classification accuracy, when the a priori probability conditions differ from the training set to the real-world data. The gain in classification accuracy can be significant.

Bayesian Assessment of Null Values Via Parameter Estimation and Model Comparison

Abstract: Psychologists have been trained to do data analysis by asking whether null values can be rejected. Is the difference between groups nonzero? Is choice accuracy not at chance level? These questions have been traditionally addressed by null hypothesis significance testing (NHST). NHST has deep problems that are solved by Bayesian data analysis. As psychologists transition to Bayesian data analysis, it is natural to ask how Bayesian analysis assesses null values. The article explains and evaluates two different Bayesian approaches. One method involves Bayesian model comparison (and uses Bayes factors). The second method involves Bayesian parameter estimation and assesses whether the null value falls among the most credible values. Which method to use depends on the specific question that the analyst wants to answer, but typically the estimation approach (not using Bayes factors) provides richer information than the model comparison approach.