This paper considers Bayesian counterparts of the classical tests for good- ness of fit and their use in judging the fit of a single Bayesian model to the observed data. We focus on posterior predictive assessment, in a framework that also includes conditioning on auxiliary statistics. The Bayesian formulation facilitates the con- struction and calculation of a meaningful reference distribution not only for any (classical) statistic, but also for any parameter-dependent "statistic" or discrep- ancy. The latter allows us to propose the realized discrepancy assessment of model fitness, which directly measures the true discrepancy between data and the posited model, for any aspect of the model which we want to explore. The computation required for the realized discrepancy assessment is a straightforward byproduct of the posterior simulation used for the original Bayesian analysis. We illustrate with three applied examples. The first example, which serves mainly to motivate the work, illustrates the difficulty of classical tests in assessing the fitness of a Poisson model to a positron emission tomography image that is constrained to be nonnegative. The second and third examples illustrate the details of the posterior predictive approach in two problems: estimation in a model with inequality constraints on the parameters, and estimation in a mixture model. In all three examples, standard test statistics (either a χ 2 or a likelihood ratio) are not pivotal: the difficulty is not just how to compute the reference distribution for the test, but that in the classical framework no such distribution exists, independent of the unknown model parameters.

/pdf/posterior-predictive-assessment-of-model-fitness-via-1wc3dk7ymj.pdf

Posterior predictive assessment of model fitness via realized discrepancies

The Analysis of Variance.

Extending work of Rubin, this paper explores a Bayesian counterpart of the classical $p$-value, namely, a tail-area probability of a "test statistic" under a null hypothesis. The Bayesian formulation, using posterior predictive replications of the data, allows a "test statistic" to depend on both data and unknown (nuisance) parameters and thus permits a direct measure of the discrepancy between sample and population quantities. The tail-area probability for a "test statistic" is then found under the joint posterior distribution of replicate data and the (nuisance) parameters, both conditional on the null hypothesis. This posterior predictive $p$-value can also be viewed as the posterior mean of a classical $p$-value, averaging over the posterior distribution of (nuisance) parameters under the null hypothesis, and thus it provides one general method for dealing with nuisance parameters. Two classical examples, including the Behrens-Fisher problem, are used to illustrate the posterior predictive $p$-value and some of its interesting properties, which also reveal a new (Bayesian) interpretation for some classical $p$-values. An application to multiple-imputation inference is also presented. A frequency evaluation shows that, in general, if the replication is defined by new (nuisance) parameters and new data, then the Type I frequentist error of an $\alpha$-level posterior predictive test is often close to but less than $\alpha$ and will never exceed $2\alpha$.

/pdf/posterior-predictive-p-values-5qvbpwbiaz.pdf

Posterior Predictive $p$-Values

The definition of a confidence interval is generalized so that problems such as constructing exact confidence regions for the difference in two normal means can be tackled without the assumption of equal variances. Under certain conditions, the extended definition is shown to preserve a repeated sampling property that a practitioner expects from exact confidence intervals. The proposed procedure is also applied to the problem of constructing confidence intervals for the difference in two exponential means and for variance components in mixed models. A repeated sampling property of generalized p values is also given. With this characterization one can carry out fixed level tests of parameters of continuous distributions on the basis of generalized p values. Finally, Pratt's paradox is revisited, and a procedure that resolves the paradox is given.

Generalized Confidence Intervals

Customer attrition analysis for financial services using proportional hazard models

This article examines some problems of significance testing for one-sided hypotheses of the form H 0 : θ ≤ θ 0 versus H 1 : θ > θ 0, where θ is the parameter of interest. In the usual setting, let x be the observed data and let T(X) be a test statistic such that the family of distributions of T(X) is stochastically increasing in θ. Define Cx as {X : T(X) — T(x) ≥ 0}. The p value is p(x) = sup θ≤θ0 Pr(X ∈ Cx | θ). In the presence of a nuisance parameter η, there may not exist a nontrivial Cx with a p value independent of η. We consider tests based on generalized extreme regions of the form Cx (θ, η) = {X : T(X; x, θ, η) ≥ T(x; x, θ, η)}, and conditions on T(X; x, θ, η) are given such that the p value p(x) = sup θ≤θ0 Pr(X ∈ Cx (θ, η)) is free of the nuisance parameter η, where T is stochastically increasing in θ. We provide a solution to the problem of testing hypotheses about the differences in means of two independent exponential distributions, a problem for which the fixed-level testing approach...

/pdf/generalized-p-values-in-significance-testing-of-hypotheses-4bb50fg6b0.pdf

Generalized p-Values in Significance Testing of Hypotheses in the Presence of Nuisance Parameters

Exact Statistical Methods for Data Analysis

1 Preliminary Notions.- 1.1 Introduction.- 1.2 Sufficiency.- 1.3 Complete Sufficient Statistics.- 1.4 Exponential Families of Distributions.- 1.5 Invariance.- 1.6 Maximum Likelihood Estimation.- 1.7 Unbiased Estimation.- 1.8 Least Squares Estimation.- 1.9 Interval Estimation.- Exercises.- 2 Notions in significance testing of hypotheses.- 2.1 Introduction.- 2.2 Test Statistics and Test Variables.- 2.3 Definition of p-Value.- 2.4 Generalized Likelihood Ratio Method.- 2.5 Invariance in Significance Testing.- 2.6 Unbiasedness and Similarity.- 2.7 Interval Estimation and Fixed-Level Testing.- Exercises.- 3 Review of Special Distributions.- 3.1 Poisson and Binomial Distributions.- 3.2 Point Estimation and Interval Estimation.- 3.3 Significance Testing of Parameters.- 3.4 Bayesian Inference.- 3.5 The Normal Distribution.- 3.6 Inferences About the Mean.- 3.7 Inferences About the Variance.- 3.8 Quantiles of a Normal Distribution.- 3.9 Conjugate Prior and Posterior Distributions.- 3.10 Bayesian Inference About the Mean and the Variance.- Exercises.- 4 Exact Nonparametric Methods.- 4.1 Introduction.- 4.2 The Sign Test.- 4.3 The Signed Rank Test and the Permutation Test.- 4.4 The Rank Sum Test and Allied Tests.- 4.5 Comparing k Populations.- 4.6 Contingency Tables.- 4.7 Testing the Independence of Criteria of Classification.- 4.8 Testing the Homogeneity of Populations.- Exercises.- 5 Generalized p-Values.- 5.1 Introduction.- 5.2 Generalized Test Variables.- 5.3 Definition of Generalized p-Values.- 5.4 Frequency Interpretations and Generalized Fixed-Level Tests.- 5.5 Invariance.- 5.6 Comparing the Means of Two Exponential Distributions.- 5.7 Unbiasedness and Similarity.- 5.7 Comparing the Means of an Exponential Distribution and a Normal Distribution.- Exercises.- 6 Generalized Confidence Intervals.- 6.1 Introduction.- 6.2 Generalized Definitions.- 6.3 Frequency Interpretations and Repeated Sampling Properties.- 6.4 Invariance in Interval Estimation.- 6.5 Interval Estimation of the Difference Between Two Exponential Means.- 6.6 Similarity in Interval Estimation.- 6.7 Generalized Confidence Intervals Based on p-Values.- 6.8 Resolving an Undesirable Feature of Confidence Intervals.- 6.9 Bayesian and Conditional Confidence Intervals.- Exercises.- 7 Comparing Two Normal Populations.- 7.1 Introduction.- 7.2 Comparing the Means when the Variances are Equal.- 7.3 Solving the Behrens-Fisher Problem.- 7.4 Inferences About the Ratio of Two Variances.- 7.5 Inferences About the Difference in Two Variances.- 7.6 Bayesian Inference.- 7.7 Inferences About the Reliability Parameter.- 7.8 The Case of Known Stress Distribution.- Exercises.- 8 Analysis of Variance.- 8.1 Introduction.- 8.2 One-way Layout.- 8.3 Testing the Equality of Means.- 8.4 ANOVA with Unequal Error Variances.- 8.5 Multiple Comparisons.- 8.6 Testing the Equality of Variances.- 8.7 Two-way ANOVA without Replications.- 8.8 ANOVA in a Balanced Two-way Layout with Replications.- 8.9 Two-way ANOVA under Heteroscedasticity.- Exercises.- 9 Mixed Models.- 9.1 Introduction.- 9.2 One-way Layout.- 9.3 Testing Variance Components.- 9.4 Confidence Intervals.- 9.5 Two-way Layout.- 9.6 Comparing Variance Components.- Exercises.- 10 Regression.- 10.1 Introduction.- 10.2 Simple Linear Regression Model.- 10.3. Inferences about Parameters of the Simple Regression Model.- 10.3 Multiple Linear Regression.- 10.4 Distributions of Estimators and Significance Tests.- 10.5 Comparing Two Regressions with Equal Variances.- 10.6 Comparing Regressions without Common Parameters.- 10.7 Comparison of Two General Models.- Exercises.- Appendix A.- Elements of Bayesian Inference.- A.1 Introduction.- A.2 The Prior Distribution.- A.3 The Posterior Distribution.- A.4 Bayes Estimators.- A.5 Bayesian Interval Estimation.- A.6 Bayesian Hypothesis Testing.- Appendix B Technical Arguments.- References.

We consider the stress-strength problem in which a unit of strength X is subjected to environmental stress Y. An important problem in stress-strength reliability concerns testing hypotheses about the reliability parameter R = P[X > yl. In this article, we consider situations in which X and Y are independent and have normal distributions or can be transformed to normality. We do not require the two population variances to be equal. Our approach leads to test statistics which are exact p values that are represented as one-dimensional integrals. On the basis of the p value, one can also construct approximate confidence intervals for the parameter of interest. We also present an extension of the testing procedure to the case in which both strength and stress depend on covariates. For comparative purposes, the Bayesian solution to the problem is also presented. We use data from a rocket-motor experiment to illustrate the procedure.

Samaradasa Weerahandi

Papers

Generalized Confidence Intervals

Generalized p-Values in Significance Testing of Hypotheses in the Presence of Nuisance Parameters

Exact Statistical Methods for Data Analysis

Exact Statistical Methods for Data Analysis

Testing reliability in a stress-strength model when X and Y are normally distributed