On Non-parametric Testing, the Uniform Behaviour of the t-test, and Related Problems
TL;DR: In this article, the authors revisited some problems in nonparametric hypothesis testing, such as testing whether the mean is rational, testing goodness-of-fit, and equivalence testing.
Abstract: In this article, we revisit some problems in non-parametric hypothesis testing. First, we extend the classical result of Bahadur & Savage (Ann. Math. ,Statist. 25 (1956) 11151 to other testing problems, and we answer a conjecture of theirs. Other examples considered are testing whether or not the mean is rational, testing goodness-of-fit, and equivalence testing. Next, we discuss the uniform behaviour of the classical t-test. For most non-parametric models, the Baha- dur-Savage result yields that the size of the t-test is one for every sample size. Even if we restrict attention to the family of symmetric distributions supported on a fixed compact set, the t-test is not even uniformly asymptotically level a. However, the convergence of the rejection probability is established uniformly over a large family with a very weak uniform integrability type of condition. Furthermore, under such a restriction, the t-test possesses an asymptotic maximin optimality property.
Citations
More filters
TL;DR: The authors proposed robust methods for inference about the effect of a treatment variable on a scalar outcome in the presence of very many regressors in a model with possibly non-Gaussian and heteroscedastic disturbances.
Abstract: We propose robust methods for inference about the effect of a treatment variable on a scalar outcome in the presence of very many regressors in a model with possibly non-Gaussian and heteroscedastic disturbances. We allow for the number of regressors to be larger than the sample size. To make informative inference feasible, we require the model to be approximately sparse; that is, we require that the effect of confounding factors can be controlled for up to a small approximation error by including a relatively small number of variables whose identities are unknown. The latter condition makes it possible to estimate the treatment effect by selecting approximately the right set of regressors. We develop a novel estimation and uniformly valid inference method for the treatment effect in this setting, called the “post-double-selection” method. The main attractive feature of our method is that it allows for imperfect selection of the controls and provides confidence intervals that are valid uniformly across a large class of models. In contrast, standard post-model selection estimators fail to provide uniform inference even in simple cases with a small, fixed number of controls. Thus, our method resolves the problem of uniform inference after model selection for a large, interesting class of models. We also present a generalization of our method to a fully heterogeneous model with a binary treatment variable. We illustrate the use of the developed methods with numerical simulations and an application that considers the effect of abortion on crime rates.
825 citations
Posted Content•
TL;DR: This work develops a novel estimation and uniformly valid inference method for the treatment effect in this setting, called the "post-double-selection" method, which resolves the problem of uniform inference after model selection for a large, interesting class of models.
Abstract: We propose robust methods for inference on the effect of a treatment variable on a scalar outcome in the presence of very many controls. Our setting is a partially linear model with possibly non-Gaussian and heteroscedastic disturbances. Our analysis allows the number of controls to be much larger than the sample size. To make informative inference feasible, we require the model to be approximately sparse; that is, we require that the effect of confounding factors can be controlled for up to a small approximation error by conditioning on a relatively small number of controls whose identities are unknown. The latter condition makes it possible to estimate the treatment effect by selecting approximately the right set of controls. We develop a novel estimation and uniformly valid inference method for the treatment effect in this setting, called the "post-double-selection" method. Our results apply to Lasso-type methods used for covariate selection as well as to any other model selection method that is able to find a sparse model with good approximation properties.
The main attractive feature of our method is that it allows for imperfect selection of the controls and provides confidence intervals that are valid uniformly across a large class of models. In contrast, standard post-model selection estimators fail to provide uniform inference even in simple cases with a small, fixed number of controls. Thus our method resolves the problem of uniform inference after model selection for a large, interesting class of models. We illustrate the use of the developed methods with numerical simulations and an application to the effect of abortion on crime rates.
568 citations
Cites methods from "On Non-parametric Testing, the Unif..."
...In that regard, our contribution is in the spirit and builds upon the classical contribution by Romano (2004) on the uniform validity of t-tests for the univariate mean....
[...]
...Our approach to uniformity analysis is most similar to that of Romano (2004), Theorem 4....
[...]
TL;DR: In this article, robust inference on average treatment effects following model selection is studied. Butler et al. construct confidence intervals using a doubly-robust estimator that are robust to model selection errors and prove their uniform validity over a large class of models that allows for multivalued treatments with heterogeneous effects and selection amongst (possibly) more covariates than observations.
300 citations
Cites background or methods from "On Non-parametric Testing, the Unif..."
...This method of proving uniformity follows Belloni, Chernozhukov, and Hansen (2014) and Romano (2004), and is distinct from the approach of Andrews and Guggenberger (2009)....
[...]
...This assumption is a form of “ignorability” coined by Rosenbaum and Rubin (1983). This model allows arbitrary treatment effect heterogeneity in observables, but not unobservables....
[...]
Book•
01 Jan 2006TL;DR: In this paper, the authors consider a class of models where the objective function is defined by a pop ulation objective function Q(6, P) for 6 e 0.9 and seek random sets that contain this objective function with at least some prespecified probability asymptotically.
Abstract: distribution of the observed data. The class of models we consider is defined by a pop ulation objective function Q(6, P) for 6 e 0. The point of departure from the classical extremum estimation framework is that it is not assumed that Q(6,P) has a unique minimizer in the parameter space 0. The goal may be either to draw inferences about some unknown point in the set of minimizers of the population objective function or to draw inferences about the set of minimizers itself. In this paper, the object of interest is 0o(P) = argminee(9 Q(0, P), and so we seek random sets that contain this set with at least some prespecified probability asymptotically. We also consider situations where the object of interest is the image of 0o(P) under a known function. Random sets that satisfy the desired coverage property are constructed under weak assumptions. Condi tions are provided under which the confidence regions are asymptotically valid not only pointwise in P, but also uniformly in P. We illustrate the use of our methods with an empirical study of the impact of top-coding outcomes on inferences about the parame ters of a linear regression. Finally, a modest simulation study sheds some light on the finite-sample behavior of our procedure.
254 citations
Cites background from "On Non-parametric Testing, the Unif..."
...Romano (2004) extended this nonexistence result to a number of other problems....
[...]
Book•
01 Jan 2006TL;DR: In this article, the authors consider the problem of inference for partially identified econometric models, where the objective function is defined by a population objective function Q ( θ, P ) for θ ∈ Θ.
Abstract: This paper considers the problem of inference for partially identified econometric models. The class of models studied are defined by a population objective function Q ( θ , P ) for θ ∈ Θ . The second argument indicates the dependence of the objective function on P , the distribution of the observed data. Unlike the classical extremum estimation framework, it is not assumed that Q ( θ , P ) has a unique minimizer in the parameter space Θ . The goal may be either to draw inferences about some unknown point in the set of minimizers of the population objective function or to draw inferences about the set of minimizers itself. In this paper, the object of interest is some unknown point θ ∈ Θ 0 ( P ) , where Θ 0 ( P ) = arg min θ ∈ Θ Q ( θ , P ) , and so we seek random sets that contain each θ ∈ Θ 0 ( P ) with at least some prespecified probability asymptotically. We also consider situations where the object of interest is the image of some point θ ∈ Θ 0 ( P ) under a known function. Computationally intensive, yet feasible procedures for constructing random sets satisfying the desired coverage property under weak assumptions are provided. We also provide conditions under which the confidence regions are uniformly consistent in level.
227 citations
Cites background from "On Non-parametric Testing, the Unif..."
...Romano (2004) extends this nonexistence result to a number of other problems....
[...]
References
More filters
11,456 citations
Book•
01 Jan 1986
TL;DR: In this article, the authors present a framework for the analysis of decision spaces in decision theory, including the space of risk functions and the spaces of decision processes, and propose a method for measuring the suitability of a decision space.
Abstract: 1 Experiments-Decision Spaces.- 1 Introduction.- 2 Vector Lattices-L-Spaces-Transitions.- 3 Experiments-Decision Procedures.- 4 A Basic Density Theorem.- 5 Building Experiments from Other Ones.- 6 Representations-Markov Kernels.- 2 Some Results from Decision Theory: Deficiencies.- 1 Introduction.- 2 Characterization of the Spaces of Risk Functions: Minimax Theorem.- 3 Deficiencies Distances.- 4 The Form of Bayes Risks-Choquet Lattices.- 3 Likelihood Ratios and Conical Measures.- 1 Introduction.- 2 Homogeneous Functions of Measures.- 3 Deficiencies for Binary Experiments: Isometries.- 4 Weak Convergence of Experiments.- 5 Boundedly Complete Experiments.- 6 Convolutions: Hellinger Transforms.- 7 The Blackwell-Sherman-Stein Theorem.- 4 Some Basic Inequalities.- 1 Introduction.- 2 Hellinger Distances: L1-Norm.- 3 Approximation Properties for Likelihood Ratios.- 4 Inequalities for Conditional Distributions.- 5 Sufficiency and Insufficiency.- 1 Introduction.- 2 Projections and Conditional Expectations.- 3 Equivalent Definitions for Sufficiency.- 4 Insufficiency.- 5 Estimating Conditional Distributions.- 6 Domination, Compactness, Contiguity.- 1 Introduction.- 2 Definitions and Elementary Relations.- 3 Contiguity.- 4 Strong Compactness and a Result of D. Lindae.- 7 Some Limit Theorems.- 1 Introduction.- 2 Convergence in Distribution or in Probability.- 3 Distinguished Sequences of Statistics.- 4 Lower-Semicontinuity for Spaces of Risk Functions.- 5 A Result on Asymptotic Admissibility.- 8 Invariance Properties.- 1 Introduction.- 2 The Markov-Kakutani Fixed Point Theorem.- 3 A Lifting Theorem and Some Applications.- 4 Automatic Invariance of Limits.- 5 Invariant Exponential Families.- 6 The Hunt-Stein Theorem and Related Results.- 9 Infinitely Divisible, Gaussian, and Poisson Experiments.- 1 Introduction.- 2 Infinite Divisibility.- 3 Gaussian Experiments.- 4 Poisson Experiments.- 5 A Central Limit Theorem.- 10 Asymptotically Gaussian Experiments: Local Theory.- 1 Introduction.- 2 Convergence to a Gaussian Shift Experiment.- 3 A Framework which Arises in Many Applications.- 4 Weak Convergence of Distributions.- 5 An Application of a Martingale Limit Theorem.- 6 Asymptotic Admissibility and Minimaxity.- 11 Asymptotic Normality-Global.- 1 Introduction.- 2 Preliminary Explanations.- 3 Construction of Centering Variables.- 4 Definitions Relative to Quadratic Approximations.- 5 Asymptotic Properties of the Centerings $$\hat{Z}$$.- 6 The Asymptotically Gaussian Case.- 7 Some Particular Cases.- 8 Reduction to the Gaussian Case by Small Distortions.- 9 The Standard Tests and Confidence Sets.- 10 Minimum ?2 and Relatives.- 12 Posterior Distributions and Bayes Solutions.- 1 Introduction.- 2 Inequalities on Conditional Distributions.- 3 Asymptotic behavior of Bayes Procedures.- 4 Approximately Gaussian Posterior Distributions.- 13 An Approximation Theorem for Certain Sequential Experiments.- 1 Introduction.- 2 Notations and Assumptions.- 3 Basic Auxiliary Lemmas.- 4 Reduction Theorems.- 5 Remarks on Possible Applications.- 14 Approximation by Exponential Families.- 1 Introduction.- 2 A Lemma on Approximate Sufficiency.- 3 Homogeneous Experiments of Finite Rank.- 4 Approximation by Experiments of Finite Rank.- 5 Construction of Distinguished Sequences of Estimates.- 15 Sums of Independent Random Variables.- 1 Introduction.- 2 Concentration Inequalities.- 3 Compactness and Shift-Compactness.- 4 Poisson Exponentials and Approximation Theorems.- 5 Limit Theorems and Related Results.- 6 Sums of Independent Stochastic Processes.- 16 Independent Observations.- 1 Introduction.- 2 Limiting Distributions for Likelihood Ratios.- 3 Conditions for Asymptotic Normality.- 4 Tests and Distances.- 5 Estimates for Finite Dimensional Parameter Spaces.- 6 The Risk of Formal Bayes Procedures.- 7 Empirical Measures and Cumulatives.- 8 Empirical Measures on Vapnik-?ervonenkis Classes.- 17 Independent Identically Distributed Observations.- 1 Introduction.- 2 Hilbert Spaces Around a Point.- 3 A Special Role for $$\sqrt{n}$$: Differentiability in Quadratic Mean.- 4 Asymptotic Normality for Rates Other than $$\sqrt{n}$$.- 5 Existence of Consistent Estimates.- 6 Estimates Converging at the $$\sqrt{n}$$-Rate.- 7 The Behavior of Posterior Distributions.- 8 Maximum Likelihood.- 9 Some Cases where the Number of Observations Is Random.- Appendix: Results from Classical Analysis.- 1 The Language of Set Theory.- 2 Topological Spaces.- 3 Uniform Spaces.- 4 Metric Spaces.- 5 Spaces of Functions.- 6 Vector Spaces.- 7 Vector Lattices.- 8 Vector Lattices Arising from Experiments.- 9 Lattices of Numerical Functions.- 10 Extensions of Positive Linear Functions.- 11 Smooth Linear Functionals.- 12 Derivatives and Tangents.
1,427 citations
"On Non-parametric Testing, the Unif..." refers background or methods in this paper
...The total variation metric s is well known and has many statistical uses (see Le Cam, 1986, Chapter 4)....
[...]
...efficient method can be obtained by inverting the confidence interval of Romano & Wolf (2000). As an alternative to the family of distributions supported on a compact set, it is interesting to study the behaviour of the t-test under the assumption of symmetry (so again the Bahadur–Savage result does not hold)....
[...]
...Both Hoeffding & Wolfowitz (1958) and LeCam & Schwartz (1960) consider the general question of consistency of tests and estimates, which result in abstract topological criteria....
[...]
...The total variation metric s is well known and has many statistical uses (see Le Cam, 1986, Chapter 4). Donoho (1988) effectively employed its use in proving the impossibility of constructing useful two-sided confidence intervals for certain functionals of a density....
[...]
...Both Hoeffding & Wolfowitz (1958) and LeCam & Schwartz (1960) consider the general question of consistency of tests and estimates, which result in abstract topological criteria. In the context of confidence interval construction, some results are provided by Gleser & Hwang (1987), Donoho (1988) and Pfanzagl (1998)....
[...]
01 Jan 1956
329 citations
"On Non-parametric Testing, the Unif..." refers background or methods in this paper
...The idea of using a parametric submodel to obtain efficiency results in non-parametric models dates back to Stein (1956)....
[...]
...The problem of demonstrating equivalence (or bioequivalence) can be formulated as testing H0:|h(F)| ‡ versus H1:|h(F)| < ; see Wellek (2003). In particular, if h(F) is the mean of F, and F is the set of all distributions with finite mean (or more generally satisfying the conditions of theorem 3), then (6) and (7) hold....
[...]
...The idea of using a parametric submodel to obtain efficiency results in non-parametric models dates back to Stein (1956). So, introduce the parametric submodel with density phðxÞ 1⁄4 expðhx CðhÞÞ ð19Þ...
[...]