scispace - formally typeset
Search or ask a question
Journal ArticleDOI

A Simple Sequentially Rejective Multiple Test Procedure

TL;DR: In this paper, a simple and widely accepted multiple test procedure of the sequentially rejective type is presented, i.e. hypotheses are rejected one at a time until no further rejections can be done.
Abstract: This paper presents a simple and widely ap- plicable multiple test procedure of the sequentially rejective type, i.e. hypotheses are rejected one at a tine until no further rejections can be done. It is shown that the test has a prescribed level of significance protection against error of the first kind for any combination of true hypotheses. The power properties of the test and a number of possible applications are also discussed.

Content maybe subject to copyright    Report

Citations
More filters
Journal ArticleDOI
TL;DR: In this paper, a different approach to problems of multiple significance testing is presented, which calls for controlling the expected proportion of falsely rejected hypotheses -the false discovery rate, which is equivalent to the FWER when all hypotheses are true but is smaller otherwise.
Abstract: SUMMARY The common approach to the multiplicity problem calls for controlling the familywise error rate (FWER). This approach, though, has faults, and we point out a few. A different approach to problems of multiple significance testing is presented. It calls for controlling the expected proportion of falsely rejected hypotheses -the false discovery rate. This error rate is equivalent to the FWER when all hypotheses are true but is smaller otherwise. Therefore, in problems where the control of the false discovery rate rather than that of the FWER is desired, there is potential for a gain in power. A simple sequential Bonferronitype procedure is proved to control the false discovery rate for independent test statistics, and a simulation study shows that the gain in power is substantial. The use of the new procedure and the appropriateness of the criterion are illustrated with examples.

83,420 citations


Cites background or methods from "A Simple Sequentially Rejective Mul..."

  • ...The first difficulty has been partially addressed by the recent line of research advancing Bonferroni-type procedures, which use the observed individual p-values, while remaining faithful to FWER control: Simes (1986), Hommel (1988), Hochberg (1988) and Rom (1990)....

    [...]

  • ...Whereas Simes (1986) showed that his procedure controls the FWER under the intersection null hypothesis, Hommel (1988) showed that the extended procedure for inference on individual hypotheses does not control the FWER in the strong sense: for some configuration of the false null hypotheses, the probability of an erroneous rejection is greater than a....

    [...]

Journal ArticleDOI
TL;DR: EELAB as mentioned in this paper is a toolbox and graphic user interface for processing collections of single-trial and/or averaged EEG data of any number of channels, including EEG data, channel and event information importing, data visualization (scrolling, scalp map and dipole model plotting, plus multi-trial ERP-image plots), preprocessing (including artifact rejection, filtering, epoch selection, and averaging), Independent Component Analysis (ICA) and time/frequency decomposition including channel and component cross-coherence supported by bootstrap statistical methods based on data resampling.

17,362 citations


Cites background from "A Simple Sequentially Rejective Mul..."

  • ...To compensate for multiple comparisons, significance thresholds may need to be decreased (e.g. Bonferroni, 1950; Holm, 1979)....

    [...]

Journal ArticleDOI
TL;DR: Technique non parametrique pour la signification statistique de tables de tests utilisees dans les etudes sur l'evolution notamment.
Abstract: Technique non parametrique pour la signification statistique de tables de tests utilisees dans les etudes sur l'evolution notamment

14,666 citations


Cites background or methods from "A Simple Sequentially Rejective Mul..."

  • ...When no such alternative is available, the sequential Bonferroni is a useful choice, since much ofthe true conservativeness of the standard Bonferroni test is eliminated (Holm, 1979)....

    [...]

  • ...I then describe a nonparametric technique, originally proposed by Holm (1979), to eliminate this bias....

    [...]

  • ...To increase power in detecting more than one false H o.i , Holm (1979) introduced the sequential Bonferroni technique....

    [...]

  • ...The standard Bonferroni test has substantially reduced power in detecting more than one false HO•i (see Holm, 1979)....

    [...]

  • ...A non parametric technique that can be used in virtually all applications is the sequential Bonferroni test, originally developed by Holm (1979)....

    [...]

Journal ArticleDOI
TL;DR: In this paper, the heterotrait-monotrait ratio of correlations is used to assess discriminant validity in variance-based structural equation modeling. But it does not reliably detect the lack of validity in common research situations.
Abstract: Discriminant validity assessment has become a generally accepted prerequisite for analyzing relationships between latent variables. For variance-based structural equation modeling, such as partial least squares, the Fornell-Larcker criterion and the examination of cross-loadings are the dominant approaches for evaluating discriminant validity. By means of a simulation study, we show that these approaches do not reliably detect the lack of discriminant validity in common research situations. We therefore propose an alternative approach, based on the multitrait-multimethod matrix, to assess discriminant validity: the heterotrait-monotrait ratio of correlations. We demonstrate its superior performance by means of a Monte Carlo simulation study, in which we compare the new approach to the Fornell-Larcker criterion and the assessment of (partial) cross-loadings. Finally, we provide guidelines on how to handle discriminant validity issues in variance-based structural equation modeling.

12,855 citations


Cites methods from "A Simple Sequentially Rejective Mul..."

  • ...Furthermore, Bonferroni is a rather conservative approach to maintain the familywise error rate at a predefined level (Hochberg 1988; Holm 1979)....

    [...]

Journal Article
TL;DR: A set of simple, yet safe and robust non-parametric tests for statistical comparisons of classifiers is recommended: the Wilcoxon signed ranks test for comparison of two classifiers and the Friedman test with the corresponding post-hoc tests for comparisons of more classifiers over multiple data sets.
Abstract: While methods for comparing two learning algorithms on a single data set have been scrutinized for quite some time already, the issue of statistical tests for comparisons of more algorithms on multiple data sets, which is even more essential to typical machine learning studies, has been all but ignored. This article reviews the current practice and then theoretically and empirically examines several suitable tests. Based on that, we recommend a set of simple, yet safe and robust non-parametric tests for statistical comparisons of classifiers: the Wilcoxon signed ranks test for comparison of two classifiers and the Friedman test with the corresponding post-hoc tests for comparison of more classifiers over multiple data sets. Results of the latter can also be neatly presented with the newly introduced CD (critical difference) diagrams.

10,306 citations


Cites methods from "A Simple Sequentially Rejective Mul..."

  • ...The simplest such methods are due to Holm (1979) and Hochberg (1988)....

    [...]

References
More filters
Journal ArticleDOI
TL;DR: In this article, a multiple comparison procedure for comparing several treatments with a control is presented, which is based on the Multiple Comparison Procedure for Comparing Several Treatments with a Control (MCPC).
Abstract: (1955). A Multiple Comparison Procedure for Comparing Several Treatments with a Control. Journal of the American Statistical Association: Vol. 50, No. 272, pp. 1096-1121.

5,756 citations


"A Simple Sequentially Rejective Mul..." refers methods in this paper

  • ...For the case of normally distributed observations the multiple test procedure suggested by Dunnett (1955) is commonly used....

    [...]

  • ...(1976) have proposed a closed test procedure, which is a refinement of the Dunnett procedure and which is more powerful. Their procedure is equivalent to a sequentially rejective procedure presented in Holm (1977). The sequentially rejective Bonferroni test can also be used in this situation although it is of course less powerful than the refined Dunnett test....

    [...]

Book
01 Jan 1966
TL;DR: In this article, the authors presented a case of two means regression method for the family error rate, which was used to estimate the probability of a family having a nonzero family error.
Abstract: 1 Introduction.- 1 Case of two means.- 2 Error rates.- 2.1 Probability of a nonzero family error rate.- 2.2 Expected family error rate.- 2.3 Allocation of error.- 3 Basic techniques.- 3.1 Repeated normal statistics.- 3.2 Maximum modulus (Tukey).- 3.3 Bonferroni normal statistics.- 3.4 ?2 projections (Scheffe).- 3.5 Allocation.- 3.6 Multiple modulus tests (Duncan).- 3.7 Least significant difference test (Fisher).- 4 p-mean significance levels.- 5 Families.- 2 Normal Univariate Techniques.- 1 Studentized range (Tukey).- 1.1 Method.- 1.2 Applications.- 1.3 Comparison.- 1.4 Derivation.- 1.5 Distributions and tables.- 2 F projections (Scheffe)48.- 2.1 Method.- 2.2 Applications.- 2.3 Comparison.- 2.4 Derivation.- 2.5 Distributions and tables.- 3 Bonferroni t statistics.- 3.1 Method.- 3.2 Applications.- 3.3 Comparison.- 3.4 Derivation.- 3.5 Distributions and tables.- 4 Studentized maximum modulus.- 4.1 Method.- 4.2 Applications.- 4.3 Comparison.- 4.4 Derivation.- 4.5 Distributions and tables.- 5 Many-one t statistics76.- 5.1 Method.- 5.2 Applications.- 5.3 Comparison.- 5.4 Derivation.- 5.5 Distributions and tables.- 6 Multiple range tests (Duncan).- 6.1 Method.- 6.2 Applications.- 6.3 Comparison.- 6.4 Derivation.- 6.5 Distributions and tables.- 7 Least significant difference test (Fisher).- 7.1 Method.- 7.2 Applications.- 7.3 Comparison.- 7.4 Derivation.- 7.5 Distributions and tables.- 8 Other techniques.- 8.1 Tukey's gap-straggler-variance test.- 8.2 Shortcut methods.- 8.3 Multiple F tests.- 8.4 Two-sample confidence intervals of predetermined length.- 8.5 An improved Bonferroni inequality.- 9 Power.- 10 Robustness.- 3 Regression Techniques.- 1 Regression surface confidence bands.- 1.1 Method.- 1.2 Comparison.- 1.3 Derivation.- 2 Prediction.- 2.1 Method.- 2.2 Comparison.- 2.3 Derivation.- 3 Discrimination.- 3.1 Method.- 3.2 Comparison.- 3.3 Derivation.- 4 Other techniques.- 4.1 Linear confidence bands.- 4.2 Tolerance intervals.- 4.3 Unlimited discrimination intervals.- 4 Nonparametric Techniques.- 1 Many-one sign statistics (Steel).- 1.1 Method.- 1.2 Applications.- 1.3 Comparison.- 1.4 Derivation.- 1.5 Distributions and tables.- 2 k-sample sign statistics.- 2.1 Method.- 2.2 Applications.- 2.3 Comparison.- 2.4 Derivation.- 2.5 Distributions and tables.- 3 Many-one rank statistics (Steel).- 3.1 Method.- 3.2 Applications.- 3.3 Comparison.- 3.4 Derivation.- 3.5 Distributions and tables.- 4 k-sample rank statistics.- 4.1 Method.- 4.2 Applications.- 4.3 Comparison.- 4.4 Derivation.- 4.5 Distributions and tables.- 5 Signed-rank statistics.- 6 Kruskal-Wallis rank statistics (Nemenyi).- 6.1 Method.- 6.2 Applications.- 6.3 Comparison.- 6.4 Derivation.- 6.5 Distributions and tables.- 7 Friedman rank statistics (Nemenyi).- 7.1 Method.- 7.2 Applications.- 7.3 Comparison.- 7.4 Derivation.- 7.5 Distributions and tables.- 8 Other techniques.- 8.1 Permutation tests.- 8.2 Median tests (Nemenyi).- 8.3 Kolmogorov-Smirnov statistics.- 5 Multivariate Techniques.- 1 Single population covariance scalar unknown.- 1.1 Method.- 1.2 Applications.- 1.3 Comparison.- 1.4 Derivation.- 1.5 Distributions and tables.- 2 Single population covariance matrix unknown.- 2.1 Method.- 2.2 Applications.- 2.3 Comparison.- 2.4 Derivation.- 2.5 Distributions and tables.- 3 k populations covariance matrix unknown.- 3.1 Method.- 3.2 Applications.- 3.3 Comparison.- 3.4 Derivation.- 3.5 Distributions and tables.- 4 Other techniques.- 4.1 Variances known covariances unknown.- 4.2 Variance-covariance intervals.- 4.3 Two-sample confidence intervals of predetermined length.- 6 Miscellaneous Techniques.- 1 Outlier detection.- 2 Multinomial populations.- 2.1 Single population.- 2.2 Several populations.- 2.3 Cross-product ratios.- 2.4 Logistic response curves.- 3 Equality of variances.- 4 Periodogram analysis.- 5 Alternative approaches: selection, ranking, slippage.- A Strong Law For The Expected Error Rate.- B TABLES.- I Percentage points of the studentized range.- II Percentage points of the Bonferroni t statistic.- III Percentage points of the studentized maximum modulus.- IV Percentage points of the many-one t statistics.- V Percentage points of the Duncan multiple range test.- VI Percentage points of the many-one sign statistics.- VIII Percentage points of the many-one rank statistics.- IX Percentage points of the k-sample rank statistics.- Developments in Multiple Comparisons 1966-).- 3.5 Allocation.- 3.6 Multiple modulus tests (Duncan).- 3.7 Least significant difference test (Fisher).- 4 p-mean significance levels.- 5 Families.- 2 Normal Univariate Techniques.- 1 Studentized range (Tukey).- 1.1 Method.- 1.2 Applications.- 1.3 Comparison.- 1.4 Derivation.- 1.5 Distributions and tables.- 2 F projections (Scheffe)48.- 2.1 Method.- 2.2 Applications.- 2.3 Comparison.- 2.4 Derivation.- 2.5 Distributions and tables.- 3 Bonferroni t statistics.- 3.1 Method.- 3.2 Applications.- 3.3 Comparison.- 3.4 Derivation.- 3.5 Distributions and tables.- 4 Studentized maximum modulus.- 4.1 Method.- 4.2 Applications.- 4.3 Comparison.- 4.4 Derivation.- 4.5 Distributions and tables.- 5 Many-one t statistics76.- 5.1 Method.- 5.2 Applications.- 5.3 Comparison.- 5.4 Derivation.- 5.5 Distributions and tables.- 6 Multiple range tests (Duncan).- 6.1 Method.- 6.2 Applications.- 6.3 Comparison.- 6.4 Derivation.- 6.5 Distributions and tables.- 7 Least significant difference test (Fisher).- 7.1 Method.- 7.2 Applications.- 7.3 Comparison.- 7.4 Derivation.- 7.5 Distributions and tables.- 8 Other techniques.- 8.1 Tukey's gap-straggler-variance test.- 8.2 Shortcut methods.- 8.3 Multiple F tests.- 8.4 Two-sample confidence intervals of predetermined length.- 8.5 An improved Bonferroni inequality.- 9 Power.- 10 Robustness.- 3 Regression Techniques.- 1 Regression surface confidence bands.- 1.1 Method.- 1.2 Comparison.- 1.3 Derivation.- 2 Prediction.- 2.1 Method.- 2.2 Comparison.- 2.3 Derivation.- 3 Discrimination.- 3.1 Method.- 3.2 Comparison.- 3.3 Derivation.- 4 Other techniques.- 4.1 Linear confidence bands.- 4.2 Tolerance intervals.- 4.3 Unlimited discrimination intervals.- 4 Nonparametric Techniques.- 1 Many-one sign statistics (Steel).- 1.1 Method.- 1.2 Applications.- 1.3 Comparison.- 1.4 Derivation.- 1.5 Distributions and tables.- 2 k-sample sign statistics.- 2.1 Method.- 2.2 Applications.- 2.3 Comparison.- 2.4 Derivation.- 2.5 Distributions and tables.- 3 Many-one rank statistics (Steel).- 3.1 Method.- 3.2 Applications.- 3.3 Comparison.- 3.4 Derivation.- 3.5 Distributions and tables.- 4 k-sample rank statistics.- 4.1 Method.- 4.2 Applications.- 4.3 Comparison.- 4.4 Derivation.- 4.5 Distributions and tables.- 5 Signed-rank statistics.- 6 Kruskal-Wallis rank statistics (Nemenyi).- 6.1 Method.- 6.2 Applications.- 6.3 Comparison.- 6.4 Derivation.- 6.5 Distributions and tables.- 7 Friedman rank statistics (Nemenyi).- 7.1 Method.- 7.2 Applications.- 7.3 Comparison.- 7.4 Derivation.- 7.5 Distributions and tables.- 8 Other techniques.- 8.1 Permutation tests.- 8.2 Median tests (Nemenyi).- 8.3 Kolmogorov-Smirnov statistics.- 5 Multivariate Techniques.- 1 Single population covariance scalar unknown.- 1.1 Method.- 1.2 Applications.- 1.3 Comparison.- 1.4 Derivation.- 1.5 Distributions and tables.- 2 Single population covariance matrix unknown.- 2.1 Method.- 2.2 Applications.- 2.3 Comparison.- 2.4 Derivation.- 2.5 Distributions and tables.- 3 k populations covariance matrix unknown.- 3.1 Method.- 3.2 Applications.- 3.3 Comparison.- 3.4 Derivation.- 3.5 Distributions and tables.- 4 Other techniques.- 4.1 Variances known covariances unknown.- 4.2 Variance-covariance intervals.- 4.3 Two-sample confidence intervals of predetermined length.- 6 Miscellaneous Techniques.- 1 Outlier detection.- 2 Multinomial populations.- 2.1 Single population.- 2.2 Several populations.- 2.3 Cross-product ratios.- 2.4 Logistic response curves.- 3 Equality of variances.- 4 Periodogram analysis.- 5 Alternative approaches: selection, ranking, slippage.- A Strong Law For The Expected Error Rate.- B TABLES.- I Percentage points of the studentized range.- II Percentage points of the Bonferroni t statistic.- III Percentage points of the studentized maximum modulus.- IV Percentage points of the many-one t statistics.- V Percentage points of the Duncan multiple range test.- VI Percentage points of the many-one sign statistics.- VIII Percentage points of the many-one rank statistics.- IX Percentage points of the k-sample rank statistics.- Developments in Multiple Comparisons 1966-1976.- 1 Introduction.- 2 Papers of special interest.- 2.1 Probability inequalities.- 2.2 Methods for unbalanced ANOVA.- 2.3 Conditional confidence levels.- 2.4 Empirical Bayes approach.- 2.5 Confidence bands in regression.- 3 References.- 4 Bibliography 1966-1976.- 4.1 Survey articles.- 4.2 Probability inequalities.- 4.3 Tables.- 4.4 Normal multifactor methods.- 4.5 Regression.- 4.6 Categorical data.- 4.7 Nonparametric techniques.- 4.8 Multivariate methods.- 4.9 Miscellaneous.- 4.10 Pre-1966 articles missed in [6].- 4.11 Late additions.- 5 List of journals scanned.- Addendum New Table of the Studentized Maximum Modulus.- Table IIIA Percentage points of the studentized maximum modulus.- Author Index.

4,763 citations

Book
01 Jan 1974

637 citations


"A Simple Sequentially Rejective Mul..." refers methods in this paper

  • ...Marcus et al. (1976) have proposed a closed test procedure, which is a refinement of the Dunnett procedure and which is more powerful....

    [...]

Journal ArticleDOI
TL;DR: The Simultaneous Test Procedure (STP) as discussed by the authors is a generalization of the notion of multiple comparisons for means, which was originally proposed by Aitchison and Roy.
Abstract: 1. Introduction and summary. When a hypothesis is tested by a significanice test and is not rejected, it is generally agreed that all hypotheses implied by that hypothesis (its "components") must also be considered as non-rejected. However, when the hypothesis is rejected the question remains as to which components may also be rejected. Various writers have given attention to this question and have proposed a variety of multiple comparisons methods based either on tests of each one of the components or on simultaneous confidence bounds on parametric functions related to the various hypotheses. An approach to such methods, apparently originally due to Tukey [27], is to test each component hypothesis by comparing its statistic with the a level critical value of the statistic for the overall hypothesis. This is called a Simultaneous Test Procedure (STP for short) as all hypotheses may be tested simultaneously and without reference to one another. An STP involves no stepwise testing of the kind employed by some other methods of multiple comparisons for means, in which subsets are tested for equality only if they are contained in sets which have already been found significant. (See 13], [4], [10], [18]). A general formalization of STP's is attempted in this paper. Section 2 introduces the requisite concepts of families of hypotheses and the implication relations between them, as well as the monotonicity relations between the related statistics. Section 3 defines STP's and shows conditions for coherence and consonance of their decisions, these properties being that hypothesis implication relations are preserved in the decisions of the STP. Section 4 discusses comparison of various STP's for the same hypotheses and shows the advantages of the unionintersection type of statistics and of reducing the family of hypotheses tested as much as possible. Section 5 translates all these results to simultaneous confidence statements after introducing the definitions necessary to allow such translation. The analogy between simultaneous test and confidence methods is of special importance as it brings a wide spectrum of methods within this framework, most of which was originally formulated in confidence region terms. This covers the original work of Tukey [27] and Scheff6 [25] and continues with that of Roy and his associates [21] and most recently Krishnaiah [12], [13]. A general discussion of this confidence approach has been given by Aitchison [1) since the first draft of the present paper. In view of the close analogies pointed out in Section 5, it is

365 citations


"A Simple Sequentially Rejective Mul..." refers background in this paper

  • ...are minimal in the sense of Gabriel (1969). This means that if o,, con, ....

    [...]