(1955). A Multiple Comparison Procedure for Comparing Several Treatments with a Control. Journal of the American Statistical Association: Vol. 50, No. 272, pp. 1096-1121.

https://www.jstor.org/stable/pdfplus/2281208.pdf

A Multiple Comparison Procedure for Comparing Several Treatments with a Control

https://www-sop.inria.fr/asclepios/Publications/Tom.Vercauteren/Eval14NonlinearRegistration-Klein-NeuroImage09.pdf

Evaluation of 14 nonlinear deformation algorithms applied to human brain MRI registration.

INTRODUCTION ..................................................................................................................... 561 ORGANIZING CO CEPTS ..................................................................................................... 564 Primary Hypotheses, Closure, Hierarchical Sets, and Minimal Hypotheses ...................... 564 Families ................................................................................................................................ 565 Type 1 Error Control ............................................................................................................ 566 Power ................................................................................................................................... 567 P-Values and Adjusted P-Values ......................................................................................... 568 Closed Test Procedures ....................................................................................................... 569 METHODS BA ED ON ORDERED P-VALUES ................................................................... 569 Methods Based on the First-Order Bonferroni Inequality .................................................. 569 Methods Based on the Simes Equality ................................................................................. 570 Modifications for Logically Related Hypotheses ................................................................. 571 Methods Controlling the False Discovery Rate ................................................................... 572 COMPARING NORMALLY DISTRIBUTED M ANS ......................................................... 573 OTHER ISSUES ........................................................................................................................ 575 Tests vs Confidence I tervals ............................................................................................... 575 Directional vs Nondirectional Inference ............................................................................. 576 Robustness ............................................................................................................................ 577 Others ........................................................................ ....................................................... 578 CONCLUSION .......................................................................................................................... 580

Multiple Hypothesis Testing

The Analysis of Variance.

The stochastic multi-armed bandit model is a simple abstraction that has proven useful in many different contexts in statistics and machine learning. Whereas the achievable limit in terms of regret minimization is now well known, our aim is to contribute to a better understanding of the performance in terms of identifying the m best arms. We introduce generic notions of complexity for the two dominant frameworks considered in the literature: fixed-budget and fixed-confidence settings. In the fixed-confidence setting, we provide the first known distribution-dependent lower bound on the complexity that involves information-theoretic quantities and holds when m ≥ 1 under general assumptions. In the specific case of two armed-bandits, we derive refined lower bounds in both the fixedcon fidence and fixed-budget settings, along with matching algorithms for Gaussian and Bernoulli bandit models. These results show in particular that the complexity of the fixed-budget setting may be smaller than the complexity of the fixed-confidence setting, contradicting the familiar behavior observed when testing fully specified alternatives. In addition, we also provide improved sequential stopping rules that have guaranteed error probabilities and shorter average running times. The proofs rely on two technical results that are of independent interest: a deviation lemma for self-normalized sums (Lemma 7) and a novel change of measure inequality for bandit models (Lemma 1).

/pdf/on-the-complexity-of-best-arm-identification-in-multi-armed-7ed20kt1lj.pdf

On the complexity of best-arm identification in multi-armed bandit models

This paper is concerned with a single-sample multiple decision procedure for ranking means of normal populations with known variances. Problems which conventionally are handled by the analysis of variance (Model I) which tests the hypothesis that $k$ means are equal are reformulated as multiple decision procedures involving rankings. It is shown how to design experiments so that useful statements can be made concerning these rankings on the basis of a predetermined number of independent observations taken from each population. The number of observations required is determined by the desired probability of a correct ranking when certain differences between population means are specified.

https://projecteuclid.org/download/pdf_1/euclid.aoms/1177728845

A Single-Sample Multiple Decision Procedure for Ranking Means of Normal Populations with known Variances

Sequential Identification and Ranking Procedures.

A multiple decision approach to the problem of ranking populations according to their population means has been formulated by Bechhofer (1954). A single-sample solution to this problem was presented by him for the case of normal populations with known variances. In the present paper the case of normal populations with unknown equal variances is considered. A two-sample procedure is proposed as a solution to the latter problem; this procedure is of the type used by Stein (1945) for obtaining a test of Student's hypothesis with power independent of the variance. Tables are available which enable the experimenter to apply this procedure with little computational effort to the ranking of two or three populations with unknown but equal variances. The same tables can be used when the variance ratios are arbitrary and known. Graphic comparisons of the expected sample size are made for the single-sample and two-sample procedures.

A tow-sample multiple decision procedure for ranking means of normal populations with a common unknown variance

The problem of selecting the multinomial event which has the highest probability is formulated as a multiple-decision selection problem. Before experimentation starts the experimenter must specify two constants $(\theta^{\ast}, P^{\ast})$ which are incorporated into the requirement: "The probability of a correct selection is to be equal to or greater than $P^{\ast}$ whenever the true (but unknown) ratio of the largest to the second largest of the population probabilities is equal to or greater than $\theta^{\ast}$." A single-sample procedure which meets the requirement is proposed. The heart of the procedure is the proper choice of $N$, the number of trials. Two methods of determining $N$ are described: the first is exact and is to be used when $N$ is small; the second is approximate and is to be used when $N$ is large. Tables and sample calculations are provided.

/pdf/a-single-sample-multiple-decision-procedure-for-selecting-abu33lxbdp.pdf

Robert E. Bechhofer

Papers

A Single-Sample Multiple Decision Procedure for Ranking Means of Normal Populations with known Variances

Sequential Identification and Ranking Procedures.

A tow-sample multiple decision procedure for ranking means of normal populations with a common unknown variance

A Single-Sample Multiple-Decision Procedure for Selecting the Multinomial Event Which Has the Highest Probability

Sequential identification and ranking procedures : with special reference to Koopman-Darmois populations