It is well known that statistical power calculations can be valuable in planning an experiment. There is also a large literature advocating that power calculations be made whenever one performs a statistical test of a hypothesis and one obtains a statistically nonsignificant result. Advocates of such post-experiment power calculations claim the calculations should be used to aid in the interpretation of the experimental results. This approach, which appears in various forms, is fundamentally flawed. We document that the problem is extensive and present arguments to demonstrate the flaw in the logic.

/pdf/the-abuse-of-power-the-pervasive-fallacy-of-power-2pgs8gsae8.pdf

The Abuse of Power: The Pervasive Fallacy of Power Calculations for Data Analysis

background We examined whether a fixed dose of both isosorbide dinitrate and hydralazine provides additional benefit in blacks with advanced heart failure, a subgroup previously noted to have a favorable response to this therapy. methods A total of 1050 black patients who had New York Heart Association class III or IV heart failure with dilated ventricles were randomly assigned to receive a fixed dose of isosorbide dinitrate plus hydralazine or placebo in addition to standard therapy for heart failure. The primary end point was a composite score made up of weighted values for death from any cause, a first hospitalization for heart failure, and change in the quality of life. results The study was terminated early owing to a significantly higher mortality rate in the placebo group than in the group given isosorbide dinitrate plus hydralazine (10.2 percent vs. 6.2 percent, P = 0.02). The mean primary composite score was significantly better in the group given isosorbide dinitrate plus hydralazine than in the placebo group (i0.1±1.9 vs. i0.5±2.0, P=0.01; range of possible values, –6 to +2), as were its individual components (43 percent reduction in the rate of death from any cause [hazard ratio, 0.57; P=0.01] 33 percent relative reduction in the rate of first hospitalization for heart failure [16.4 percent vs. 22.4 percent, P=0.001], and an improvement in the quality of life [change in score, i5.6±20.6 vs. i2.7±21.2, with lower scores indicating better quality of life; P=0.02; range of possible values, 0 to 105]). conclusions The addition of a fixed dose of isosorbide dinitrate plus hydralazine to standard therapy for heart failure including neurohormonal blockers is efficacious and increases survival among black patients with advanced heart failure.

Combination of isosorbide dinitrate and hydralazine in blacks with heart failure.

Introduction General Concepts Error rates and general concepts Construction methods Methods based on Bonferroni's inequality Methods based on Simes' inequality Multiple Comparisons in Parametric Models General linear models Extensions to general parametric models The multcomp package Applications Multiple comparisons with a control All pairwise comparisons Dose response analyses Variable selection in regression models Simultaneous confidence bands Multiple comparisons under heteroscedasticity Multiple comparisons in logistic regression models Multiple comparisons in survival models Multiple comparisons in mixed-effects models Further Topics Resampling-based multiple comparison procedures Group sequential and adaptive designs Combining multiple comparisons with modeling Bibliography Index

Multiple Comparisons Using R

Because scientists tend to report only studies (publication bias) or analyses (p-hacking) that "work," readers must ask, "Are these effects true, or do they merely reflect selective reporting?" We introduce p-curve as a way to answer this question. P-curve is the distribution of statistically significant p values for a set of studies (ps < .05). Because only true effects are expected to generate right-skewed p-curves-containing more low (.01s) than high (.04s) significant p values--only right-skewed p--curves are diagnostic of evidential value. By telling us whether we can rule out selective reporting as the sole explanation for a set of findings, p-curve offers a solution to the age-old inferential problems caused by file-drawers of failed studies and analyses.

/pdf/p-curve-a-key-to-the-file-drawer-1237pz4gnm.pdf

P-curve: a key to the file-drawer.

Because scientists tend to report only studies (publication bias) or analyses (p-hacking) that “work”, readers must ask, “Are these effects true, or do they merely reflect selective reporting?” We introduce p-curve as a way to answer this question. P-curve is the distribution of statistically significant p-values for a set of studies (ps

P-Curve: A Key to the File Drawer

In group sequential clinical trials, sample size reestimation can be a complicated issue when it allows for change of sample size to be influenced by an observed sample path. Our simulation studies show that increasing sample size based on an interim estimate of the treatment difference can substantially inflate the probability of type I error in most practical situations. A new group sequential test procedure is developed by modifying the weights used in the traditional repeated significance two-sample mean test. The new test has the type I error probability preserved at the target level and can provide a substantial gain in power with the increase of sample size. Generalization of the new procedure is discussed.

Modification of Sample Size in Group Sequential Clinical Trials

The P-value is a random variable derived from the distribution of the test statistic used to analyze a data set and to test a null hypothesis. Under the null hypothesis, the P-value based on a continuous test statistic has a uniform distribution over the interval [0, 1], regardless of the sample size of the experiment. In contrast, the distribution of the P-value under the alternative hypothesis is a function of both sample size and the true value or range of true values of the tested parameter. The characteristics, such as mean and percentiles, of the P-value distribution can give valuable insight into how the P-value behaves for a variety of parameter values and sample sizes. Potential applications of the P-value distribution under the alternative hypothesis to the design, analysis, and interpretation of results of clinical trials are considered.

/pdf/the-behavior-of-the-p-value-when-the-alternative-hypothesis-18c0tk4cvi.pdf

The behavior of the p-value when the alternative hypothesis is true

We consider the problem of studying several dose combinations of two drugs for a therapeutic endpoint in a multilevel factorial clinical trial. Two test statistics are constructed to test whether there exists at least one dose combination that is more effective than its component doses. Their distributions involve nuisance parameters quantifying the mean differences among the doses of the two component drugs. It is shown that their power functions achieve maxima as all the nuisance parameters approach infinity in absolute value. The significance levels of the two tests are derived and two alpha-level tests are proposed. Tables are given to provide the alpha-level critical values for these tests and to gain insights into their power performances.

H. M. J. Hung

Papers

Modification of Sample Size in Group Sequential Clinical Trials

The behavior of the p-value when the alternative hypothesis is true

Testing for the existence of a desirable dose combination.