There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias. In this essay, I discuss the implications of these problems for the conduct and interpretation of research.

Why most published research findings are false

A focus on novel, confirmatory, and statistically significant results leads to substantial bias in the scientific literature. One type of bias, known as “p-hacking,” occurs when researchers collect or select data or statistical analyses until nonsignificant results become significant. Here, we use text-mining to demonstrate that p-hacking is widespread throughout science. We then illustrate how one can test for p-hacking when performing a meta-analysis and show that, while p-hacking is probably common, its effect seems to be weak relative to the real effect sizes being measured. This result suggests that p-hacking probably does not drastically alter scientific consensuses drawn from meta-analyses.

/pdf/the-extent-and-consequences-of-p-hacking-in-science-hxdoh51lek.pdf

The Extent and Consequences of P-Hacking in Science

In this paper we focus on robust linear optimization problems with uncertainty regions defined by φ-divergences for example, chi-squared, Hellinger, Kullback--Leibler. We show how uncertainty regions based on φ-divergences arise in a natural way as confidence sets if the uncertain parameters contain elements of a probability vector. Such problems frequently occur in, for example, optimization problems in inventory control or finance that involve terms containing moments of random variables, expected utility, etc. We show that the robust counterpart of a linear optimization problem with φ-divergence uncertainty is tractable for most of the choices of φ typically considered in the literature. We extend the results to problems that are nonlinear in the optimization variables. Several applications, including an asset pricing example and a numerical multi-item newsvendor example, illustrate the relevance of the proposed approach.

This paper was accepted by Gerard P. Cachon, optimization.

/pdf/robust-solutions-of-optimization-problems-affected-by-1xadn6mpyh.pdf

Robust Solutions of Optimization Problems Affected by Uncertain Probabilities

More than 50 years ago, John Tukey called for a reformation of academic statistics. In “The Future of Data Analysis,” he pointed to the existence of an as-yet unrecognized science, whose subject of...

https://courses.csail.mit.edu/18.337/2015/docs/50YearsDataScience.pdf

50 Years of Data Science

The last decade witnessed an explosion in the availability of data for operations research applications. Motivated by this growing availability, we propose a novel schema for utilizing data to design uncertainty sets for robust optimization using statistical hypothesis tests. The approach is flexible and widely applicable, and robust optimization problems built from our new sets are computationally tractable, both theoretically and practically. Furthermore, optimal solutions to these problems enjoy a strong, finite-sample probabilistic guarantee whenever the constraints and objective function are concave in the uncertainty. We describe concrete procedures for choosing an appropriate set for a given application and applying our approach to multiple uncertain constraints. Computational evidence in portfolio management and queueing confirm that our data-driven sets significantly outperform traditional robust optimization techniques whenever data are available.

/pdf/data-driven-robust-optimization-1ixumlcymg.pdf

Data-driven robust optimization

The accuracy of published medical research is critical for scientists, physicians and patients who rely on these results. However, the fundamental belief in the medical literature was called into serious question by a paper suggesting that most published medical research is false. Here we adapt estimation methods from the genomics community to the problem of estimating the rate of false discoveries in the medical literature using reported $P$-values as the data. We then collect $P$-values from the abstracts of all 77 430 papers published in The Lancet, The Journal of the American Medical Association, The New England Journal of Medicine, The British Medical Journal, and The American Journal of Epidemiology between 2000 and 2010. Among these papers, we found 5322 reported $P$-values. We estimate that the overall rate of false discoveries among reported results is 14% (s.d. 1%), contrary to previous claims. We also found that there is no a significant increase in the estimated rate of reported false discovery results over time (0.5% more false positives (FP) per year, $P = 0.18$) or with respect to journal submissions (0.5% more FP per 100 submissions, $P = 0.12$). Statistical analysis must allow for false discoveries in order to make claims on the basis of noisy data. But our analysis suggests that the medical literature remains a reliable record of scientific progress.

/pdf/an-estimate-of-the-science-wise-false-discovery-rate-and-2ein83wsqw.pdf

An estimate of the science-wise false discovery rate and application to the top medical literature

A unified family of goodness-of-fit tests based on $\phi$-divergences is introduced and studied. The new family of test statistics $S_n(s)$ includes both the supremum version of the Anderson--Darling statistic and the test statistic of Berk and Jones [Z. Wahrsch. Verw. Gebiete 47 (1979) 47--59] as special cases ($s=2$ and $s=1$, resp.). We also introduce integral versions of the new statistics. We show that the asymptotic null distribution theory of Berk and Jones [Z. Wahrsch. Verw. Gebiete 47 (1979) 47--59] and Wellner and Koltchinskii [High Dimensional Probability III (2003) 321--332. Birkhauser, Basel] for the Berk--Jones statistic applies to the whole family of statistics $S_n(s)$ with $s\in[-1,2]$. On the side of power behavior, we study the test statistics under fixed alternatives and give extensions of the ``Poisson boundary'' phenomena noted by Berk and Jones for their statistic. We also extend the results of Donoho and Jin [Ann. Statist. 32 (2004) 962--994] by showing that all our new tests for $s\in[-1,2]$ have the same ``optimal detection boundary'' for normal shift mixture alternatives as Tukey's ``higher-criticism'' statistic and the Berk--Jones statistic.

Goodness-of-fit tests via phi-divergences

A unified family of goodness-of-fit tests based on O-divergences is introduced and studied. The new family of test statistics S n (s) includes both the supremum version of the Anderson-Darling statistic and the test statistic of Berk and Jones [Z. Wahrsch. Verw. Gebiete 47 (1979) 47-59] as special cases (s = 2 and s = 1, resp.). We also introduce integral versions of the new statistics. We show that the asymptotic null distribution theory of Berk and Jones [Z. Wahrsch. Verw. Gebiete 47 (1979) 47-59] and Wellner and Koltchinskii [High Dimensional Probability III (2003) 321-332. Birkhauser, Basel] for the Berk-Jones statistic applies to the whole family of statistics S n (s) with s ∈ [-1, 2]. On the side of power behavior, we study the test statistics under fixed alternatives and give extensions of the "Poisson boundary" phenomena noted by Berk and Jones for their statistic. We also extend the results of Donoho and Jin [Ann. Statist. 32 (2004) 962-994] by showing that all our new tests for s ∈ [- 1, 2] have the same "optimal detection boundary" for normal shift mixture alternatives as Tukey's "higher-criticism" statistic and the Berk-Jones statistic.

/pdf/goodness-of-fit-tests-via-phi-divergences-42wfv7f92w.pdf

Graph theoretic analysis of structural connectivity across the spectrum of Alzheimer's disease: The importance of graph creation methods.

There has been an increasing concern in both the scientific and lay communities that most published medical findings are false. But what does it mean to be false? Here we describe the range of definitions of false discoveries in the scientific literature. We summarize the philosophical, statistical, and experimental evidence for each type of false discovery. We discuss common underpinning problems with the scientific and data analytic practices and point to tools and behaviors that can be implemented to reduce the problems with published scientific results.

Leah Jager

Papers

An estimate of the science-wise false discovery rate and application to the top medical literature

Goodness-of-fit tests via phi-divergences

Goodness-of-fit tests via phi-divergences

Graph theoretic analysis of structural connectivity across the spectrum of Alzheimer's disease: The importance of graph creation methods.

Is Most Published Research Really False