Publication bias in clinical research
TL;DR: The presence of publication bias in a cohort of clinical research studies is confirmed and it is suggested that conclusions based only on a review of published data should be interpreted cautiously, especially for observational studies.
About: This article is published in The Lancet.The article was published on 1991-04-13. It has received 2800 citations till now. The article focuses on the topics: Publication bias & Observational study.
Citations
More filters
TL;DR: Funnel plots, plots of the trials' effect estimates against sample size, are skewed and asymmetrical in the presence of publication bias and other biases Funnel plot asymmetry, measured by regression analysis, predicts discordance of results when meta-analyses are compared with single large trials.
Abstract: Objective: Funnel plots (plots of effect estimates against sample size) may be useful to detect bias in meta-analyses that were later contradicted by large trials. We examined whether a simple test of asymmetry of funnel plots predicts discordance of results when meta-analyses are compared to large trials, and we assessed the prevalence of bias in published meta-analyses. Design: Medline search to identify pairs consisting of a meta-analysis and a single large trial (concordance of results was assumed if effects were in the same direction and the meta-analytic estimate was within 30% of the trial); analysis of funnel plots from 37 meta-analyses identified from a hand search of four leading general medicine journals 1993-6 and 38 meta-analyses from the second 1996 issue of the Cochrane Database of Systematic Reviews . Main outcome measure: Degree of funnel plot asymmetry as measured by the intercept from regression of standard normal deviates against precision. Results: In the eight pairs of meta-analysis and large trial that were identified (five from cardiovascular medicine, one from diabetic medicine, one from geriatric medicine, one from perinatal medicine) there were four concordant and four discordant pairs. In all cases discordance was due to meta-analyses showing larger effects. Funnel plot asymmetry was present in three out of four discordant pairs but in none of concordant pairs. In 14 (38%) journal meta-analyses and 5 (13%) Cochrane reviews, funnel plot asymmetry indicated that there was bias. Conclusions: A simple analysis of funnel plots provides a useful test for the likely presence of bias in meta-analyses, but as the capacity to detect bias will be limited when meta-analyses are based on a limited number of small trials the results from such analyses should be treated with considerable caution. Key messages Systematic reviews of randomised trials are the best strategy for appraising evidence; however, the findings of some meta-analyses were later contradicted by large trials Funnel plots, plots of the trials9 effect estimates against sample size, are skewed and asymmetrical in the presence of publication bias and other biases Funnel plot asymmetry, measured by regression analysis, predicts discordance of results when meta-analyses are compared with single large trials Funnel plot asymmetry was found in 38% of meta-analyses published in leading general medicine journals and in 13% of reviews from the Cochrane Database of Systematic Reviews Critical examination of systematic reviews for publication and related biases should be considered a routine procedure
37,989 citations
TL;DR: In this review the usual methods applied in systematic reviews and meta-analyses are outlined, and the most common procedures for combining studies with binary outcomes are described, illustrating how they can be done using Stata commands.
31,656 citations
TL;DR: A checklist contains specifications for reporting of meta-analyses of observational studies in epidemiology, including background, search strategy, methods, results, discussion, and conclusion should improve the usefulness ofMeta-an analyses for authors, reviewers, editors, readers, and decision makers.
Abstract: ObjectiveBecause of the pressure for timely, informed decisions in public health
and clinical practice and the explosion of information in the scientific literature,
research results must be synthesized. Meta-analyses are increasingly used
to address this problem, and they often evaluate observational studies. A
workshop was held in Atlanta, Ga, in April 1997, to examine the reporting
of meta-analyses of observational studies and to make recommendations to aid
authors, reviewers, editors, and readers.ParticipantsTwenty-seven participants were selected by a steering committee, based
on expertise in clinical practice, trials, statistics, epidemiology, social
sciences, and biomedical editing. Deliberations of the workshop were open
to other interested scientists. Funding for this activity was provided by
the Centers for Disease Control and Prevention.EvidenceWe conducted a systematic review of the published literature on the
conduct and reporting of meta-analyses in observational studies using MEDLINE,
Educational Research Information Center (ERIC), PsycLIT, and the Current Index
to Statistics. We also examined reference lists of the 32 studies retrieved
and contacted experts in the field. Participants were assigned to small-group
discussions on the subjects of bias, searching and abstracting, heterogeneity,
study categorization, and statistical methods.Consensus ProcessFrom the material presented at the workshop, the authors developed a
checklist summarizing recommendations for reporting meta-analyses of observational
studies. The checklist and supporting evidence were circulated to all conference
attendees and additional experts. All suggestions for revisions were addressed.ConclusionsThe proposed checklist contains specifications for reporting of meta-analyses
of observational studies in epidemiology, including background, search strategy,
methods, results, discussion, and conclusion. Use of the checklist should
improve the usefulness of meta-analyses for authors, reviewers, editors, readers,
and decision makers. An evaluation plan is suggested and research areas are
explored.
17,663 citations
Additional excerpts
...In addition, methodologic issues related specifically to metaanalysis, such as publication bias, could have particular impact when combining results of observational studies.(44,47) Despite these challenges, metaanalyses of observational studies continue to be one of the few methods for assessing efficacy and effectiveness and are being published in increasing numbers....
[...]
TL;DR: In this paper, an adjusted rank correlation test is proposed as a technique for identifying publication bias in a meta-analysis, and its operating characteristics are evaluated via simulations, and the test statistic is a direct statistical analogue of the popular funnel-graph.
Abstract: An adjusted rank correlation test is proposed as a technique for identifying publication bias in a meta-analysis, and its operating characteristics are evaluated via simulations. The test statistic is a direct statistical analogue of the popular "funnel-graph." The number of component studies in the meta-analysis, the nature of the selection mechanism, the range of variances of the effect size estimates, and the true underlying effect size are all observed to be influential in determining the power of the test. The test is fairly powerful for large meta-analyses with 75 component studies, but has only moderate power for meta-analyses with 25 component studies. However, in many of the configurations in which there is low power, there is also relatively little bias in the summary effect size estimate. Nonetheless, the test must be interpreted with caution in small meta-analyses. In particular, bias cannot be ruled out if the test is not significant. The proposed technique has potential utility as an exploratory tool for meta-analysts, as a formal procedure to complement the funnel-graph.
13,373 citations
TL;DR: In this paper, a rank-based data augmentation technique is proposed for estimating the number of missing studies that might exist in a meta-analysis and the effect that these studies might have had on its outcome.
Abstract: We study recently developed nonparametric methods for estimating the number of missing studies that might exist in a meta-analysis and the effect that these studies might have had on its outcome. These are simple rank-based data augmentation techniques, which formalize the use of funnel plots. We show that they provide effective and relatively powerful tests for evaluating the existence of such publication bias. After adjusting for missing studies, we find that the point estimate of the overall effect size is approximately correct and coverage of the effect size confidence intervals is substantially improved, in many cases recovering the nominal confidence levels entirely. We illustrate the trim and fill method on existing meta-analyses of studies in clinical trials and psychometrics.
9,163 citations
References
More filters
TL;DR: Quantitative procedures for computing the tolerance for filed and future null results are reported and illustrated, and the implications are discussed.
Abstract: For any given research area, one cannot tell how many studies have been conducted but never reported. The extreme view of the "file drawer problem" is that journals are filled with the 5% of the studies that show Type I errors, while the file drawers are filled with the 95% of the studies that show nonsignificant results. Quantitative procedures for computing the tolerance for filed and future null results are reported and illustrated, and the implications are discussed. (15 ref) (PsycINFO Database Record (c) 2012 APA, all rights reserved)
7,159 citations
TL;DR: Some methods of calculating confidence intervals for means and differences between means are given, with similar information for proportions, and the paper also gives suggestions for graphical display.
Abstract: Overemphasis on hypothesis testing--and the use of P values to dichotomise significant or non-significant results--has detracted from more useful approaches to interpreting study results, such as estimation and confidence intervals. In medical studies investigators are usually interested in determining the size of difference of a measured outcome between groups, rather than a simple indication of whether or not it is statistically significant. Confidence intervals present a range of values, on the basis of the sample data, in which the population value for such a difference may lie. Some methods of calculating confidence intervals for means and differences between means are given, with similar information for proportions. The paper also gives suggestions for graphical display. Confidence intervals, if appropriate to the type of study, should be used for major findings in both the main text of a paper and its abstract.
1,841 citations
TL;DR: Concern for the probability of missing an important therapeutic improvement because of small sample sizes deserves more attention in the planning of clinical trials.
Abstract: Seventy-one "negative" randomized control trials were re-examined to determine if the investigators had studied large enough samples to give a high probability (greater than 0.90) of detecting a 25 per cent and 50 per cent therapeutic improvement in the response. Sixty-seven of the trials had a greater than 10 per cent risk of missing a true 25 per cent therapeutic improvement, and with the same risk, 50 of the trials could have missed a 50 per cent improvement. Estimates of 90 per cent confidence intervals for the true improvement in each trial showed that in 57 of these "negative" trials, a potential 25 per cent improvement was possible, and 34 of the trials showed a potential 50 per cent improvement. Many of the therapies labeled as "no different from control" in trials using inadequate samples have not received a fair test. Concern for the probability of missing an important therapeutic improvement because of small sample sizes deserves more attention in the planning of clinical trials.
1,532 citations
TL;DR: There is some evidence that, in fields where statistical tests of significance are commonly used, research which yields nonsignificant results is not published as mentioned in this paper, and such research being unknown to other investigators may be repeated independently until eventually by chance a significant result occurs, an "error of the first kind" and is published.
Abstract: There is some evidence that in fields where statistical tests of significance are commonly used, research which yields nonsignificant results is not published. Such research being unknown to other investigators may be repeated independently until eventually by chance a significant result occurs—an “error of the first kind”—and is published. Significant results published in these fields are seldom verified by independent replication. The possibility thus arises that the literature of such a field consists in substantial part of false conclusions resulting from errors of the first kind in statistical tests of significance. * The author wishes to express his thanks to Sir Ronald Fisher whose discussion on related topics stimulated this research in the first place, and to Leo Katz, Oliver Lacey, Enders Robinson, and Paul Siegel for reading and criticizing earlier drafts of this manuscript.
958 citations
TL;DR: In this paper, the authors review the available research, discuss alternative suggestions for conducting unbiased meta-analysis and suggest some scientific policy measures which could improve the quality of published data in the long term.
Abstract: Publication bias, the phenomenon in which studies with positive results are more likely to be published than studies with negative results, is a serious problem in the interpretation of scientific research. Various hypothetical models have been studied which clarify the potential for bias and highlight characteristics which make a study especially susceptible to bias. Empirical investigations have supported the hypothesis that bias exists and have provided a quantitative assessment of the magnitude of the problem. The use of meta‐analysis as a research tool has focused attention on the issue, since naive methodologies in this area are especially susceptible to bias. In this paper we review the available research, discuss alternative suggestions for conducting unbiased meta‐analysis and suggest some scientific policy measures which could improve the quality of published data in the long term.
744 citations