An exploration of the use of simple statistics to measure consensus and stability in Delphi studies
TL;DR: Following the original use of Delphi in social sciences, Delphi is suggested to be an effective way to gain and measure group consensus in healthcare to ensure maximum validity of results in Delphi methodology for improved evidence of consensual decision-making.
Abstract: The criteria for stopping Delphi studies are often subjective. This study aimed to examine whether consensus and stability in the Delphi process can be ascertained by descriptive evaluation of trends in participants' views. A three round email-based Delphi required participants (n = 12) to verify their level of agreement with 8 statements, write comments on each if they considered it necessary and rank the statements for importance. Each statement was analysed quantitatively by the percentage of agreement ratings, importance rankings and the amount of comments made for each statement, and qualitatively using thematic analysis. Importance rankings between rounds were compared by calculating Kappa values to observe trends in how the process impacts on subject's views. Evolution of consensus was shown by increase in agreement percentages, convergence of range with standard deviations of importance ratings, and a decrease in the number of comments made. Stability was demonstrated by a trend of increasing Kappa values. Following the original use of Delphi in social sciences, Delphi is suggested to be an effective way to gain and measure group consensus in healthcare. However, the proposed analytical process should be followed to ensure maximum validity of results in Delphi methodology for improved evidence of consensual decision-making.
Citations
More filters
14 Jan 2011
TL;DR: The Delphi Technique has been used extensively in nursing and health care research as discussed by the authors, where the Delphi is used to analyze data from a Delphi and report the results of the analysis.
Abstract: Preface. Acknowledgements. 1 The Delphi Technique. 2 Debates, Criticisms and Limitations of the Delphi. 3 Applications of the Delphi in Nursing and Health Research. 4 How to Get Started with the Delphi Technique. 5 Conducting the Research Using the Delphi Technique. 6 Analysing Data from a Delphi and Reporting Results. 7 Reliability and Validity. 8 Ethical Considerations. 10 A Modified Delphi Case Study. 11 e-Delphi Case Study. Annotated Bibliography. References. Index.
887 citations
TL;DR: Nominal group technique was used in the context of four focus groups involving clinical experts from the emergency department (ED) and obstetric and midwifery areas of a busy regional hospital to assess the triage and management of pregnant women in the ED.
Abstract: This paper aims to demonstrate the versatility and application of nominal group technique as a method for generating priority information. Nominal group technique was used in the context of four focus groups involving clinical experts from the emergency department (ED) and obstetric and midwifery areas of a busy regional hospital to assess the triage and management of pregnant women in the ED. The data generated were used to create a priority list of discussion triggers for the subsequent Participatory Action Research Group. This technique proved to be a productive and efficient data collection method which produced information in a hierarchy of perceived importance and identified real world problems. This information was vital in initiating the participatory action research project and is recommended as an effective and reliable data collection method, especially when undertaking research with clinical experts.
452 citations
TL;DR: The aim of this article was to reflect on Delphi methodology and provide guidance useful to researchers in integrative medicine and to help future researchers.
338 citations
Cites background or methods from "An exploration of the use of simple..."
...A study exploring different methods to measure consensus and stability in Delphi studies advised using a combination of statistics to reduce subjectivity and ensure validity of results [28]....
[...]
...Additionally, this approach means stability of response cannot be assessed across all statements as data is needed from both rounds [28,29]....
[...]
...Kappa statistics have been advocated with high or increasing kappa values demonstrating stability of individuals' views within the group [28]....
[...]
...The Chisquared (X) has been used to test for stability, but there has been advice against using it in Delphi studies as it determines “the independence of the rounds from responses found in them” and not the stability of response between rounds [28]....
[...]
...Consensuscaneitherbeusedto determineifagreementexists [20] or as a stopping guideline [28]....
[...]
TL;DR: An overview of the fields of application for Delphi techniques in health sciences in this map is provided and criteria for evaluating the quality of their execution and reporting also appear to be necessary.
Abstract: Objectives: In health sciences, the Delphi technique is primarily used by researchers when the available knowledge is incomplete or subject to uncertainty and other methods that provide higher levels of evidence cannot be used. The aim is to collect expert-based judgments and often to use them to identify consensus. In this map, we provide an overview of the fields of application for Delphi techniques in health sciences in this map and discuss the processes used and the quality of the findings. We use systematic reviews of Delphi techniques for the map, summarize their findings and examine them from a methodological perspective. Methods: Twelve systematic reviews of Delphi techniques from different sectors of the health sciences were identified and systematically analyzed. Results: The 12 systematic reviews show, that Delphi studies are typically carried out in two to three rounds with a deliberately selected panel of experts. A large number of modifications to the Delphi technique have now been developed. Significant weaknesses exist in the quality of the reporting. Conclusion: Based on the results, there is a need for clarification with regard to the methodological approaches of Delphi techniques, also with respect to any modification. Criteria for evaluating the quality of their execution and reporting also appear to be necessary. However, it should be noted that we cannot make any statements about the quality of execution of the Delphi studies but rather our results are exclusively based on the reported findings of the systematic reviews.
278 citations
TL;DR: A practical, systematic approach to the design and delivery of a Delphi survey, where the Delphi administrator has a range of choice options and discussion of the pros and cons of each option is provided in order that the overall design and Delivery of a particularDelphi survey is both well-founded and defensible.
165 citations
References
More filters
Book•
15 Jun 2006
TL;DR: Practical Statistics for Medical Research is a problem-based text for medical researchers, medical students, and others in the medical arena who need to use statistics but have no specialized mathematics background.
Abstract: Most medical researchers, whether clinical or non-clinical, receive some background in statistics as undergraduates. However, it is most often brief, a long time ago, and largely forgotten by the time it is needed. Furthermore, many introductory texts fall short of adequately explaining the underlying concepts of statistics, and often are divorced from the reality of conducting and assessing medical research.
Practical Statistics for Medical Research is a problem-based text for medical researchers, medical students, and others in the medical arena who need to use statistics but have no specialized mathematics background.
The author draws on twenty years of experience as a consulting medical statistician to provide clear explanations to key statistical concepts, with a firm emphasis on practical aspects of designing and analyzing medical research. The text gives special attention to the presentation and interpretation of results and the many real problems that arise in medical research
17,322 citations
Book•
01 Jan 1981TL;DR: In this paper, the basic theory of Maximum Likelihood Estimation (MLE) is used to detect a difference between two different proportions of a given proportion in a single proportion.
Abstract: Preface.Preface to the Second Edition.Preface to the First Edition.1. An Introduction to Applied Probability.2. Statistical Inference for a Single Proportion.3. Assessing Significance in a Fourfold Table.4. Determining Sample Sizes Needed to Detect a Difference Between Two Proportions.5. How to Randomize.6. Comparative Studies: Cross-Sectional, Naturalistic, or Multinomial Sampling.7. Comparative Studies: Prospective and Retrospective Sampling.8. Randomized Controlled Trials.9. The Comparison of Proportions from Several Independent Samples.10. Combining Evidence from Fourfold Tables.11. Logistic Regression.12. Poisson Regression.13. Analysis of Data from Matched Samples.14. Regression Models for Matched Samples.15. Analysis of Correlated Binary Data.16. Missing Data.17. Misclassification Errors: Effects, Control, and Adjustment.18. The Measurement of Interrater Agreement.19. The Standardization of Rates.Appendix A. Numerical Tables.Appendix B. The Basic Theory of Maximum Likelihood Estimation.Appendix C. Answers to Selected Problems.Author Index.Subject Index.
16,435 citations
10,442 citations
9,528 citations
"An exploration of the use of simple..." refers background in this paper
...Therefore Excel spreadsheets and handwritten crosstabs were used based on the descriptions by Armitage et al. [27] and interpretation by Anthony [28], Table 3 (see Fliess [29] for a full mathematical explanation and justification of the Kappa validity)....
[...]
...[27] and interpretation by Anthony [28], Table 3 (see Fliess [29] for a full mathematical explanation and justification of the Kappa validity)....
[...]
Book•
01 Jan 1975
TL;DR: The present model clarifies some of the conceptual problems associated with cross-impact analysis, and supplies a relatively sound basis for revising probability estimates in the limited case where interactions can be approximated by relative probabilities.
Abstract: Cross-impact analysis is a method for revising estimated probabilities of future events in terms of estimated interactions among those events. This Report presents an elementary cross-impact model where the cross-impacts are formulated as relative probabilities. Conditions are derived for the consistency of the matrix of relative probabilities of n events. An extension also provides a necessary condition for the vector of absolute probabilities to be consistent with the relative probability matrix. An averaging technique is formulated for resolving inconsistencies in the matrix, and a nearest-point computation derived for resolving inconsistencies between the set of absolute probabilities and the matrix. Although elementary, the present model clarifies some of the conceptual problems associated with cross-impact analysis, and supplies a relatively sound basis for revising probability estimates in the limited case where interactions can be approximated by relative probabilities.
5,102 citations