Features and uses of high-fidelity medical simulations that lead to effective learning: a BEME systematic review
TL;DR: While research in this field needs improvement in terms of rigor and quality, high-fidelity medical simulations are educationally effective and simulation-based education complements medical education in patient care settings.
Abstract: SUMMARY Review date: 1969 to 2003, 34 years. Background and context: Simulations are now in widespread use in medical education and medical personnel evaluation. Outcomes research on the use and effectiveness of simulation technology in medical education is scattered, inconsistent and varies widely in methodological rigor and substantive focus. Objectives: Review and synthesize existing evidence in educational science that addresses the question, ‘What are the features and uses of high-fidelity medical simulations that lead to most effective learning?’. Search strategy: The search covered five literature databases (ERIC, MEDLINE, PsycINFO, Web of Science and Timelit) and employed 91 single search terms and concepts and their Boolean combinations. Hand searching, Internet searches and attention to the ‘grey literature’ were also used. The aim was to perform the most thorough literature search possible of peer-reviewed publications and reports in the unpublished literature that have been judged for academic quality. Inclusion and exclusion criteria: Four screening criteria were used to reduce the initial pool of 670 journal articles to a focused set of 109 studies: (a) elimination of review articles in favor of empirical studies; (b) use of a simulator as an educational assessment or intervention with learner outcomes measured quantitatively; (c) comparative research, either experimental or quasi-experimental; and (d) research that involves simulation as an educational intervention. Data extraction: Data were extracted systematically from the 109 eligible journal articles by independent coders. Each coder used a standardized data extraction protocol. Data synthesis: Qualitative data synthesis and tabular presentation of research methods and outcomes were used. Heterogeneity of research designs, educational interventions, outcome measures and timeframe precluded data synthesis using meta-analysis. Headline results: Coding accuracy for features of the journal articles is high. The extant quality of the published research is generally weak. The weight of the best available evidence suggests that high-fidelity medical simulations facilitate learning under the right conditions. These include the following:
Citations
More filters
TL;DR: This article reviews and critically evaluates historical and contemporary research on simulation‐based medical education (SBME) and presents and discusses 12 features and best practices that teachers should know in order to use medical simulation technology to maximum educational benefit.
Abstract: Objectives This article reviews and critically evaluates historical and contemporary research on simulation-based medical education (SBME). It also presents and discusses 12 features and best practices of SBME that teachers should know in order to use medical simulation technology to maximum educational benefit.
Methods This qualitative synthesis of SBME research and scholarship was carried out in two stages. Firstly, we summarised the results of three SBME research reviews covering the years 1969–2003. Secondly, we performed a selective, critical review of SBME research and scholarship published during 2003–2009.
Results The historical and contemporary research synthesis is reported to inform the medical education community about 12 features and best practices of SBME: (i) feedback; (ii) deliberate practice; (iii) curriculum integration; (iv) outcome measurement; (v) simulation fidelity; (vi) skill acquisition and maintenance; (vii) mastery learning; (viii) transfer to practice; (ix) team training; (x) high-stakes testing; (xi) instructor training, and (xii) educational and professional context. Each of these is discussed in the light of available evidence. The scientific quality of contemporary SBME research is much improved compared with the historical record.
Conclusions Development of and research into SBME have grown and matured over the past 40 years on substantive and methodological grounds. We believe the impact and educational utility of SBME are likely to increase in the future. More thematic programmes of research are needed. Simulation-based medical education is a complex service intervention that needs to be planned and practised with attention to organisational contexts.
Medical Education 2010: 44: 50–63
1,459 citations
TL;DR: In comparison with no intervention, technology-enhanced simulation training in health professions education is consistently associated with large effects for outcomes of knowledge, skills, and behaviors and moderate effects for patient-related outcomes.
Abstract: Context Although technology-enhanced simulation has widespread appeal, its effectiveness remains uncertain. A comprehensive synthesis of evidence may inform the use of simulation in health professions education. Objective To summarize the outcomes of technology-enhanced simulation training for health professions learners in comparison with no intervention. Data Source Systematic search of MEDLINE, EMBASE, CINAHL, ERIC, PsychINFO, Scopus, key journals, and previous review bibliographies through May 2011. Study Selection Original research in any language evaluating simulation compared with no intervention for training practicing and student physicians, nurses, dentists, and other health care professionals. Data Extraction Reviewers working in duplicate evaluated quality and abstracted information on learners, instructional design (curricular integration, distributing training over multiple days, feedback, mastery learning, and repetitive practice), and outcomes. We coded skills (performance in a test setting) separately for time, process, and product measures, and similarly classified patient care behaviors. Data Synthesis From a pool of 10 903 articles, we identified 609 eligible studies enrolling 35 226 trainees. Of these, 137 were randomized studies, 67 were nonrandomized studies with 2 or more groups, and 405 used a single-group pretest-posttest design. We pooled effect sizes using random effects. Heterogeneity was large (I2>50%) in all main analyses. In comparison with no intervention, pooled effect sizes were 1.20 (95% CI, 1.04-1.35) for knowledge outcomes (n = 118 studies), 1.14 (95% CI, 1.03-1.25) for time skills (n = 210), 1.09 (95% CI, 1.03-1.16) for process skills (n = 426), 1.18 (95% CI, 0.98-1.37) for product skills (n = 54), 0.79 (95% CI, 0.47-1.10) for time behaviors (n = 20), 0.81 (95% CI, 0.66-0.96) for other behaviors (n = 50), and 0.50 (95% CI, 0.34-0.66) for direct effects on patients (n = 32). Subgroup analyses revealed no consistent statistically significant interactions between simulation training and instructional design features or study quality. Conclusion In comparison with no intervention, technology-enhanced simulation training in health professions education is consistently associated with large effects for outcomes of knowledge, skills, and behaviors and moderate effects for patient-related outcomes.
1,420 citations
Cites background or methods from "Features and uses of high-fidelity ..."
...Criterion B was fulfilled if (1) a randomized study concealed allocation or (2) an observational study controlled for another baseline learner characteristic....
[...]
...We sought to answer 2 questions: (1) To what extent are simulation technologies for training health care professionals associated with improved outcomes in comparison with no intervention? and (2) How do outcomes vary for different simulation instructional designs? Based on the strength of the theoretical foundations and currency in the field, we selected 5 instructional design features(2,9) (curricular integration, distributed practice, feedback, mastery learning, and range of difficulty) for subgroup analyses (see eBox for definitions; available at http://www ....
[...]
...Beyond descriptive analysis (2) 560 (91....
[...]
...When authors reported multiple measures of a single outcome (eg, multiple measures of efficiency), we selected in decreasing order of priority (1) the authordefined primary outcome; (2) a global or summary measure of effect; (3) the most clinically relevant measure; or (4) the mean of the measures reported....
[...]
...cComparability of cohorts criterion A was fulfilled if the study (1) was randomized or (2) controlled for a baseline learning outcome....
[...]
TL;DR: The aim of this paper is to critically review what is felt to be important about the role of debriefing in the field of simulation-based learning, how it has come about and developed over time, and the different styles or approaches that are used and how effective the process is.
Abstract: The aim of this paper is to critically review what is felt to be important about the role of debriefing in the field of simulation-based learning, how it has come about and developed over time, and the different styles or approaches that are used and how effective the process is. A recent systematic
1,351 citations
TL;DR: Although the number of reports analyzed in this meta-analysis is small, these results show that SBME with DP is superior to traditional clinical medical education in achieving specific clinical skill acquisition goals.
Abstract: Purpose This article presents a comparison of the effectiveness of traditional clinical education toward skill acquisition goals versus simulation-based medical education (SBME) with deliberate practice (DP). Method This is a quantitative meta-analysis that spans 20 years, 1990 to 2010. A search strategy involving three literature databases, 12 search terms, and four inclusion criteria was used. Four authors independently retrieved and reviewed articles. Main outcome measures were extracted to calculate effect sizes.
1,311 citations
TL;DR: Internet-based learning is associated with large positive effects compared with no intervention and with non-Internet instructional methods, suggesting effectiveness similar to traditional methods.
Abstract: Context The increasing use of Internet-based learning in health professions education may be informed by a timely, comprehensive synthesis of evidence of effectiveness. Objectives To summarize the effect of Internet-based instruction for health professions learners compared with no intervention and with non-Internet interventions. Data Sources Systematic search of MEDLINE, Scopus, CINAHL, EMBASE, ERIC, TimeLit, Web of Science, Dissertation Abstracts, and the University of Toronto Research and Development Resource Base from 1990 through 2007. Study Selection Studies in any language quantifying the association of Internet-based instruction and educational outcomes for practicing and student physicians, nurses, pharmacists, dentists, and other health care professionals compared with a no-intervention or non-Internet control group or a preintervention assessment. Data Extraction Two reviewers independently evaluated study quality and abstracted information including characteristics of learners, learning setting, and intervention (including level of interactivity, practice exercises, online discussion, and duration). Data Synthesis There were 201 eligible studies. Heterogeneity in results across studies was large (I2 ≥ 79%) in all analyses. Effect sizes were pooled using a random effects model. The pooled effect size in comparison to no intervention favored Internet-based interventions and was 1.00 (95% confidence interval [CI], 0.90-1.10; P Conclusions Internet-based learning is associated with large positive effects compared with no intervention. In contrast, effects compared with non-Internet instructional methods are heterogeneous and generally small, suggesting effectiveness similar to traditional methods. Future research should directly compare different Internet-based interventions.
1,241 citations
References
More filters
TL;DR: In groups of 10, first-year medical students explore Starling's law of the heart, the physiology of the Valsalva maneuver, and the function of the baroreceptor in a clinically realistic context using the simulator.
Abstract: The popularity of the problem-based learning paradigm has stimulated new interest in small group, interactive teaching techniques. Medical educators of physiology have long recognized the value of such methods, using animal-based laboratories to demonstrate difficult physiological principles. Due to ethical and other concerns, a replacement of this teaching tool has been sought. Here, the author describes the use of a full-scale human patient simulator for such a workshop. The simulator is a life-size mannequin with physical findings (palpable pulses, breath/heart sounds, blinking eyes, etc.) and sophisticated mechanical and software models of the cardiovascular and pulmonary systems. It can be connected to standard physiological monitors to reproduce a realistic clinical environment. In groups of 10, first-year medical students explore Starling's law of the heart, the physiology of the Valsalva maneuver, and the function of the baroreceptor in a clinically realistic context using the simulator. With the use of a novel pre-/postworkshop assessment instrument that included student confidence in their answers, student confidence improved for all questions and survey items following the simulator session (P 85% of the students rating the workshop "very good" or "excellent".
80 citations
TL;DR: This study evaluated the effectiveness of a high‐fidelity simulator‐based curriculum in improving the performance of amniocentesis by obstetric trainees.
Abstract: Objective Learning curves pose a difficult problem in the teaching of technical skills: how do you teach procedural skills without compromising patients’ health? A simulatorbased curriculum has been designed to minimize the risks to patients undergoing amniocentesis by shifting the learning curve away from patients and into the laboratory. This study evaluated the effectiveness of a high-fidelity simulator-based curriculum in improving the performance of amniocentesis by obstetric trainees. Design Thirty trainees received a course on the practice of amniocentesis. The curriculum consisted of a lecture, a syllabus, and a hands-on training session with the simulator. Pre- and post-training performance were evaluated with two rating scales. Training and performance evaluation were completed using the same simulator. The effectiveness of the simulator-based workshop and the effect of year of training were assessed using a two-way analysis of variance. Results Performance scores improved from a mean score of 55% to 94% using checklist scoring and from 57% to 88% using global ratings. The two-way analysis of variance revealed a significant effect of training (F 1,60 = 43.57; P < 0.001) accounting for 45% of the variance in scores, and a significant effect of experience level (F 2,60 = 9.16; P < 0.001) accounting for 25% of the variance in scores. Conclusions A comprehensive curriculum based on a highfidelity simulator was effective at improving skills demonstrated on the simulator. The challenge remains to establish that skills acquired on a simulator are transferable to the clinical setting.
79 citations
TL;DR: To determine the reliability of assessments of medical students’ performance using the simulator as an evaluation tool, to compare these assessments to written and clinical evaluations and to elicit student opinion was determined.
Abstract: Objectives
A pilot project assessing clinical performance was undertaken using the Anaesthesia Simulation Centre at the University of Toronto. The purpose of this study was to determine the reliability of assessments of medical students’ performance using the simulator as an evaluation tool, to compare these assessments to written and clinical evaluations and to elicit student opinion. Simulator assessments were performed at the completion of the anaesthesia rotation.
Design
Twenty-four of 177 University of Toronto medical students participated in a videotaped simulator session with an attending faculty. These 24 students were based at Sunnybrook Health Science Centre. During the session, each student worked through one of six predetermined cases involving discrete patient problems based on the list of core objectives. Five evaluators independently assessed each student’s videotaped performance using standardized performance evaluation criteria and data were examined for inter-rater reliability. Clinical and written examination marks were compared to simulator assessments. A student questionnaire was administered and descriptive data obtained.
Setting
The University of Toronto.
Subjects
Medical students.
Results
The intraclass correlation coefficient (ICC) of inter-rater reliability was 0·87. There was poor correlation between the simulator and written examination marks (r=0·19, P=0·38) and between the simulator and clinical marks (r=0·04, P=0·87). The simulator experience was highly rated by students: learning experience 4·6 ± 0·51, appropriate content 4·4 ± 0·74, use as evaluation tool, 4·1 ± 0·92 (1=poor, 5=excellent).
Conclusions
Our pilot data suggest that the simulator is a reliable assessment method for medical students’ performance. Further work may justify the inclusion of the simulator as an evaluation and education tool and expanded to incorporate learning objectives of other medical disciplines.
77 citations
TL;DR: Virtual environments and computer-based simulators, although well established training tools in other fields, have not been widely incorporated into surgical education and developers of this technology may need to focus on educating potential users and addressing their concerns.
Abstract: BACKGROUND: The use of advanced technology, such as virtual environments and computer-based simulators (VR/CBS), in training has been well established by both industry and the military. In contrast the medical profession, including surgery, has been slow to incorporate such technology in its training. In an attempt to identify factors limiting the regular incorporation of this technology into surgical training programs, a survey was developed and distributed to all general surgery program directors in the United States. STUDY DESIGN: A 22-question survey was sent to 254 general surgery program directors. The survey was designed to reflect attitudes of the program directors regarding the use of computer-based simulation in surgical training. Questions were scaled from 1 to 5 with 1=strongly disagree and 5=strongly agree. RESULTS: A total of 139 responses (55%) were returned. The majority of respondents (58%) had seen VR/CBS, but only 19% had "hands-on" experience with these systems. Respondents strongly agreed that there is a need for learning opportunities outside of the operating room and a role for VR/CBS in surgical training. Respondents believed both staff and residents would support this type of training. Concerns included VR/CBS' lack of validation and potential requirements for frequent system upgrades. CONCLUSIONS: Virtual environments and computer-based simulators, although well established training tools in other fields, have not been widely incorporated into surgical education. Our results suggest that program directors believe this type of technology would be beneficial in surgical education, but they lack adequate information regarding VR/CBS. Developers of this technology may need to focus on educating potential users and addressing their concerns.
75 citations
17 Oct 2003
TL;DR: In this article, the authors present a concise source of information on effective and practical methods for constructing simulation exercises for the assessment of psychological characteristics relevant to effectiveness in work organizations, such as abilities, skills, and competencies.
Abstract: This book provides a concise source of information on effective and practical methods for constructing simulation exercises for the assessment of psychological characteristics relevant to effectiveness in work organizations. Simulation exercises present the examinee with descriptions of complex situations that stimulate aspects of real-world settings and problems. Examinees are required to demonstrate overt behavior in handling the problems presented. The process and/or products of this behavior are observed by trained assessors who observe behavior, classify behaviors into relevant dimensions, and evaluate effectiveness. Simulations can provide assessments of abilities, skills, and competencies not readily measured by other testing techniques. Developing Organizational Simulations provides practical guidance for defining the attributes to be assessed, constructing the stimulus material, and designing methods for administration and scoring. Several different situational exercises are presented, including business games, leaderless group discussions, in-baskets, one-on-one interaction simulations, and case studies/presentations. Steps to ensure the reliability, validity, and legal defensibility of assessments from simulations are described. In addition, the book presents the use of simulation exercises for the purposes of personnel selection, training, development, and certification. Professional standards and guidelines relevant to the construction of simulation exercises are also covered.
73 citations
"Features and uses of high-fidelity ..." refers background in this paper
...This definition of simulation exercises squares in nearly all respects with that of Thornton & Mueller-Hanson (2004) in their recent book, Developing Organizational Simulations: A Guide for Practitioners and Students, who emphasize the importance of using ‘‘trained assessors to observe behavior,…...
[...]