SUMMARY Review date: 1969 to 2003, 34 years. Background and context: Simulations are now in widespread use in medical education and medical personnel evaluation. Outcomes research on the use and effectiveness of simulation technology in medical education is scattered, inconsistent and varies widely in methodological rigor and substantive focus. Objectives: Review and synthesize existing evidence in educational science that addresses the question, ‘What are the features and uses of high-fidelity medical simulations that lead to most effective learning?’. Search strategy: The search covered five literature databases (ERIC, MEDLINE, PsycINFO, Web of Science and Timelit) and employed 91 single search terms and concepts and their Boolean combinations. Hand searching, Internet searches and attention to the ‘grey literature’ were also used. The aim was to perform the most thorough literature search possible of peer-reviewed publications and reports in the unpublished literature that have been judged for academic quality. Inclusion and exclusion criteria: Four screening criteria were used to reduce the initial pool of 670 journal articles to a focused set of 109 studies: (a) elimination of review articles in favor of empirical studies; (b) use of a simulator as an educational assessment or intervention with learner outcomes measured quantitatively; (c) comparative research, either experimental or quasi-experimental; and (d) research that involves simulation as an educational intervention. Data extraction: Data were extracted systematically from the 109 eligible journal articles by independent coders. Each coder used a standardized data extraction protocol. Data synthesis: Qualitative data synthesis and tabular presentation of research methods and outcomes were used. Heterogeneity of research designs, educational interventions, outcome measures and timeframe precluded data synthesis using meta-analysis. Headline results: Coding accuracy for features of the journal articles is high. The extant quality of the published research is generally weak. The weight of the best available evidence suggests that high-fidelity medical simulations facilitate learning under the right conditions. These include the following:

/pdf/features-and-uses-of-high-fidelity-medical-simulations-that-51wb8k5opy.pdf

Features and uses of high-fidelity medical simulations that lead to effective learning: a BEME systematic review

The discovery of a series of genetic and serological markers associated with disease susceptibility and phenotype in inflammatory bowel disease has led to the prospect of an integrated classification system involving clinical, serological and genetic parameters. The Working Party has reviewed current clinical classification systems in Crohn's disease, ulcerative colitis and indeterminate colitis, and provided recommendations for clinical classification in practice. Progress with respect to integrating serological and genetic markers has been examined in detail, and the implications are discussed. While an integrated system is not proposed for clinical use at present, the introduction of a widely acceptable clinical subclassification is strongly advocated, which would allow detailed correlations among serotype, genotype and clinical phenotype to be examined and confirmed in independent cohorts of patients and, thereby, provide a vital foundation for future work.

/pdf/toward-an-integrated-clinical-molecular-and-serological-4qawah6knq.pdf

Toward an Integrated Clinical, Molecular and Serological Classification of Inflammatory Bowel Disease: Report of a Working Party of the 2005 Montreal World Congress of Gastroenterology

ContextCurrent assessment formats for physicians and trainees reliably test
core knowledge and basic skills. However, they may underemphasize some important
domains of professional medical practice, including interpersonal skills,
lifelong learning, professionalism, and integration of core knowledge into
clinical practice.ObjectivesTo propose a definition of professional competence, to review current
means for assessing it, and to suggest new approaches to assessment.Data SourcesWe searched the MEDLINE database from 1966 to 2001 and reference lists
of relevant articles for English-language studies of reliability or validity
of measures of competence of physicians, medical students, and residents.Study SelectionWe excluded articles of a purely descriptive nature, duplicate reports,
reviews, and opinions and position statements, which yielded 195 relevant
citations.Data ExtractionData were abstracted by 1 of us (R.M.E.). Quality criteria for inclusion
were broad, given the heterogeneity of interventions, complexity of outcome
measures, and paucity of randomized or longitudinal study designs.Data SynthesisWe generated an inclusive definition of competence: the habitual and
judicious use of communication, knowledge, technical skills, clinical reasoning,
emotions, values, and reflection in daily practice for the benefit of the
individual and the community being served. Aside from protecting the public
and limiting access to advanced training, assessments should foster habits
of learning and self-reflection and drive institutional change. Subjective,
multiple-choice, and standardized patient assessments, although reliable,
underemphasize important domains of professional competence: integration of
knowledge and skills, context of care, information management, teamwork, health
systems, and patient-physician relationships. Few assessments observe trainees
in real-life situations, incorporate the perspectives of peers and patients,
or use measures that predict clinical outcomes.ConclusionsIn addition to assessments of basic skills, new formats that assess
clinical reasoning, expert judgment, management of ambiguity, professionalism,
time management, learning strategies, and teamwork promise a multidimensional
assessment while maintaining adequate reliability and validity. Institutional
support, reflection, and mentoring must accompany the development of assessment
programs.

/pdf/defining-and-assessing-professional-competence-1sy606vpxu.pdf

Defining and Assessing Professional Competence

Many students are being left behind by an educational system that some people believe is in crisis. Improving educational outcomes will require efforts on many fronts, but a central premise of this monograph is that one part of a solution involves helping students to better regulate their learning through the use of effective learning techniques. Fortunately, cognitive and educational psychologists have been developing and evaluating easy-to-use learning techniques that could help students achieve their learning goals. In this monograph, we discuss 10 learning techniques in detail and offer recommendations about their relative utility. We selected techniques that were expected to be relatively easy to use and hence could be adopted by many students. Also, some techniques (e.g., highlighting and rereading) were selected because students report relying heavily on them, which makes it especially important to examine how well they work. The techniques include elaborative interrogation, self-explanation, summarization, highlighting (or underlining), the keyword mnemonic, imagery use for text learning, rereading, practice testing, distributed practice, and interleaved practice. To offer recommendations about the relative utility of these techniques, we evaluated whether their benefits generalize across four categories of variables: learning conditions, student characteristics, materials, and criterion tasks. Learning conditions include aspects of the learning environment in which the technique is implemented, such as whether a student studies alone or with a group. Student characteristics include variables such as age, ability, and level of prior knowledge. Materials vary from simple concepts to mathematical problems to complicated science texts. Criterion tasks include different outcome measures that are relevant to student achievement, such as those tapping memory, problem solving, and comprehension. We attempted to provide thorough reviews for each technique, so this monograph is rather lengthy. However, we also wrote the monograph in a modular fashion, so it is easy to use. In particular, each review is divided into the following sections: General description of the technique and why it should work How general are the effects of this technique? 2a. Learning conditions 2b. Student characteristics 2c. Materials 2d. Criterion tasks Effects in representative educational contexts Issues for implementation Overall assessment The review for each technique can be read independently of the others, and particular variables of interest can be easily compared across techniques. To foreshadow our final recommendations, the techniques vary widely with respect to their generalizability and promise for improving student learning. Practice testing and distributed practice received high utility assessments because they benefit learners of different ages and abilities and have been shown to boost students' performance across many criterion tasks and even in educational contexts. Elaborative interrogation, self-explanation, and interleaved practice received moderate utility assessments. The benefits of these techniques do generalize across some variables, yet despite their promise, they fell short of a high utility assessment because the evidence for their efficacy is limited. For instance, elaborative interrogation and self-explanation have not been adequately evaluated in educational contexts, and the benefits of interleaving have just begun to be systematically explored, so the ultimate effectiveness of these techniques is currently unknown. Nevertheless, the techniques that received moderate-utility ratings show enough promise for us to recommend their use in appropriate situations, which we describe in detail within the review of each technique. Five techniques received a low utility assessment: summarization, highlighting, the keyword mnemonic, imagery use for text learning, and rereading. These techniques were rated as low utility for numerous reasons. Summarization and imagery use for text learning have been shown to help some students on some criterion tasks, yet the conditions under which these techniques produce benefits are limited, and much research is still needed to fully explore their overall effectiveness. The keyword mnemonic is difficult to implement in some contexts, and it appears to benefit students for a limited number of materials and for short retention intervals. Most students report rereading and highlighting, yet these techniques do not consistently boost students' performance, so other techniques should be used in their place (e.g., practice testing instead of rereading). Our hope is that this monograph will foster improvements in student learning, not only by showcasing which learning techniques are likely to have the most generalizable effects but also by encouraging researchers to continue investigating the most promising techniques. Accordingly, in our closing remarks, we discuss some issues for how these techniques could be implemented by teachers and students, and we highlight directions for future research.

https://www.wku.edu/senate/documents/improving_student_learning_dunlosky_2013.pdf

Improving Students’ Learning With Effective Learning Techniques: Promising Directions From Cognitive and Educational Psychology

Research from numerous corners of psychological inquiry suggests that self-assessments of skill and character are often flawed in substantive and systematic ways. We review empirical findings on the imperfect nature of self-assessment and discuss implications for three real-world domains: health, education, and the workplace. In general, people's self-views hold only a tenuous to modest relationship with their actual behavior and performance. The correlation between self-ratings of skill and actual performance in many domains is moderate to meager-indeed, at times, other people's predictions of a person's outcomes prove more accurate than that person's self-predictions. In addition, people overrate themselves. On average, people say that they are "above average" in skill (a conclusion that defies statistical possibility), overestimate the likelihood that they will engage in desirable behaviors and achieve favorable outcomes, furnish overly optimistic estimates of when they will complete future projects, and reach judgments with too much confidence. Several psychological processes conspire to produce flawed self-assessments. Research focusing on health echoes these findings. People are unrealistically optimistic about their own health risks compared with those of other people. They also overestimate how distinctive their opinions and preferences (e.g., discomfort with alcohol) are among their peers-a misperception that can have a deleterious impact on their health. Unable to anticipate how they would respond to emotion-laden situations, they mispredict the preferences of patients when asked to step in and make treatment decisions for them. Guided by mistaken but seemingly plausible theories of health and disease, people misdiagnose themselves-a phenomenon that can have severe consequences for their health and longevity. Similarly, research in education finds that students' assessments of their performance tend to agree only moderately with those of their teachers and mentors. Students seem largely unable to assess how well or poorly they have comprehended material they have just read. They also tend to be overconfident in newly learned skills, at times because the common educational practice of massed training appears to promote rapid acquisition of skill-as well as self-confidence-but not necessarily the retention of skill. Several interventions, however, can be introduced to prompt students to evaluate their skill and learning more accurately. In the workplace, flawed self-assessments arise all the way up the corporate ladder. Employees tend to overestimate their skill, making it difficult to give meaningful feedback. CEOs also display overconfidence in their judgments, particularly when stepping into new markets or novel projects-for example, proposing acquisitions that hurt, rather then help, the price of their company's stock. We discuss several interventions aimed at circumventing the consequences of such flawed assessments; these include training people to routinely make cognitive repairs correcting for biased self-assessments and requiring people to justify their decisions in front of their peers. The act of self-assessment is an intrinsically difficult task, and we enumerate several obstacles that prevent people from reaching truthful self-impressions. We also propose that researchers and practitioners should recognize self-assessment as a coherent and unified area of study spanning many subdisciplines of psychology and beyond. Finally, we suggest that policymakers and other people who makes real-world assessments should be wary of self-assessments of skill, expertise, and knowledge, and should consider ways of repairing self-assessments that may be flawed.

https://journals.sagepub.com/doi/pdf/10.1111/j.1529-1006.2004.00018.x

Flawed Self-Assessment Implications for Health, Education, and the Workplace

Background The technical skill of surgical trainees is not well assessed. This study aimed (1) to compare the reliability of three scoring systems, (2) to compare live and bench formats and (3) to assess construct validity of a test of operative skill.

Methods Parallel examinations of operative skill, one using live animals and one using simulations, were developed. Performance was graded using operation-specific checklists, detailed global rating forms and pass/fail judgements. Twenty surgical residents each took both formats.

Results Disattenuated correlations between live and bench scores were high (0.69–0.72). Mean inter-rater reliability across stations ranged from 0.64 to 0.72. Internal consistency was moderate to high (α: 0.61–0.74) for the live format using the checklist and for live and bench formats using global ratings. Global ratings discriminated between resident levels for both formats (bench: F(2,17) = 4.45, P < 0.05; live: F(2,17) = 3.5, P < 0.05), checklists did not.

Conclusion This preliminary study suggests that the Objective Structured Assessment of Technical Skill can reliably and validly assess surgical skills. Global ratings are a better method of assessment than task-specific checklists. Bench model simulation gives equivalent results to use of live animals for this test format.

Objective structured assessment of technical skill (OSATS) for surgical residents.

Traditionally, surgeons have been trained and evaluated on the basis of their performance of surgical procedures in live patients. This article in the Medical Education series explores the use of mechanical devices for the teaching and evaluation of surgical skills.

Teaching Surgical Skills — Changes in the Wind

Background A new approach to testing operative technical skills, the Objective Structured Assessment of Technical Skill (OSATS), formally assesses discrete segments of surgical tasks using bench model simulations. This study examines the interstation reliability and construct validity of a large-scale administration of the OSATS. Methods A 2-hour, eight-station OSATS was administered to 48 general surgery residents. Residents were assessed at each station by one of 48 surgeons who evaluated the resident using two methods of scoring: task-specific checklists and global rating scales. Results lnterstation reliability was 0.78 for the checklist score, and 0.85 for the global score. Analysis of variance revealed a significant effect of training for both the checklist score, F(3,44) = 20.08, P Conclusions The OSATS demonstrates high reliability and construct validity, suggesting that we can effectively measure residents' technical ability outside the operating room using bench model simulations.

/pdf/testing-technical-skill-via-an-innovative-bench-station-356r0szbhn.pdf

Testing technical skill via an innovative “bench station” examination

PURPOSE: To compare the psychometric properties of checklists, global rating scales preceded by a checklist, and global rating scales alone in assessing surgery residents' performances on an OSCE-like technical skills examination. METHOD: In 1996, 53 general surgery residents with one to six years o

Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examination.

Changes in the surgical training curriculum, precipitated by limitations to resident work hours,1,2 concerns over patient safety,3,4 and budgetary constraints in the operating room5,6 have compelled surgical educators to search for more effective and creative means of teaching surgical skills. Emerging technologies have also created a need for strategies to deliver technical education to surgeons already in practice. Consequently, laboratories dedicated to teaching technical aspects of surgical skill have become increasingly popular worldwide. These laboratories deploy an increasing array of training models, both low and high fidelity, many of which have been validated as effective teaching tools.7,8 Much work has been done on the development of performance metrics of assessment that can chart the progress through training regimens, demonstrate effects of intervention, and differentiate between varying levels of experience.9,10 Recently, there has been an important focus on ensuring that skills taught in laboratory environments can translate to the real world of human operations.11,12

Despite the important work on model development, simulators, and assessment methodologies, there is needed effort in the realm of curricular development.13 The ways in which the training is delivered (how, what, when, and how often) are equally important. For example, in many programs, junior residents learn a particular skill 1 week and return the next week to learn a different skill, with very little opportunity structured within the curriculum to rehearse what has previously been taught. Likewise, many CME programs offer courses that are short and intensive (1-day or weekend), not allowing for rehearsal of skills after a period of rest (and forgetting) has elapsed. Commonly, there is a delay between the time the skill is learned in the laboratory and the time the skill is needed in the operating room. Performance may be adequate immediately following training, but how much is retained is uncertain. The ultimate value of skills courses should be measured not by performance immediately after training, but by performance after a time delay, preferably in a realistic setting.

Drawing upon the motor skill learning principle of massed versus distributed practice found in the domains of psychology and athletics, there is good evidence that practice interspersed with periods of rest (distributed practice), leads to better acquisition and retention of skill compared with practice delivered in continuous blocks with little or no rest in between (massed practice).14–16 How these principles transfer to the domain of surgical skill acquisition have yet to be demonstrated. This randomized controlled trial was designed to assess the impact that scheduling of practice (massed versus distributed) has on surgical skill acquisition. The study was designed to have maximal relevance to current residents' surgical skills courses and curricula. Using previously validated outcome measures and microsurgery as the technical skill domain, this study evaluates not only the immediate impact of each training schedule, but more importantly, the durability of skill acquisition (“retention” testing 1 month later) and the clinical transferability of the skill to a more realistic setting (testing on live anesthetized rats).

Helen MacRae

Papers

Objective structured assessment of technical skill (OSATS) for surgical residents.

Teaching Surgical Skills — Changes in the Wind

Testing technical skill via an innovative “bench station” examination

Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examination.

Teaching surgical skills: what kind of practice makes perfect?: a randomized, controlled trial.