자연주의적 탐구조사(Naturalistic Inquiry)

ContextCore physician activities of lifelong learning, continuing medical education credit, relicensure, specialty recertification, and clinical competence are linked to the abilities of physicians to assess their own learning needs and choose educational activities that meet these needs.ObjectiveTo determine how accurately physicians self-assess compared with external observations of their competence.Data SourcesThe electronic databases MEDLINE (1966-July 2006), EMBASE (1980-July 2006), CINAHL (1982-July 2006), PsycINFO (1967-July 2006), the Research and Development Resource Base in CME (1978-July 2006), and proprietary search engines were searched using terms related to self-directed learning, self-assessment, and self-reflection.Study SelectionStudies were included if they compared physicians' self-rated assessments with external observations, used quantifiable and replicable measures, included a study population of at least 50% practicing physicians, residents, or similar health professionals, and were conducted in the United Kingdom, Canada, United States, Australia, or New Zealand. Studies were excluded if they were comparisons of self-reports, studies of medical students, assessed physician beliefs about patient status, described the development of self-assessment measures, or were self-assessment programs of specialty societies. Studies conducted in the context of an educational or quality improvement intervention were included only if comparative data were obtained before the intervention.Data ExtractionStudy population, content area and self-assessment domain of the study, methods used to measure the self-assessment of study participants and those used to measure their competence or performance, existence and use of statistical tests, study outcomes, and explanatory comparative data were extracted.Data SynthesisThe search yielded 725 articles, of which 17 met all inclusion criteria. The studies included a wide range of domains, comparisons, measures, and methodological rigor. Of the 20 comparisons between self- and external assessment, 13 demonstrated little, no, or an inverse relationship and 7 demonstrated positive associations. A number of studies found the worst accuracy in self-assessment among physicians who were the least skilled and those who were the most confident. These results are consistent with those found in other professions.ConclusionsWhile suboptimal in quality, the preponderance of evidence suggests that physicians have a limited ability to accurately self-assess. The processes currently used to undertake professional development and evaluate competence may need to focus more on external assessment.

https://jamanetwork.com/journals/jama/articlepdf/203258/JRV60012.pdf

Accuracy of Physician Self-assessment Compared With Observed Measures of Competence: A Systematic Review

support or fail to support the proposed score interpretations, at a given point in time. Data and logic are assembled into arguments – pro and con – for some specific interpretation of assessment data. Examples of types of validity evidence, data and information from each source are discussed in the context of a high-stakes written and performance examination in medical education. Conclusion All assessments require evidence of the reasonableness of the proposed interpretation, as test data in education have little or no intrinsic meaning. The constructs purported to be measured by our assessments are important to students, faculty, administrators, patients and society and require solid scientific evidence of their meaning.

https://www.wiser.pitt.edu/sites/wiser/research/extra_docs/Downing2003Validity-onmeaningful.pdf

Validity: on the meaningful interpretation of assessment data

Health measurement scales

INTRODUCTION

We use a utility model to illustrate that, firstly, selecting an assessment method involves context-dependent compromises, and secondly, that assessment is not a measurement problem but an instructional design problem, comprising educational, implementation and resource aspects. In the model, assessment characteristics are differently weighted depending on the purpose and context of the assessment.

EMPIRICAL AND THEORETICAL DEVELOPMENTS

Of the characteristics in the model, we focus on reliability, validity and educational impact and argue that they are not inherent qualities of any instrument. Reliability depends not on structuring or standardisation but on sampling. Key issues concerning validity are authenticity and integration of competencies. Assessment in medical education addresses complex competencies and thus requires quantitative and qualitative information from different sources as well as professional judgement. Adequate sampling across judges, instruments and contexts can ensure both validity and reliability. Despite recognition that assessment drives learning, this relationship has been little researched, possibly because of its strong context dependence.

ASSESSMENT AS INSTRUCTIONAL DESIGN

When assessment should stimulate learning and requires adequate sampling, in authentic contexts, of the performance of complex competencies that cannot be broken down into simple parts, we need to make a shift from individual methods to an integral programme, intertwined with the education programme. Therefore, we need an instructional design perspective.

IMPLICATIONS FOR DEVELOPMENT AND RESEARCH

Programmatic instructional design hinges on a careful description and motivation of choices, whose effectiveness should be measured against the intended outcomes. We should not evaluate individual methods, but provide evidence of the utility of the assessment programme as a whole.

/pdf/assessing-professional-competence-from-methods-to-programmes-sy9hhzj3t6.pdf

Assessing professional competence: from methods to programmes

There seems to be a widely held assumption that the long-term outcomes of most faculty development initiatives will include some degree of organizational change. In many cases they do; most people engaged in faculty development have observed change at the institutional level after faculty development interventions. However, not all these interventions lead to change. Some organizational changes take place without much faculty development taking place at all. In this chapter, we will explore the mechanisms and strategies that can be used to promote or assist in organizational change, in the form of ‘strategies for success’, for faculty developers, organizational leaders, and those participating in development. Faculty development for organizational change requires attention to the educational and institutional milieu, the workforce, and the organization itself. Most of the time, change will be slow, and also affected by external factors. But faculty development, astutely deployed, can make a significant contribution to that change.

Faculty Development for Organizational Change

• The new curriculum framework for doctors in postgraduate years 1 and 2 is a step towards seamless medical education. • The framework will need additional components to make "the curriculum" deliverable. • Assessment is an essential element of most curricula, and assessment systems should be carefully planned. • Diligent observation and rating in the workplace may provide a suitable approach. • In the future, Australia must also thoroughly engage with the debate on continuing validation of competence.

/pdf/the-new-curriculum-framework-and-assessment-practices-35l1hn9ixo.pdf

The new curriculum framework and assessment practices: current challenges for postgraduate years 1 and 2.

Objective : To determine perceived preparedness of Australian hospital-based prevocational doctors for resuscitation skills and management of emergencies, and to identify differences between doctors who perceive themselves well prepared and those who perceive themselves poorly prepared for emergencies, in demographics and exposure to desired learning methods. Methods : Questionnaire consisting of a mix of graded Likert scales and free-text answers distributed to 36 Australian hospitals for secondary distribution to hospital medical officers. Results : From 2607 questionnaires posted, 470 (18.1%) were returned. Thirty-one per cent (95% confidence interval [CI] 26–35%) felt well prepared for resuscitation and management of emergencies, 41% (CI 37–45%) felt adequately prepared and 28% (CI 24–32%) felt they were not well prepared. Those who felt well prepared reported that they had experienced more exposure to a range of educational methods, including consultant contact, supervisor feedback, clinical skills, high fidelity simulator sessions and unit meetings. Well-prepared and poorly prepared doctors had similar opinions of the usefulness of various learning methods, but the poorly prepared group more frequently expressed a desire for increased exposure to contact with registrars and consultants, clinical skills sessions and hospital and unit meetings. There were no differences in gender, age or country of origin (Australia vs international medical graduates) between those who felt well or poorly prepared. Conclusions : Many prevocational hospital doctors feel inadequately prepared for the management of emergencies. Perceived preparedness is associated with more exposure to particular educational activities. Increasing exposure to learning of emergencies in undergraduate and prevocational years could reduce the number of junior doctors who feel poorly prepared for emergencies.

Self-rated preparedness of Australian prevocational hospital doctors for emergencies

This chapter looks at how faculty development has assisted in implementing new curricula. It starts by elucidating the concept of faculty development. Essential features include both individual and corporate responsibility for anticipating personal and professional needs, the delivery of activities to meet these needs and the necessity for continuation of and reflection on this process.

Faculty Development for Curricular Implementation

Background: Despite evidence supporting simulation training and awareness that trainee exposure to surgery is suboptimal, it is not known how simulation is being incorporated in obstetrics and gynaecology (O&G) training across Australia and New Zealand. Aim: To investigate the current availability and utilisation of simulation training, and the attitudes, perceived barriers and enablers towards simulation in Australia and New Zealand. Method: A survey was distributed to O&G trainees and fellows in Australia and New Zealand. The survey recorded demographic data, current exposure to simulation and beliefs about simulation training. Results: The survey returned 624 responses (24.3%). Most trainees had access to at least one type of simulation (87%). Access to simulators was higher for trainees at tertiary hospitals (92% vs 76%). Few trainees had a simulation curriculum, allocated time or supervision for simulation training. ‘Limited access’ was the highest rated barrier to using simulation. Lack of time, other training priorities and cost were identified as further barriers. More than 80% of respondents believed simulation improves surgical skills, skills transfer to the operating theatre, and the addition of simulation to the RANZCOG curriculum would benefit trainees. However, a minority of respondents believed simulator proficiency should be shown prior to performing surgery. The need for a curriculum and supervision were highlighted as necessary supports for simulation training. Conclusions: Despite simulator availability, few trainees are supported by simulation training curricula, allocated time or supervision. Participants believed that simulation training benefits trainees and should be supported with a curriculum and teaching.

Brian Jolly

Papers

Faculty Development for Organizational Change

The new curriculum framework and assessment practices: current challenges for postgraduate years 1 and 2.

Self-rated preparedness of Australian prevocational hospital doctors for emergencies

Faculty Development for Curricular Implementation

Simulation training in obstetrics and gynaecology: What's happening on the frontline?