자연주의적 탐구조사(Naturalistic Inquiry)

ContextCore physician activities of lifelong learning, continuing medical education credit, relicensure, specialty recertification, and clinical competence are linked to the abilities of physicians to assess their own learning needs and choose educational activities that meet these needs.ObjectiveTo determine how accurately physicians self-assess compared with external observations of their competence.Data SourcesThe electronic databases MEDLINE (1966-July 2006), EMBASE (1980-July 2006), CINAHL (1982-July 2006), PsycINFO (1967-July 2006), the Research and Development Resource Base in CME (1978-July 2006), and proprietary search engines were searched using terms related to self-directed learning, self-assessment, and self-reflection.Study SelectionStudies were included if they compared physicians' self-rated assessments with external observations, used quantifiable and replicable measures, included a study population of at least 50% practicing physicians, residents, or similar health professionals, and were conducted in the United Kingdom, Canada, United States, Australia, or New Zealand. Studies were excluded if they were comparisons of self-reports, studies of medical students, assessed physician beliefs about patient status, described the development of self-assessment measures, or were self-assessment programs of specialty societies. Studies conducted in the context of an educational or quality improvement intervention were included only if comparative data were obtained before the intervention.Data ExtractionStudy population, content area and self-assessment domain of the study, methods used to measure the self-assessment of study participants and those used to measure their competence or performance, existence and use of statistical tests, study outcomes, and explanatory comparative data were extracted.Data SynthesisThe search yielded 725 articles, of which 17 met all inclusion criteria. The studies included a wide range of domains, comparisons, measures, and methodological rigor. Of the 20 comparisons between self- and external assessment, 13 demonstrated little, no, or an inverse relationship and 7 demonstrated positive associations. A number of studies found the worst accuracy in self-assessment among physicians who were the least skilled and those who were the most confident. These results are consistent with those found in other professions.ConclusionsWhile suboptimal in quality, the preponderance of evidence suggests that physicians have a limited ability to accurately self-assess. The processes currently used to undertake professional development and evaluate competence may need to focus more on external assessment.

https://jamanetwork.com/journals/jama/articlepdf/203258/JRV60012.pdf

Accuracy of Physician Self-assessment Compared With Observed Measures of Competence: A Systematic Review

support or fail to support the proposed score interpretations, at a given point in time. Data and logic are assembled into arguments – pro and con – for some specific interpretation of assessment data. Examples of types of validity evidence, data and information from each source are discussed in the context of a high-stakes written and performance examination in medical education. Conclusion All assessments require evidence of the reasonableness of the proposed interpretation, as test data in education have little or no intrinsic meaning. The constructs purported to be measured by our assessments are important to students, faculty, administrators, patients and society and require solid scientific evidence of their meaning.

https://www.wiser.pitt.edu/sites/wiser/research/extra_docs/Downing2003Validity-onmeaningful.pdf

Validity: on the meaningful interpretation of assessment data

Health measurement scales

INTRODUCTION

We use a utility model to illustrate that, firstly, selecting an assessment method involves context-dependent compromises, and secondly, that assessment is not a measurement problem but an instructional design problem, comprising educational, implementation and resource aspects. In the model, assessment characteristics are differently weighted depending on the purpose and context of the assessment.

EMPIRICAL AND THEORETICAL DEVELOPMENTS

Of the characteristics in the model, we focus on reliability, validity and educational impact and argue that they are not inherent qualities of any instrument. Reliability depends not on structuring or standardisation but on sampling. Key issues concerning validity are authenticity and integration of competencies. Assessment in medical education addresses complex competencies and thus requires quantitative and qualitative information from different sources as well as professional judgement. Adequate sampling across judges, instruments and contexts can ensure both validity and reliability. Despite recognition that assessment drives learning, this relationship has been little researched, possibly because of its strong context dependence.

ASSESSMENT AS INSTRUCTIONAL DESIGN

When assessment should stimulate learning and requires adequate sampling, in authentic contexts, of the performance of complex competencies that cannot be broken down into simple parts, we need to make a shift from individual methods to an integral programme, intertwined with the education programme. Therefore, we need an instructional design perspective.

IMPLICATIONS FOR DEVELOPMENT AND RESEARCH

Programmatic instructional design hinges on a careful description and motivation of choices, whose effectiveness should be measured against the intended outcomes. We should not evaluate individual methods, but provide evidence of the utility of the assessment programme as a whole.

/pdf/assessing-professional-competence-from-methods-to-programmes-sy9hhzj3t6.pdf

Assessing professional competence: from methods to programmes

Objectives

Undergraduate medical curricula now include increasing amounts of project work aimed at developing skills related to lifelong learning. One course allows students to choose from a wide range of projects, including ‘conventional’ hospital specialties and also from topics outside the mainstream of medicine.



Design

‘Conventional’ and ‘external’ projects were compared in terms of the prior academic abilities of the students undertaking them, the assessment results and student and supervisor feedback, in order to consider whether the unconventional projects were equally valid within the undergraduate medical curriculum.



Setting

School of Medicine, University of Leeds, UK.



Subjects

Medical students.



Results

No difference between the assessment results of the student groups was present, with over 85% of all students reaching a standard of ‘excellent’ or ‘good’ in their overall final grade. There was no difference in prior academic abilities between the student groups. Enjoyment of modules was comparable between student groups (‘conventional’ 89%, ‘external’ 93%) with good levels of satisfaction with the quality of supervision. There were no differences in students’ self-appraisal of generic skill acquisition. Students who had undertaken ‘external’ projects felt they had gained less experience in data-handling and problem-solving skills. However, ‘external’ projects were rated higher by students in terms of having realistic and achievable objectives, and the supervisors of these projects were also more realistic about time commitments involved in project supervision.



Conclusions

‘External’ modules were very popular, with over 45% of students requesting places which were available for fewer than 20% of students per year. Concerns regarding the appropriateness of self-directed undergraduate medical student projects outside the mainstream of medical practice were unfounded.

Undergraduate projects - do they have to be within the conventional medical environment?

Aim : To undertake a review of general practitioner (GP) mental health training focusing on three teaching formats; lecture-based, skills-based, and multifaceted strategies. Methods : The databases of Medline, Educational Resources Information Centre, the Cochrane Database of Systematic Reviews, the Database of Abstacts of Reviews of Effectiveness were searched from the years 1980 to 2001. Search terms included general practice, family physician, family practice, mental disorders, quality of health care, professional education and combinations of these terms. Results : Lecture-based approaches combined with case discussion demonstrated improvements in GP performance. Improvements in patient outcomes were achieved using skills-based and multifaceted teaching strategies. Conclusion : More research into GP mental health training is necessary, as well as increased focus on better sampling of GPs, and measurement of GP and patient outcomes using standardized instruments. „ 2003 Blackwell Publishing Asia and Wonca

/pdf/effective-methodology-for-mental-health-training-of-general-2bkoh9t4h1.pdf

Effective methodology for mental health training of general practitioners

Context: The accreditation of undergraduate medical education is a universal undertaking. Despite the widespread adoption of accreditation processes and an increasing focus on accreditation as a mechanism to ensure minimum standards are met in various fields, there is little evidence to support the effectiveness of accreditation. Traditionally, accreditation has worked toward achieving two ends: assuring and improving quality. Many recent articles emphasize the need for continuous quality improvement mechanisms to work, as well as the quality assurance role of accreditation.Methods: The aim of the study was to examine the purposes and outcomes of accreditation, and stakeholders’ experience of accreditation in Saudi Arabia. Triangulation of data was achieved through literature review, analysis of accreditation documents, examined the outcome of accreditation process (pre and post) through stakeholders’ experience of accreditation (learner, teacher, and academic leader perspectives). Data were inter...

Accreditation of medical schools in Saudi Arabia: A qualitative study.

There may be different ways to teach and learn emotional competences during medical training. Accordingly, different students learn different things, at different times and for different reasons. The literature presents a wide range of learnercentred and reflective experiences (pedagogic theatre, role-play, simulations, positive role modelling, video feedback, portfolios and mentoring) as means of developing students’ emotional competences. These strategies seek reflection on practice as alternatives to students’ change in (or acquisition of) attitudes.

Shining light on competence

OBJECTIVE: The fitness to practise of international medical graduates (IMGs) is usually evaluated with standardised assessment tests. Practising doctors should, however, be assessed on their performance rather than their competency, for which reason workplace-based assessment (WBA) has gained increasing attention. Our aim was to assess the composite reliability of WBA instruments for assessing the performance of IMGs. DESIGN AND SETTING: Between June 2010 and April 2015, 142 IMGs were assessed by 99 calibrated assessors; each cohort was assessed at their workplace over 6 months. The IMGs completed 970 case-based discussions (CBDs), 1741 Mini-Clinical Examination Exercises (mini-CEX) and 1020 multisource feedback (MSF) sessions. PARTICIPANTS: 103 male and 39 female candidates based in urban and rural hospitals of the Hunter New England Health region, from 28 countries (Africa, Asia, Europe, South America, South Pacific). MAIN OUTCOME MEASURES: The reliability of the three WBA tools; the composite reliability of the tools as a group. RESULTS: The composite reliability of our WBA toolbox program was good: the composite reliability coefficient for five CBDs and 12 mini-CEX was 0.895 (standard error of measurement, 0.138). When the six MSF results were included, the composite reliability coefficient was 0.899 (standard error of measurement, 0.125). CONCLUSIONS: WBA is a reliable method for assessing IMGs when multiple tools and assessors are used over a period of time. This form of assessment meets the criteria for "good assessment" (reliability >/= 0.8) and can be applied in other settings.

Brian Jolly

Papers

Undergraduate projects - do they have to be within the conventional medical environment?

Effective methodology for mental health training of general practitioners

Accreditation of medical schools in Saudi Arabia: A qualitative study.

Shining light on competence

Composite reliability of workplace-based assessment of international medical graduates.