Medical Education 2011: 45: 1048–1060



Objectives  This study was intended to develop a conceptual framework of the factors impacting on faculty members’ judgements and ratings of resident doctors (residents) after direct observation with patients.



Methods  In 2009, 44 general internal medicine faculty members responsible for out-patient resident teaching in 16 internal medicine residency programmes in a large urban area in the eastern USA watched four videotaped scenarios and two live scenarios of standardised residents engaged in clinical encounters with standardised patients. After each, faculty members rated the resident using a mini-clinical evaluation exercise and were individually interviewed using a semi-structured interview. Interviews were videotaped, transcribed and analysed using grounded theory methods.



Results  Four primary themes that provide insights into the variability of faculty assessments of residents’ performance were identified: (i) the frames of reference used by faculty members when translating observations into judgements and ratings are variable; (ii) high levels of inference are used during the direct observation process; (iii) the methods by which judgements are synthesised into numerical ratings are variable, and (iv) factors external to resident performance influence ratings. From these themes, a conceptual model was developed to describe the process of observation, interpretation, synthesis and rating.



Conclusions  It is likely that multiple factors account for the variability in faculty ratings of residents. Understanding these factors informs potential new approaches to faculty development to improve the accuracy, reliability and utility of clinical skills assessment.

https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1365-2923.2011.04025.x

Opening the black box of clinical skills assessment via observation: a conceptual model.

Objectives To determine whether paperless medical records contained less information than paper based medical records and whether that information was harder to retrieve.

Design Cross sectional study with review of medical records and interviews with general practitioners.

Setting 25 general practices in Trent region.

Participants 53 British general practitioners (25 using paperless records and 28 using paper based records) who each provided records of 10 consultations.

Main outcome measures Content of a sample of records and doctor recall of consultations for which paperless or paper based records had been made.

Results Compared with paper based records, more paperless records were fully understandable (89.2% v 69.9%, P=0.0001) and fully legible (100% v 64.3%, P < 0.0001). Paperless records were significantly more likely to have at least one diagnosis recorded (48.2% v 33.2%, P=0.05), to record that advice had been given (23.7% vs 10.7%, P=0.017), and, when a referral had been made, were more likely to contain details of the specialty (77.4% v 59.5%, P=0.03). When a prescription had been issued, paperless records were more likely to specify the drug dose (86.6% v 66.2%, P=0.005). Paperless records contained significantly more words, abbreviations, and symbols (P < 0.01 for all). At doctor interview, there was no difference between the groups for the proportion of patients or consultations that could be recalled. Doctors using paperless records were able to recall more advice given to patients (38.6% v 26.8%, P=0.03).

Conclusion We found no evidence to support our hypotheses that paperless records would be truncated and contain more local abbreviations; and that the absence of writing would decrease subsequent recall. Conversely we found that the paperless records compared favourably with manual records.

https://www.bmj.com/content/bmj/326/7404/1439.full.pdf

The electronic patient record in primary care—regression or progression? A cross sectional study

One hundred years after Abraham Flexner released his report Medical Education in the United States and Canada, the spirit of reform is alive again. Reports in the United States and Canada have called for significant changes to medical education that will allow doctors to adapt to complex environments, work in teams, and meet a wide range of social needs. These reports call for clear educational outcomes but also for a flexible, individualized approach to learning. Whether or not change will result has much to do with the alignment between what is proposed and the nature of current societal discourses about how medical education should be conducted. Currently, two powerful and competing models of competence development are operating at odds with one another. The traditional one is time-based (a "tea-steeping" model, in which the student "steeps" in an educational program for a historically determined fixed time period to become a successful practitioner). This model directs attention to processes such as admission and curriculum design. The newer one is outcomes-based (an "i-Doc" model, a name suggested by the Apple i-Pod that infers that medical schools and residencies, like factories, can produce highly desirable products adapted to user needs and desires). This model focuses more on the functional capabilities of the end product (the graduate student, resident, or practicing physician). The author explores the implications of both time-based and outcomes-based models for medical education reform and proposes an integration of their best features.

A Tea-Steeping or i-Doc Model for Medical Education?

The Accreditation Council for Graduate Medical Education (ACGME) has challenged residency programs to provide documentation via outcomes assessment that all residents have successfully mastered the six core competencies. A variety of assessment "tools" has been identified by the ACGME for outcomes assessment determination. Although rarely cited in the medical literature, 360-degree feedback is currently in widespread use in the business sector. This tool provides timely, consolidated feedback from sources in the resident's sphere of influence (emergency medicine faculty, emergency medicine residents, off-service residents and faculty, nurses, ancillary personnel, patients, out-of-hospital care providers, and a self-assessment). This is a significant deviation from both the peer review process and the resident review process that almost exclusively use physicians as raters. Because of its relative lack of development, utilization, and validation as a method of resident assessment in graduate medical education, a great opportunity exists to develop the 360-degree feedback tool for resident assessment.

https://onlinelibrary.wiley.com/doi/pdf/10.1197/aemj.9.11.1300

360-degree Feedback: Possibilities for Assessment of the ACGME Core Competencies for Emergency Medicine Residents

The recent change in working patterns of doctors in training has meant that the traditional systems of education are under increasing pressure and that there is the need to maximise new opportunities for learning. One new opportunity may arise after the introduction of the mandatory assessment systems (Mini-CEX, DOPPS, Multi-source feedback, and Case based discussion) in the Foundation Programmes. In this review the new assessment procedures for the Foundation Programmes are outlined and the potential of these assessments (using Mini-CEX as main example) as an opportunity to give feedback to trainees discussed. The importance of feedback in professional development and some of the techniques available for giving feedback are described. The Foundation Programme assessments will occupy a significant amount of trainees' and trainers' time and it is important that opportunity for feedback and learning is maximised.

https://pmj.bmj.com/content/postgradmedj/82/971/576.full.pdf

The Foundation Programme assessment tools: An opportunity to enhance feedback to trainees?

The General Medical Council procedures to assess the performance of doctors who may be seriously deficient include peer review of the doctor's practice at the workplace and tests of competence and skills. Peer reviews are conducted by three trained assessors, two from the same speciality as the doctor being assessed, with one lay assessor. The doctor completes a portfolio to describe his/her training, experience, the circumstances of practice and self rate his/her competence and familiarity in dealing with the common problems of his/her own discipline. The assessment includes a review of the doctor's medical records; discussion of cases selected from these records; observation of consultations for clinicians, or of relevant activities in non-clinicians; a tour of the doctor's workplace; interviews with at least 12 third parties (five nominated by the doctor); and structured interviews with the doctor. The content and structure of the peer review are designed to assess the doctor against the standards defined in Good Medical Practice, as applied to the doctor's speciality. The assessment methods are based on validated instruments and gather 700-1000 judgements on each doctor. Early experience of the peer review visits has confirmed their feasibility and effectiveness.

Peter McCrorie

Papers

The General Medical Council's Performance Procedures: peer review of performance in the workplace.