Acknowledgements the authors thank the UK General Medical Council (GMC) for commissioning an initial rapid review of the literature on selection and widening access to medicine in 2013. Further thanks are due to the UK Medical Schools Council (MSC) for commissioning a significantly updated review of selection methods in 2014, funding for which was provided by Health Education England (HEE) and the Office for Fair Access (OFFA). Working together on both of these projects encouraged us to further develop our ideas and produce an updated systematic review for publication in 2015. We also thank those who contributed to the original project funded by the GMC, notably John McLachlan, Member of the Centre for Medical Education Research, Durham University, and Emma Dunlop, Medical Admissions, University of Aberdeen. Funding funding was provided by HEE and the OFFA.

/pdf/how-effective-are-selection-methods-in-medical-education-a-1qnthmo28f.pdf

How effective are selection methods in medical education? A systematic review

Context Many medical schools assert that a racially and ethnically diverse student body is an important element in educating physicians to meet the needs of a diverse society. However, there is limited evidence addressing the educational effects of student body racial diversity. Objective To determine whether student body racial and ethnic diversity is associated with diversity-related outcomes among US medical students. Design, Setting, and Participants A Web-based survey (Graduation Questionnaire) administered by the Association of American Medical Colleges of 20112 graduating medical students (64% of all graduating students in 2003 and 2004) from 118 allopathic medical schools in the United States. Historically black and Puerto Rican medical schools were excluded. Main Outcome Measures Students' self-rated preparedness to care for patients from other racial and ethnic backgrounds, attitudes about equity and access to care, and intent to practice in an underserved area. Results White students within the highest quintile for student body racial and ethnic diversity, measured by the proportion of underrepresented minority (URM) students, were more likely to rate themselves as highly prepared to care for minority populations than those in the lowest diversity quintile (61.1 % vs 53.9 %, respectively; P<.001; adjusted odds ratio [OR], 1.33; 95% confidence interval [Cl], 1.13-1.57). This association was strongest in schools in which students perceived a positive climate for interracial interaction. White students in the highest URM quintile were also more likely to have strong attitudes endorsing equitable access to care (54.8% vs 44.2%, respectively; P<.001; adjusted OR, 1.42; 95% Cl, 1.15-1.74). For nonwhite students, after adjustment there were no significant associations between student body URM proportions and diversity-related outcomes. Student body URM proportions were not associated with white or nonwhite students' plans to practice in underserved communities, although URM students were substantially more likely than white or nonwhite/ non-URM students to plan to serve the underserved (48.7% vs 18.8% vs 16.2%, respectively; P<.001). Conclusion Student body racial and ethnic diversity within US medical schools is associated with outcomes consistent with the goal of preparing students to meet the needs of a diverse population.

Student Body Racial and Ethnic Composition and Diversity-Related Outcomes in US Medical Schools. Commentary

Why use SJTs? Traditionally, selection into medical education professions has focused primarily upon academic ability alone. This approach has been questioned more recently, as although academic attainment predicts performance early in training, research shows it has less predictive power for demonstrating competence in postgraduate clinical practice. Such evidence, coupled with an increasing focus on individuals working in healthcare roles displaying the core values of compassionate care, benevolence and respect, illustrates that individuals should be selected on attributes other than academic ability alone. Moreover, there are mounting calls to widen access to medicine, to ensure that selection methods do not unfairly disadvantage individuals from specific groups (e.g. regarding ethnicity or socio-economic status), so that the future workforce adequately represents society as a whole. These drivers necessitate a method of assessment that allows individuals to be selected on important non-academic attributes that are desirable in healthcare professionals, in a fair, reliable and valid way. What are SJTs? Situational judgement tests (SJTs) are tests used to assess individuals' reactions to a number of hypothetical role-relevant scenarios, which reflect situations candidates are likely to encounter in the target role. These scenarios are based on a detailed analysis of the role and should be developed in collaboration with subject matter experts, in order to accurately assess the key attributes that are associated with competent performance. From a theoretical perspective, SJTs are believed to measure prosocial Implicit Trait Policies (ITPs), which are shaped by socialisation processes that teach the utility of expressing certain traits in different settings such as agreeable expressions (e.g. helping others in need), or disagreeable actions (e.g. advancing ones own interest at others, expense). Are SJTs reliable, valid and fair? Several studies, including good quality meta-analytic and longitudinal research, consistently show that SJTs used in many different occupational groups are reliable and valid. Although there is over 40 years of research evidence available on SJTs, it is only within the past 10 years that SJTs have been used for recruitment into medicine. Specifically, evidence consistently shows that SJTs used in medical selection have good reliability, and predict performance across a range of medical professions, including performance in general practice, in early years (foundation training as a junior doctor) and for medical school admissions. In addition, SJTs have been found to have significant added value (incremental validity) over and above other selection methods such as knowledge tests, measures of cognitive ability, personality tests and application forms. Regarding differential attainment, generally SJTs have been found to have lower adverse impact compared to other selection methods, such as cognitive ability tests. SJTs have the benefit of being appropriate both for use in selection where candidates are novices (i.e. have no prior role experience or knowledge such as in medical school admissions) as well as settings where candidates have substantial job knowledge and specific experience (as in postgraduate recruitment for more senior roles). An SJT specification (e.g. scenario content, response instructions and format) may differ depending on the level of job knowledge required. Research consistently shows that SJTs are usually found to be positively received by candidates compared to other selection tests such as cognitive ability and personality tests. Practically, SJTs are difficult to design effectively, and significant expertise is required to build a reliable and valid SJT. Once designed however, SJTs are cost efficient to administer to large numbers of candidates compared to other tests of non-academic attributes (e.g. personal statements, structured interviews), as they are standardised and can be computer-delivered and machine-marked.

/pdf/situational-judgement-tests-in-medical-education-and-v594z8o7y5.pdf

Situational judgement tests in medical education and training: Research, theory and practice: AMEE Guide No. 100.

Health Professions' Educators (HPEs) and their learners have to adapt their educational provision to rapidly changing and uncertain circumstances linked to the COVID-19 pandemic. This paper reports on an AMEE-hosted webinar: Adapting to the impact of COVID-19: Sharing stories, sharing practice. Attended by over 500 colleagues from five continents, this webinar focused on the impact of the virus across the continuum of education and training. Short formal presentations on teaching and learning, assessment, selection and postgraduate training generated wide-ranging questions via the Chatbox. A thematic analysis of the Chatbox thread indicated the most pressing concerns and challenges educators were experiencing in having to adapt programmes and learning across the continuum of medical education and training. The main areas of concern were: campus-based teaching and learning; clinical teaching; selection and assessment, and educator needs. While there is clearly no one simple solution to the unprecedented issues medical education and training face currently, there were two over-arching messages. First, this is a time for colleagues across the globe to help and support each other. Second, many local responses and innovations could have the potential to change the shape of medical education and training in the future.

/pdf/adapting-to-the-impact-of-covid-19-sharing-stories-sharing-2jj6c4fmsa.pdf

Adapting to the impact of COVID-19: Sharing stories, sharing practice.

Medical students are exposed to high amounts of stress. Stress and poor academic performance can become part of a vicious circle. In order to counteract this circularity, it seems important to better understand the relationship between stress and performance during medical education. The most widespread stress questionnaire designed for use in Medical School is the “Perceived Medical School Stress Instrument” (PMSS). It addresses a wide range of stressors, including workload, competition, social isolation and financial worries. Our aim was to examine the relation between the perceived Medical School stress of undergraduate medical students and academic performance. We measured Medical School stress using the PMSS at two different time points (at the end of freshman year and at the end of sophomore year) and matched stress scores together with age and gender to the first medical examination (M1) grade of the students (n = 456). PMSS scores from 2 and 14 months before M1 proved to be significant predictors for medical students’ M1 grade. Age and gender also predict academic performance, making older female students with high stress scores a potential risk group for entering the vicious circle of stress and poor academic performance. PMSS sum scores 2 and 14 months before the M1 exam seem to have an independent predictive validity for medical students’ M1 grade. More research is needed to identify potential confounders.

/pdf/perceived-medical-school-stress-of-undergraduate-medical-1504icqqof.pdf

Perceived Medical School stress of undergraduate medical students predicts academic performance: an observational study

Traditional methods of assessing personality traits in medical school selection have been heavily criticised. To address this at the point of selection, "non-cognitive" tests were included in the UK Clinical Aptitude Test, the most widely-used aptitude test in UK medical education (UKCAT: http://www.ukcat.ac.uk/ ). We examined the predictive validity of these non-cognitive traits with performance during and on exit from medical school. We sampled all students graduating in 2013 from the 30 UKCAT consortium medical schools. Analysis included: candidate demographics, UKCAT non-cognitive scores, medical school performance data-the Educational Performance Measure (EPM) and national exit situational judgement test (SJT) outcomes. We examined the relationships between these variables and SJT and EPM scores. Multilevel modelling was used to assess the relationships adjusting for confounders. The 3343 students who had taken the UKCAT non-cognitive tests and had both EPM and SJT data were entered into the analysis. There were four types of non-cognitive test: (1) libertariancommunitarian, (2) NACE-narcissism, aloofness, confidence and empathy, (3) MEARS-self-esteem, optimism, control, self-discipline, emotional-nondefensiveness (END) and faking, (4) an abridged version of 1 and 2 combined. Multilevel regression showed that, after correcting for demographic factors, END predicted SJT and EPM decile. Aloofness and empathy in NACE were predictive of SJT score. This is the first large-scale study examining the relationship between performance on non-cognitive selection tests and medical school exit assessments. The predictive validity of these tests was limited, and the relationships revealed do not fit neatly with theoretical expectations. This study does not support their use in selection.

/pdf/do-personality-traits-assessed-on-medical-school-admission-57rlddsx14.pdf

Do personality traits assessed on medical school admission predict exit performance? A UK-wide longitudinal cohort study.

Background: The Personal Qualities Assessment (PQA) was developed to enhance medical student selection by measuring a range of non-cognitive attributes in the applicants to medical school. Applicants to the five Scottish medical schools were invited to pilot the test in 2001 and 2002. Aims: To evaluate the predictive validity of PQA for selecting medical students. Methods: A longitudinal cohort study was conducted in which PQA scores were compared with senior year medical school performance. Results: Consent to access performance markers was obtained from 626 students (61.6% of 1017 entrants in 2002–2003). Linkable Foundation Year (4th) rankings were available for 411 (66%) students and objective structured clinical examination (OSCE) rankings for 335 (54%) of those consenting. Both samples were representative of the original cohort. No significant correlations were detected between separate elements of the PQA assessment and student performance. However, using the algorithm advocated by Powis et al. those defined as ‘nonextreme’ (5� 1.5 SD from the cohort mean scores; SD, standard deviation) character types on the involved-detached and on the libertarian–communitarian moral orientation scales were ranked higher in OSCEs (average of 7.5% or 25 out of 335, p ¼ 0.049). Conclusions: This study was limited by high attrition and basic outcome markers which are insensitive to relevant noncognitive characteristics. However, it is the largest currently available study of predictive validity for the PQA assessment. There was one finding of significance: that those students who were identified by PQA as ‘not extreme’ on the two personal characteristics scales performed better in an OSCE measure of professionalism. Futures studies are required since psychometric testing for both cognitive and non-cognitive attributes are increasingly used in admission process and these should include more and better measures of professionalism against which to correlate non-cognitive traits.

https://www.tandfonline.com/doi/pdf/10.3109/0142159X.2011.599448

Predictive validity of the personal qualities assessment for selection of medical students in Scotland.

Dundee University School of Medicine established a pilot for a 40 week long comprehensive Longitudinal Integrated Clerkship (LIC) in 2016. Ten places for year 4 students are available which...

/pdf/dundee-s-longitudinal-integrated-clerkship-drivers-4am66ws23g.pdf

Dundee's Longitudinal Integrated Clerkship: drivers, implementation and early evaluation.

Medical schools globally are encouraged to widen access and participation for students from less privileged backgrounds. Many strategies have been implemented to address this inequality, but much still needs to be done to ensure fair access for all. In the literature, adverse circumstances include financial issues, poor educational experience and lack of professional-status parents. In order to take account of adverse circumstances faced by applicants, The University of Dundee School of Medicine offers applicants the opportunity to report circumstances which may have resulted in disadvantage. Applicants do this by completing a free text statement, known as an ‘adversity statement’, in addition to the other application information. This study analysed adversity statements submitted by applicants during two admissions cycles. Analysis of content and theme was done to identify the information applicants wished to be taken into consideration, and what range of adverse circumstances individuals reported. This study used a qualitative approach with thematic analysis to categorise the adversity statements. The data was initially analysed to create a coding framework which was then applied to the whole data set. Each coded segment was then analysed for heterogeneity and homogeneity, segments merged into generated themes, or to create sub-themes. The data set comprised a total of 384 adversity statements. These showed a wide range of detail involving family, personal health, education and living circumstances. Some circumstances, such as geographical location, have been identified and explored in previous research, while others, such as long term health conditions, have had less attention in the literature. The degree of impact, the length of statement and degree of detail, demonstrated wide variation between submissions. This study adds to the debate on best practice in contextual admissions and raises awareness of the range of circumstances and impact applicants wish to be considered. The themes which emerged from the data included family, school, personal health, and geographical location issues. Descriptions of the degree of impact that an adverse circumstance had on educational or other attainment was found to vary substantially from statements indicating minor, impact through to circumstances stated as causing major impact.

/pdf/free-text-adversity-statements-as-part-of-a-contextualised-511kxur1y9.pdf

Johnathan Dowell

Papers

How effective are selection methods in medical education? A systematic review

Do personality traits assessed on medical school admission predict exit performance? A UK-wide longitudinal cohort study.

Predictive validity of the personal qualities assessment for selection of medical students in Scotland.

Dundee's Longitudinal Integrated Clerkship: drivers, implementation and early evaluation.

Free text adversity statements as part of a contextualised admissions process: a qualitative analysis