A little more than 10 years ago, the objective structured clinical examination (OSCE) was introduced. It includes several “stations,”; at which examinees perform a variety of clinical tasks. Although an OSCE may involve a range of testing methods, standardized patients (SPs), who are nonphysicians trained to play the role of a patient, are commonly used to assess clinical skills. This article provides a comprehensive review of large‐scale studies of the psychometric characteristics of SP‐based tests. Across studies, reliability analyses consistently indicate that the major source of measurement error is variation in examinee performance from station to station (termed content specificity in the medical‐problem‐solving literature). As a consequence, tests must include large numbers of stations to obtain a stable, reproducible assessment of examinee skills. Disagreements among raters observing examinee performance and differences between SPs playing the same patient role appear to have less effect on the pr...

Assessment of clinical skills with standardized patients: State of the art

This paper describes a situation where an alteration in the final-year assessment scheme led to changes in student learning activities which were the exact opposite of those intended. Students were seen to be spending a disproportionate amount of time studying the theoretical components of the course relative to the practical and clinical aspects. The paramount importance of the assessments and examinations in influencing student learning behaviour led the departments concerned to develop a new clinical examination which more clearly reflected the objectives of the course. A questionnaire survey was undertaken to determine how the different sections of the final assessment affected the students' approach to studying. The questionnaire was administered to graduates during their intern year for the 3 years following the introduction of the new clinical examination. Results were also obtained for the year preceding the change. The survey showed that the students developed a high regard for the new examination and its validity as a test of clinical competence. The students found that an increase in ward-based learning activities was essential for success in the final examinations. The new clinical examination has thus influenced students' learning and successfully restored the balance of their learning activities between the clinical and theoretical components of the course.

The effect of assessments and examinations on the learning of medical students.

The objective structured clinical examination (OSCE) is increasingly being used as a method of clinical assessment yet its measurement characteristics have not been well documented. Evidence is accumulating that many OSCEs may be too short to achieve reliable results. This paper reports detailed psychometric analyses of OSCEs which were administered as part of a well-established final-year examination. Generalizability theory guided investigation of test reliability. At the present test length the OSCE components showed low reliabilities relative to written components. Satisfactory reliabilities could potentially be achieved if test length was increased to approximately 6 hours, a time which would create significant logistic problems for most medical schools. Several strategies for dealing with this practical problem have been explored. Firstly, it was shown that more careful selection of stations based on their psychometric characteristics can significantly improve reliability. Secondly, where rater availability is a limiting factor to increasing test length, more can be gained by using one rater per station and having more stations than using two raters per station. Finally, OSCE scores can, with advantage, be combined with other test scores which are obtained by using less resource-intensive methods. By adopting such strategies, a reliable assessment of clinical competence could be obtained in about 4 hours of testing time which was equally divided between an OSCE constructed of practical and clinical stations and a written test.

Psychometric characteristics of the objective structured clinical examination.

(1985). Techniques of Identifying Competencies Needed of Doctors. Medical Teacher: Vol. 7, No. 1, pp. 15-25.

Techniques of Identifying Competencies Needed of Doctors

Summary
This paper reports a study which compared the performance of different groups of students and doctors on identical and equivalent tests set in an objective-type format and in a free-response format. The tests were designed to ensure that the content was relevant to clinical practice at the hospital intern level. In all test situations candidates’ scores were significantly higher in the objective tests than in the free-response tests. This difference was greater for the more junior and less competent students than for the more competent doctors. The cueing effect of the options was thought to be the main factor responsible for the difference in performance.

The results of a questionnaire survey demonstrated that students were aware of the deficiencies in multiple-choice tests. A large majority of the students believed that the free-response tests gave a more accurate assessment of their clinical ability.

It was found that in these tests, aimed at measuring aspects of clinical competence, multiple-choice questions appeared to overestimate the candidate's ability to an extent that made them less suitable than free-response questions for this purpose. It was also found that free-response tests, of the type used in this study, provide a suitable alternative to multiple-choice tests for use in the written section of clinical examinations. It was concluded that the written component of the final examination in the medical course should have a preponderance of free-response items over multiple-choice items.

A comparison of multiple‐choice tests and free‐response tests in examinations of clinical competence

A model has been developed for a problem-based criterion-referenced test of clinical competence The model, designed to increase the reliability and validity of traditional final year examinations for medical students, aims to provide a profile of student performance over a range of defined competence categories The level of competence has been referenced to that expected of an intern Both the content of the examination and the selection of test methods are based on patient problem blueprints which identify key areas that require testing The content and test methods were incorporated, as appropriate, into either th;e written section or the practical section of the examination The approach was found to be feasible and acceptable to staff and students

D I Newble

Papers

A problem-based criterion-referenced examination of clinical competence.