The introduction of computer-based testing in high-stakes examining in higher education is developing rather slowly due to institutional barriers (the need of extra facilities, ensuring test security) and teacher and student acceptance. From the existing literature it is unclear whether computer-based exams will result in similar results as paper-based exams and whether student acceptance can change as a result of administering computer-based exams. In this study, we compared results from a computer-based and paper-based exam in a sample of psychology students and found no differences in total scores across the two modes. Furthermore, we investigated student acceptance and change in acceptance of computer-based examining. After taking the computer-based exam, fifty percent of the students preferred paper-and-pencil exams over computer-based exams and about a quarter preferred a computer-based exam. We conclude that computer-based exam total scores are similar as paper-based exam scores, but that for the acceptance of high-stakes computer-based exams it is important that students practice and get familiar with this new mode of test administration.

/pdf/introducing-computer-based-testing-in-high-stakes-exams-in-2vj63zwf16.pdf

Introducing Computer-Based Testing in High-Stakes Exams in Higher Education: Results of a Field Experiment.

Das aktuelle Angebot digitaler Lehr- und Lernformate im Medizinstudium ist ausgesprochen heterogen. Es umfasst, neben den weit verbreiteten klassischen, statischen Formaten, soziale Kommunikationstools, audio- und videobasierte Medien, interaktive Formate und elektronische Prufungssysteme. Bei Medizinstudierenden ist die private Nutzung digitaler Medien nicht zwangslaufig mit deren sinnvollem Einsatz im Studium verbunden. Viele sammeln ihre Erfahrungen mit digitalem Lernen, im Sinne von „assessment drives learning“, vor allem durch das Absolvieren von Onlineprufungen in einer passiven, konsumierenden Rolle. Rund die Halfte aller Medizinstudierenden konnen somit als „E-Pruflinge“ bezeichnet werden, die im Wesentlichen keinen aktiven Einfluss auf ihre digitale Umwelt nehmen. Nur rund ein Viertel sind „digitale Allrounder“, die aus dem breiten Angebot der digitalen Medien ihr individuelles Lernportfolio zusammenstellen. Zum jetzigen Zeitpunkt ist der Einsatz digitaler Medien im Rahmen des Medizinstudiums noch kein integraler und flachendeckender Bestandteil der Lehre, sondern kommt vielmehr im Sinne einer punktuellen Anreicherung der Lehre zum Einsatz. Aktuelle Trends der digitalen Lehr- und Lernangebote sind mobile, interaktive und personalisierte Formate sowie die Zunahme der Relevanz von Lernplattformen. Zudem zeigt sich, dass neue didaktische Lehrformate, die sich an das veranderte Lernverhalten der Studierenden anpassen, eine hohere Akzeptanz erfahren als traditionelle Lehrformate. Zusatzlich gewinnt die Digitalisierung aktuell auch als curricularer Inhalt des Medizinstudiums an Bedeutung.

Digitale Lehr- und Lernangebote in der medizinischen Ausbildung

Nursing Students' Perceptions of the Objective Structured Clinical Examination: An Integrative Review

Construct: In this study, we examine the differences in test performance between the paper-based and the computer-based version of the Berlin formative Progress Test. In this context it is the first study that allows controlling for students’ prior performance. Background: Computer-based tests make possible a more efficient examination procedure for test administration and review. Although university staff will benefit largely from computer-based tests, the question arises if computer-based tests influence students’ test performance. Approach: A total of 266 German students from the 9th and 10th semester of medicine (comparable with the 4th-year North American medical school schedule) participated in the study (paper = 132, computer = 134). The allocation of the test format was conducted as a randomized matched-pair design in which students were first sorted according to their prior test results. The organizational procedure, the examination conditions, the room, and seating arrangements, as well as the o...

Computer versus paper--does it make any difference in test performance?

This Innovative Practice Full Paper describes lessons learned from and operational details of a full-scale Computer-Based Testing Facility (CBTF) over a period of almost 4 years. The CBTF has grown into a key resource for enabling the graceful scaling of many of the largest classes in our College of Engineering. In Fall 2017, the CBTF served 21 courses from seven different departments and over 6,000 unique students. Over 52,000 exams were delivered, including 3,500 final exams.This paper discusses five main aspects of our CBTF. First, we present the basic operation of the CBTF. Second, we discuss the precautions we take to maintain a secure exam environment. Third, we discuss how we support students that require testing accommodations like extra time and/or a distraction-reduced environment. Fourth, we discuss how we organize our policies to handle exceptional circumstances with minimal intervention by faculty. Finally, we discuss the cost of operating the CBTF and how it compares to traditional exams and online services.

Making Testing Less Trying: Lessons Learned from Operating a Computer-Based Testing Facility

Background
Computer-based examinations (CBE) ensure higher efficiency with respect to producibility and assessment compared to paper-based examinations (PBE). However, students often have objections against CBE and are afraid of getting poorer results in a CBE.

The aims of this study were (1) to assess the readiness and the objections of students to a CBE vs. PBE (2) to examine the acceptance and satisfaction with the CBE on a voluntary basis, and (3) to compare the results of the examinations, which were conducted in different formats.

https://link.springer.com/content/pdf/10.1186%2F1472-6920-11-89.pdf

Does medical students' preference of test format (computer-based vs. paper-based) have an influence on performance?

Background: Objective Structured Clinical Examinations (OSCEs) often involve a considerable amount of resources in terms of materials and organization since the scores are often recorded on paper. Computer-assisted administration is an alternative with which the need for material resources can be reduced. In particular, the use of tablets seems sensible because these are easy to transport and flexible to use.

Aim: User acceptance concerning the use of tablets during OSCEs has not yet been extensively investigated. The aim of this study was to evaluate tablet-based OSCEs from the perspective of the user (examiner) and the student examinee.

Method: For two OSCEs in Internal Medicine at the University of Heidelberg, user acceptance was analyzed regarding tablet-based administration (satisfaction with functionality) and the subjective amount of effort as perceived by the examiners. Standardized questionnaires and semi-standardized interviews were conducted (complete survey of all participating examiners). In addition, for one OSCE, the subjective evaluation of this mode of assessment was gathered from a random sample of participating students in semi-standardized interviews.

Results: Overall, the examiners were very satisfied with using tablets during the assessment. The subjective amount of effort to use the tablet was found on average to be “hardly difficult”. The examiners identified the advantages of this mode of administration as being in particular the ease of use and low rate of error. During the interviews of the examinees, acceptance for the use of tablets during the assessment was also detected.

Discussion: Overall, it was found that the use of tablets during OSCEs was well accepted by both examiners and examinees. We expect that this mode of assessment also offers advantages regarding assessment documentation, use of resources, and rate of error in comparison with paper-based assessments; all of these aspects should be followed up on in further studies.

Electronic acquisition of OSCE performance using tablets.

The development, implementation and evaluation of assessments require considerable resources and often cannot be carried out by a single faculty/institution. Therefore some medical faculties have founded cooperation projects which mainly focus on the exchange of multiple choice questions (MCQs). Since these cooperation projects do not entirely support all relevant processes in terms of preparation, implementation and evaluation of assessment, in 2006 the Medical Assessment Alliance (MAA) was founded for mutual support. In addition to MCQs the MAA started to develop innovative assessment formats and facilitate content through a coordinated exchange of experiences. To support cooperation within this network, the web-based Item Management System (IMS) was developed which supports all processes of the assessment workflow as an all-in-one working platform. At present, the Alliance has 28 partner faculties in Europe. More than 2.800 users in 750 working groups are collaborating. Currently 90.000 questions have been stored in the IMS. Since 2007, nearly 4.600 examinations have been successfully conducted. This article describes in detail the unique features of the IMS and contrasts it with the item management systems of other associations.

/pdf/good-exams-made-easy-the-item-management-system-for-multiple-1i7m9ay41p.pdf

Good exams made easy: The item management system for multiple examination formats

The COVID-19 (coronavirus disease 2019) pandemic in 2020 has had a lasting impact on all areas of personal life. However, the political, economic, legal and healthcare system, as well as the education system have also experienced the effects. Universities had to face new challenges and requirements in teaching and examinations as quickly as possible in order to be able to guarantee high-quality education for their students.

This study aims to examine how the German-speaking medical faculties of the Umbrella Consortium of Assessment Network (UCAN) have dealt with the challenges but also the opportunities that the outbreak of the COVID-19 pandemic created in medical education and whether digitalisation has been driven forward as a result. In an initial online-survey we focused our questions on the current teaching situation with regard to digitised teaching content, the support or establishment of adequate framework conditions by the medical faculties and IT facilities and also the execution of examinations during the summer semester 2020.

Between August and September 2020, a total of 88 examiners, educators, dean of study and/or technical admins from 32 partner faculties took part in the survey. Students were not included in our survey. Most respondents stated that a switch to a digital semester had worked, the use of e-learning increased compared to previous semesters and that most courses could be converted, with the exception of practical courses, which were largely cancelled. The respondents also indicated that most examinations could still be taken, with the exception of practical examination formats, like Objective Structured Clinical Examinations (OSCEs). However, in the case of face-to-face examinations, strict distance and hygiene conditions had to be obeyed or there had to be a switch to distance-online examinations, which raised many open issues such as equal opportunities of students (technical equipment, internet access, premises) and attempts at deception (third-party help with the exam, mutual exchange between students, web search).

In conclusion, we identified several issues regarding the rapid transition to a digital semester due to COVID-19 which were categorised into the following topics: Face-to-face teaching could not take place, (2) know-how of educators, (3) integrity aspects, (4) technical aspects, (5) additional personnel required, (6) additional time and effort required for implementation of digital teaching. Our study shows that a switch to digital teaching and distance online examinations is feasible, but many problems were encountered concerning academic integrity and basic ethical principles still need to be solved. In order to investigate whether above mentioned issued could be solved one year after the transition to a digital semester, we conducted a second survey in which the 32 initially surveyed institutions were questioned again.

Supplementary Information
The online version contains supplementary material available at 10.1007/s40979-021-00084-8.

/pdf/impact-of-covid-19-on-digital-medical-education-4emi3bfrgi.pdf

Impact of COVID-19 on digital medical education: compatibility of digital teaching and examinations with integrity and ethical principles

Digital assessment is becoming more and more popular within medical education. To analyse the dimensions of this digital trend, we investigated how exam questions (items) are created and designed for use in digital medical assessments in Germany. Thus, we want to explore whether different types of media are used for item creation and if a digital trend in medical assessment can be observed. In a cross-sectional descriptive study, we examined data of 30 German medical faculties stored within a common assessment platform. More precise, 23,008 exams which contained 847,137 items were analysed concerning the exam type (paper-, computer- or tablet-based) and their respective media content (picture, video and/or audio). Out of these, 5252 electronic exams with 12,214 questions were evaluated. The media types per individual question were quantified. The amount of computer- and tablet-based exams were rapidly increasing from 2012 until 2018. Computer- and tablet-based written exams showed with 45 and 66% a higher percentage of exams containing media in comparison to paper-based exams (33%). Analysis on the level of individual questions showed that 90.8% of questions had one single picture. The remaining questions contained either more than one picture (2.9%), video (2.7%), audio (0.2%) or 3.3% of questions had picture as well as video added. The main question types used for items with one picture are TypeA (54%) and Long_Menu (31%). In contrast, questions with video content contain only 11% TypeA questions, whereas Long_Menu is represented by 66%. Nearly all questions containing both picture and video are Long_Menu questions. It can be stated that digital assessment formats are indeed on the raise. Moreover, our data indicates that electronic assessments formats have easier options to embed media items and thus show a higher frequency of media addition. We even identified the usage of different media types in the same question and this innovative item design could be a useful feature for the creation of medical assessments. Moreover, the choice of media type seems to depend on the respective question type.

https://link.springer.com/content/pdf/10.1186/s12909-020-02014-7.pdf

Konstantin Brass

Papers

Does medical students' preference of test format (computer-based vs. paper-based) have an influence on performance?

Electronic acquisition of OSCE performance using tablets.

Good exams made easy: The item management system for multiple examination formats

Impact of COVID-19 on digital medical education: compatibility of digital teaching and examinations with integrity and ethical principles

Medical assessment in the age of digitalisation