Medical educators attempt to create reliable and valid tests and questionnaires in order to enhance the accuracy of their assessment and evaluations. Validity and reliability are two fundamental elements in the evaluation of a measurement instrument. Instruments can be conventional knowledge, skill or attitude tests, clinical simulations or survey questionnaires. Instruments can measure concepts, psychomotor skills or affective values. Validity is concerned with the extent to which an instrument measures what it is intended to measure. Reliability is concerned with the ability of an instrument to measure consistently.1 It should be noted that the reliability of an instrument is closely associated with its validity. An instrument cannot be valid unless it is reliable. However, the reliability of an instrument does not depend on its validity.2 It is possible to objectively measure the reliability of an instrument and in this paper we explain the meaning of Cronbach’s alpha, the most widely used objective measure of reliability.

Calculating alpha has become common practice in medical education research when multiple-item measures of a concept or construct are employed. This is because it is easier to use in comparison to other estimates (e.g. test-retest reliability estimates)3 as it only requires one test administration. However, in spite of the widespread use of alpha in the literature the meaning, proper use and interpretation of alpha is not clearly understood. 2, 4, 5 We feel it is important, therefore, to further explain the underlying assumptions behind alpha in order to promote its more effective use. It should be emphasised that the purpose of this brief overview is just to focus on Cronbach’s alpha as an index of reliability. Alternative methods of measuring reliability based on other psychometric methods, such as generalisability theory or item-response theory, can be used for monitoring and improving the quality of OSCE examinations 6-10, but will not be discussed here.

What is Cronbach alpha?
Alpha was developed by Lee Cronbach in 195111 to provide a measure of the internal consistency of a test or scale; it is expressed as a number between 0 and 1. Internal consistency describes the extent to which all the items in a test measure the same concept or construct and hence it is connected to the inter-relatedness of the items within the test. Internal consistency should be determined before a test can be employed for research or examination purposes to ensure validity. In addition, reliability estimates show the amount of measurement error in a test. Put simply, this interpretation of reliability is the correlation of test with itself. Squaring this correlation and subtracting from 1.00 produces the index of measurement error. For example, if a test has a reliability of 0.80, there is 0.36 error variance (random error) in the scores (0.80×0.80 = 0.64; 1.00 – 0.64 = 0.36).12 As the estimate of reliability increases, the fraction of a test score that is attributable to error will decrease.2 It is of note that the reliability of a test reveals the effect of measurement error on the observed score of a student cohort rather than on an individual student. To calculate the effect of measurement error on the observed score of an individual student, the standard error of measurement must be calculated (SEM).13

If the items in a test are correlated to each other, the value of alpha is increased. However, a high coefficient alpha does not always mean a high degree of internal consistency. This is because alpha is also affected by the length of the test. If the test length is too short, the value of alpha is reduced.2, 14 Thus, to increase alpha, more related items testing the same concept should be added to the test. It is also important to note that alpha is a property of the scores on a test from a specific sample of testees. Therefore investigators should not rely on published alpha estimates and should measure alpha each time the test is administered.14


Use of Cronbach’s alpha
Improper use of alpha can lead to situations in which either a test or scale is wrongly discarded or the test is criticised for not generating trustworthy results. To avoid this situation an understanding of the associated concepts of internal consistency, homogeneity or unidimensionality can help to improve the use of alpha. Internal consistency is concerned with the interrelatedness of a sample of test items, whereas homogeneity refers to unidimensionality. A measure is said to be unidimensional if its items measure a single latent trait or construct. Internal consistency is a necessary but not sufficient condition for measuring homogeneity or unidimensionality in a sample of test items. 5, 15 Fundamentally, the concept of reliability assumes that unidimensionality exists in a sample of test items16 and if this assumption is violated it does cause a major underestimate of reliability. It has been well documented that a multidimensional test does not necessary have a lower alpha than a unidimensional test. Thus a more rigorous view of alpha is that it cannot simply be interpreted as an index for the internal consistency of a test. 5, 15, 17

Factor Analysis can be used to identify the dimensions of a test.18 Other reliable techniques have been used and we encourage the reader to consult the paper “Applied Dimensionality and Test Structure Assessment with the START-M Mathematics Test” and to compare methods for assessing the dimensionality and underlying structure of a test.19

Alpha, therefore, does not simply measure the unidimensionality of a set of items, but can be used to confirm whether or not a sample of items is actually unidimensional.5 On the other hand if a test has more than one concept or construct, it may not make sense to report alpha for the test as a whole as the larger number of questions will inevitable inflate the value of alpha. In principle therefore, alpha should be calculated for each of the concepts rather than for the entire test or scale. 2, 3 The implication for a summative examination containing heterogeneous, case-based questions is that alpha should be calculated for each case.

More importantly, alpha is grounded in the ‘tau equivalent model’ which assumes that each test item measures the same latent trait on the same scale. Therefore, if multiple factors/traits underlie the items on a scale, as revealed by Factor Analysis, this assumption is violated and alpha underestimates the reliability of the test.17 If the number of test items is too small it will also violate the assumption of tau-equivalence and will underestimate reliability.20 When test items meet the assumptions of the tau-equivalent model, alpha approaches a better estimate of reliability. In practice, Cronbach’s alpha is a lower-bound estimate of reliability because heterogeneous test items would violate the assumptions of the tau-equivalent model.5 If the calculation of “standardised item alpha” in SPSS is higher than “Cronbach’s alpha”, a further examination of the tau-equivalent measurement in the data may be essential.


Numerical values of alpha
As pointed out earlier, the number of test items, item inter-relatedness and dimensionality affect the value of alpha.5 There are different reports about the acceptable values of alpha, ranging from 0.70 to 0.95. 2, 21, 22 A low value of alpha could be due to a low number of questions, poor inter-relatedness between items or heterogeneous constructs. For example if a low alpha is due to poor correlation between items then some should be revised or discarded. The easiest method to find them is to compute the correlation of each test item with the total score test; items with low correlations (approaching zero) are deleted. If alpha is too high it may suggest that some items are redundant as they are testing the same question but in a different guise. A maximum alpha value of 0.90 has been recommended.14


Summary
High quality tests are important to evaluate the reliability of data supplied in an examination or a research study. Alpha is a commonly employed index of test reliability. Alpha is affected by the test length and dimensionality. Alpha as an index of reliability should follow the assumptions of the essentially tau-equivalent approach. A low alpha appears if these assumptions are not meet. Alpha does not simply measure test homogeneity or unidimensionality as test reliability is a function of test length. A longer test increases the reliability of a test regardless of whether the test is homogenous or not. A high value of alpha (> 0.90) may suggest redundancies and show that the test length should be shortened.

/pdf/making-sense-of-cronbach-s-alpha-2pncxqblel.pdf

Making sense of Cronbach's alpha

On Educational psychology: A cognitive view.

Background: Preparing healthcare professionals for teaching is regarded as essential to enhancing teaching effectiveness. Although many reports describe various faculty development interventions, there is a paucity of research demonstrating their effectiveness.Objective: To synthesize the existing evidence that addresses the question: “What are the effects of faculty development interventions on the knowledge, attitudes and skills of teachers in medical education, and on the institutions in which they work?”Methods: The search, covering the period 1980–2002, included three databases (Medline, ERIC and EMBASE) and used the keywords: staff development; in-service training; medical faculty; faculty training/development; continuing medical education. Manual searches were also conducted.Articles with a focus on faculty development to improve teaching effectiveness, targeting basic and clinical scientists, were reviewed. All study designs that included outcome data beyond participant satisfaction were accepted....

/pdf/a-systematic-review-of-faculty-development-initiatives-3otzax43oj.pdf

A systematic review of faculty development initiatives designed to improve teaching effectiveness in medical education: BEME Guide No. 8.

A systematic review of faculty development initiatives designed to improve teaching effectiveness in medical education: BEME Guide No. 8

Contents: J. Gale, Preface. Part I:Radical Constructivism and Social Constructionism. E. von Glasersfeld, A Constructivist Approach to Teaching. K.J. Gergen, Social Construction and the Educational Process. J. Shotter, In Dialogue: Social Constructionism and Radical Constructivism. J. Richards, Construct[ion/iv]ism: Pick One of the Above. Part II:Information-Processing Constructivism and Cybernetic Systems. F. Steier, From Universing to Conversing: An Ecological Constructionist Approach to Learning and Multiple Description. R.J. Spiro, P.J. Feltovich, M.J. Jacobson, R.L. Coulson, Cognitive Flexibility, Constructivism, and Hypertext: Random Access Instruction for Advanced Knowledge Acquisition in Ill-Structured Domains. K. Tomm, Response to Chapters by Spiro et al. and Steier. P.W. Thompson, Constructivism, Cybernetics, and Information Processing: Implications for Technologies of Research on Learning. Part III:Social Constructivism and Sociocultural Approaches. H. Bauersfeld, The Structuring of the Structures: Development and Function of Mathematizing as a Social Practice. J.V. Wertsch, C. Toma, Discourse and Learning in the Classroom: A Sociocultural Approach. C. Konold, Social and Cultural Dimensions of Knowledge and Classroom Teaching. J. Confrey, How Compatible Are Radical Constructivism, Sociocultural Approaches, and Social Constructivism? Analysis and Synthesis I: Alternative Epistemologies. M.H. Bickhard, World Mirroring Versus World Making: There's Gotta Be a Better Way. Part IV:Alternative Epistemologies in Language, Mathematics, and Science Education. R. Duit, The Constructivist View: A Fashionable and Fruitful Paradigm for Science Education Research and Practice. G.B. Saxe, From the Field to the Classroom: Studies in Mathematical Understanding. N.N. Spivey, Written Discourse: A Constructivist Perspective. T. Wood, From Alternative Epistemologies to Practice in Education: Rethinking What It Means to Teach and Learn. E. Ackermann, Construction and Transference of Meaning Through Form. D. Rubin, Constructivism, Sexual Harassment, and Presupposition: A (Very) Loose Response to Duit, Saxe, and Spivey. Part V:Alternative Epistemologies in Clinical, Mathematics, and Science Education. E. von Glasersfeld, Sensory Experience, Abstraction, and Teaching. R. Driver, Constructivist Approaches to Science Teaching. T. Wood, P. Cobb, E. Yackel, Reflections on Learning and Teaching Mathematics in Elementary School. P. Lewin, The Social Already Inhabits the Epistemic: A Discussion of Driver Wood, Cobb, and Yackel and von Glasersfeld. J. Becker, M. Varelas, Assisting Construction: The Role of the Teacher in Assisting the Learner's Construction of Preexisting Cultural Knowledge. E.H. Auerswald, Shifting Paradigms: A Self-Reflective Critique. Analysis and Synthesis II: Epsitemologies in Education. P. Ernest, The One and the Many. Analysis and Synthesis III: Retrospective Comments and Future Prospects. L.P. Steffe, Alternative Epistemologies: An Educator's Perspective. J. Gale, Epilogue.

教育中的建构主义 = Constructivism in Education

Assessing the Skills of Surgical Residents Using Simulation

One of the key goals of assessment in medical education is the minimisation of all errors influencing a test in order to produce an observed score which approaches a learner's 'true' score, as reliably and validly as possible. In order to achieve this, assessors need to be aware of the potential biases that can influence all components of the assessment cycle from question creation to the interpretation of exam scores. This Guide describes and explains the processes whereby objective examination results can be analysed to improve the validity and reliability of assessments in medical education. We cover the interpretation of measures of central tendency, measures of variability and standard scores. We describe how to calculate the item-difficulty index and item-discrimination index in examination tests using different statistical procedures. This is followed by an overview of reliability estimates. The post-examination analytical methods described in this guide enable medical educators to construct reliable and valid achievement tests. They also enable medical educators to develop question banks using the collection of appropriate questions from existing examination tests in order to use computerised adaptive testing.

Post-examination analysis of objective tests.

There is no single overarching theoretical framework that accounts for how we learn in all situations. There are many theories of how we learn from our experiences and there are many theories of what experiences are.  We know that we do learn and that we have knowledge but there is no consensus on the relationship between the mechanism by which our brains learn and the optimal way in which we should be taught. In other words there is no necessary connection between epistemology and pedagogy. We seem capable of learning from a wide variety of pedagogical processes and although there is empirical evidence that some ways of learning are more effective than others in specific situations no one pedagogical method dominates. However, reflecting on twenty five years as a medical educationalist I have come to the conclusion that there is one framework that makes more connections between different epistemological and pedagogical theories than others and that could have support from neuroscience. I wish to explore the connections between a variety of educational, communication and psychotherapeutic processes and try to show that constructivism holds the promise of providing some unity to the practice of education and learning.

The constructivist model
The constructivist theory of learning, whose philosophical origins are frequently ascribed to Kant and whose educational origins to Piaget, is based on the premise that the act of learning is based on a process which connects new knowledge to pre-existing knowledge. I do not intend to go into all the arguments surrounding this theory and to describe the variety of constructivist models that have been created. The reader is referred to the extensive literature on the subject.1-6    For the purposes of my argument it is simply necessary to acknowledge some basic assumptions underpinning the theory from which important connections to other educational theories can be made as well as to therapeutic practices, neuroscience and even the nature of scientific knowledge.

Possibly the most well-known articulation of the underlying assumption of constructivism is the famous quotation of Ausubel: “The most important factor influencing learning is what the learner already knows”.7 In many years of teaching teachers how to teach I have  found many people have an intuitive grasp of this fact and tailor their teaching to take account of the background knowledge of their students. Piaget expressed it differently by stressing that experience is constantly being ‘assimilated’ or filtered through pre-existing concepts.8    New knowledge is therefore interpreted by existing knowledge and then connected to existing knowledge. The implications of this conception are manifold and spread beyond conventional education into interpersonal communication and psychotherapy. In addition this phenomenon begs the question of how the process is conducted and manifested in the brains of individuals.

The statement of Ausubel7 automatically leads to a pedagogical intervention even if it is merely ascertaining the prior knowledge of a learner by asking questions. Piaget avoided suggesting detailed pedagogy but certain ways of teaching flow from his assumptions. He stressed that learning was fundamentally about interacting with the world in order to explore the ‘rules of the game’ and to discover the causal relationships between events.  This leads to an active pedagogy involving exploration, experience and experimentation. Dewey aptly summarized this idea by saying that learners should be actors rather than spectators.9 Others have extended this concept to suggest that individals  interact with the world in order to extract meaning from it and to construct a coherent and consistent cognitive model. Another way of saying this is that our brains are programmed to support our survival. From an evolutionary perspective organisms have evolved brains which create and internalize an increasingly complex and accurate model of the world. Based on our experience of interacting with the world we have each created our own unique model of reality. Nevertheless we are social beings and we cannot ignore the power of social interactivity that has influenced this process, a concept emphasised by the Russian psychologist Vygotsky.10 Other human beings influence the way in which we construct our models giving rise to the teacher and to the various social processes of pedagogy.

Model building in the individual,  based on interactivity with the world, is the result of a cognitive process which involves the experience of the world being assimilated and filtered through prior knowledge as previously described. If sense or meaning can be attached to the experience then the experience fits with existing cognitive structures (Piaget’s ‘schemas’).  However, if the experience does not make sense then a feeling of dissatisfaction or cognitive dissonance can ensue in which the individual will seek to reach some sort of mental equilibrium by exploration or questioning. At this point the individual may make use of the faculty of imagination to suggest or hypothesise why the experience is problematic. It is the existence of human imagination, the ability of each of us to interrogate our mental model, to speculate and to ask questions that is one of the  hallmarks of our intelligence and our capacity to survive. Albert Einstein famously stated that ‘imagination is more important than knowledge’ since he realized that ultimately all knowledge is derived from an initial process of imagination that is subsequently tested against the world.  As individuals we can simulate reality by asking ourselves ‘what if?’ questions.  We can perform acts in our imagination before doing them in reality and risking having our genes deleted from the gene pool.11 Our imagined conjectures can be tested by seeking other experiences and by so doing we can resolve the dissonance and elaborate our learning and our mental model.

/pdf/constructivism-reflections-on-twenty-five-years-teaching-the-4fzhc0uqnz.pdf

Constructivism: reflections on twenty five years teaching the constructivist approach in medical education.

The importance of empathetic communication in the context of patient care has been suggested as a crucial element in the delivery of high-quality medical care. Although some countries have measured empathy among medical students, little is known formally about the state of affairs regarding empathy in the UK with regard to gender, medical years and speciality. This paper presents the results from a cross-sectional study that explored the relationship between undergraduate medical students' empathy scores relevant to gender, medical school year and future career ambitions. The Jefferson Scale of Physician Empathy, a self-administered questionnaire, was used for collecting data. It was decided to incorporate the scale into a web application that was only accessible through a confidential hyperlink. A total of 853 (68.2%) medical students completed the survey. Female students (n=470) scored significantly (t((819))=-6.63; 95% CI=-0.30, -0.16) higher than male students (n=351). No significant differences were observed between empathy mean scores and medical school year. Students choosing people-oriented specialities had significantly higher mean scores in comparison to students preferring technology oriented specialities. Three hypotheses were tested in this study. The hypothesis that female students would score higher than male students was supported by the results. The findings did not support the hypothesis that empathy scores decline during the medical school years. Finally, the findings did support the hypothesis that those preferring peopleoriented specialities would score higher on the empathy scale than students choosing technology oriented specialities. Some recommendations are made and the limitations of study are discussed.

Reg Dennick

Papers

Making sense of Cronbach's alpha

Assessing the Skills of Surgical Residents Using Simulation

Post-examination analysis of objective tests.

Constructivism: reflections on twenty five years teaching the constructivist approach in medical education.

Empathy in UK medical students: differences by gender, medical year and specialty interest.