Analyzing and Interpreting Data From Likert-Type Scales
01 Dec 2013-Journal of Graduate Medical Education (Accreditation Council for Graduate Medical Education)-Vol. 5, Iss: 4, pp 541-542
TL;DR: Reading this article is to provide readers who do not have extensive statistics background with the basics needed to understand the basics of Likert scale concepts.
Abstract: Likert-type scales are frequently used in medical education and medical education research. Common uses include end-of-rotation trainee feedback, faculty evaluations of trainees, and assessment of performance after an educational intervention. A sizable percentage of the educational research manuscripts submitted to the Journal of Graduate Medical Education employ a Likert scale for part or all of the outcome assessments. Thus, understanding the interpretation and analysis of data derived from Likert scales is imperative for those working in medical education and education research. The goal of this article is to provide readers who do not have extensive statistics background with the basics needed to understand these concepts.
Developed in 1932 by Rensis Likert1 to measure attitudes, the typical Likert scale is a 5- or 7-point ordinal scale used by respondents to rate the degree to which they agree or disagree with a statement (table). In an ordinal scale, responses can be rated or ranked, but the distance between responses is not measurable. Thus, the differences between “always,” “often,” and “sometimes” on a frequency response Likert scale are not necessarily equal. In other words, one cannot assume that the difference between responses is equidistant even though the numbers assigned to those responses are. This is in contrast to interval data, in which the difference between responses can be calculated and the numbers do refer to a measureable “something.” An example of interval data would be numbers of procedures done per resident: a score of 3 means the resident has conducted 3 procedures. Interestingly, with computer technology, survey designers can create continuous measure scales that do provide interval responses as an alternative to a Likert scale. The various continuous measures for pain are well-known examples of this (figure 1).
FIGURE 1
Continuous Measure Example
TABLE
Typical Likert Scales
Citations
More filters
TL;DR: A systematic, seven-step process for designing high-quality questionnaires, with particular emphasis on developing survey scales is presented, which synthesize multiple survey design techniques and organize them into a cohesive process for questionnaire developers of all levels.
Abstract: In this AMEE Guide, we consider the design and development of self-administered surveys, commonly called questionnaires. Questionnaires are widely employed in medical education research. Unfortunately, the processes used to develop such questionnaires vary in quality and lack consistent, rigorous standards. Consequently, the quality of the questionnaires used in medical education research is highly variable. To address this problem, this AMEE Guide presents a systematic, seven-step process for designing high-quality questionnaires, with particular emphasis on developing survey scales. These seven steps do not address all aspects of survey design, nor do they represent the only way to develop a high-quality questionnaire. Instead, these steps synthesize multiple survey design techniques and organize them into a cohesive process for questionnaire developers of all levels. Addressing each of these steps systematically will improve the probabilities that survey designers will accurately measure what they intend to measure.
685 citations
Cites methods from "Analyzing and Interpreting Data Fro..."
...When attempting to assess hard-to-measure educational constructs such as motivation, confidence and satisfaction, it usually makes sense to create a composite score for each survey scale than it does to use individual survey items as variables (Sullivan & Artino 2013)....
[...]
TL;DR: In this article, two datasets were examined regarding the use and perceptions of students and teachers on the use of digital tools, with the Learning Management System being perceived as the most useful tool.
Abstract: Digitalization in Higher Education (HE) institutions is an issue that concerns many educational stakeholders. ICT skills are becoming increasingly relevant in every context, especially in the workplace, therefore one of the prime objectives for universities has become preparing future professionals to be able to deal with problems and search for solutions, including digital competence as a vital skill set. Different policies, initiatives and strategies are currently being proposed in Germany, addressing educational technology innovations in HE. The University of Oldenburg is presented as an example, in an endeavour to gain an understanding of what is being proposed and what is actually happening in teaching and learning in German university classrooms. Two datasets were examined regarding the use and perceptions of students (n = 200) and teachers (n = 381) on the use of digital tools. Findings reveal that both teachers and students use a limited number of digital technology for predominantly assimilative tasks, with the Learning Management System being perceived as the most useful tool. In order to support the broader use of educational technology for teaching and learning purposes, strategies for HE institutions are suggested.
259 citations
TL;DR: Recommendations are provided to practitioners that desire to apply normalisation and weighting as well as to developers of the underlying methods for the identification of specific advantages and limitations.
Abstract: Building on the rhetoric question “quo vadis?” (literally “Where are you going?”), this article critically investigates the state of the art of normalisation and weighting approaches within life cycle assessment. It aims at identifying purposes, current practises, pros and cons, as well as research gaps in normalisation and weighting. Based on this information, the article wants to provide guidance to developers and practitioners. The underlying work was conducted under the umbrella of the UNEP-SETAC Life Cycle Initiative, Task Force on Cross-Cutting issues in life cycle impact assessment (LCIA). The empirical work consisted in (i) an online survey to investigate the perception of the LCA community regarding the scientific quality and current practice concerning normalisation and weighting; (ii) a classification followed by systematic expert-based assessment of existing methods for normalisation and weighting according to a set of five criteria: scientific robustness, documentation, coverage, uncertainty and complexity. The survey results showed that normalised results and weighting scores are perceived as relevant for decision-making, but further development is needed to improve uncertainty and robustness. The classification and systematic assessment of methods allowed for the identification of specific advantages and limitations. Based on the results, recommendations are provided to practitioners that desire to apply normalisation and weighting as well as to developers of the underlying methods.
177 citations
Cites methods from "Analyzing and Interpreting Data Fro..."
...Parametric tests can be successfully applied on Likert data when their distribution approximates normal (Norman 2010; Sullivan and Artino 2013), but their use is still debated and therefore the non-parametric versions of these tests were also performed, which essentially provided the same results…...
[...]
...Parametric tests can be successfully applied on Likert data when their distribution approximates normal (Norman 2010; Sullivan and Artino 2013), but their use is still debated and therefore the non-parametric versions of these tests were also performed, which essentially provided the same results (see Electronic Supplementary Material)....
[...]
TL;DR: The study aims at developing a social criteria framework, supported by categories and indicators to evaluate sustainable development in buildings, and will contribute to building practitioners in order to evaluate building projects socially, with a focus to attainustainable development in the built environment.
105 citations
TL;DR: Women surgeons are more likely to be partnered with a full-time working spouse and to be primarily responsible for managing their households, and additional consideration for improvement in recruitment and retention strategies for surgeons might address barriers to equalizing these gender disparities.
98 citations
Cites methods from "Analyzing and Interpreting Data Fro..."
...Using parametric test in survey responses that utilize a Likert scale is well established in the current literature.(8) The number of publications exhibited substantial deviation from a normal distribution....
[...]
References
More filters
Book•
01 Jan 1932
TL;DR: The instrument to be described here is not, however, indirect in the usual sense of the word; it does not seek responses to items apparently unrelated to the attitudes investigated, and seeks to measure prejudice in a manner less direct than is true of the usual prejudice scale.
Abstract: THIS paper describes a technique which has been developed for the measurement of race prejudice. This technique differs from most prejudice inventories in that it avoids the following assumptions: (a) that the individual can say, to his own or the investigator's satisfaction, "This is how prejudiced I am," and (b) that, to the extent that the individual can accurately assess his degree of antipathy, he will report honestly the findings of such introspection. Most sociologists would perhaps agree that race attitudes rarely reside on a completely articulate level. Even where the individual holds to intellectual or ideological convictions which would seem to leave no room for out-group antipathies, such do persevere. Thus, we may expect the number of Americans who honestly think themselves "unprejudiced" to be considerably larger than effective research would reveal. Moreover, the number who present themselves as unprejudiced probably exceeds considerably the number who honestly, though often inaccurately, see themselves in this light. Most indirect techniques for the measurement of attitudes have their rationale in observations such as these. The instrument to be described here is not, however, indirect in the usual sense of the word; it does not seek responses to items apparently unrelated to the attitudes investigated. We do, however, seek to measure prejudice in a manner less direct than is true of the usual prejudice scale. In our instrument we seek to measure anti-Negro prejudice. Persons are called upon to respond on social distance scales to whites and Negroes who occupy a variety of occupational positions. The measure of prejudice is derived through the summation of the differences in distance responses to Negroes as opposed to whites in the same occupations. Thus, for lack of a better label,
12,492 citations
TL;DR: It is shown that many studies, dating back to the 1930s consistently show that parametric statistics are robust with respect to violations of these assumptions, and parametric methods can be utilized without concern for “getting the wrong answer”.
Abstract: Reviewers of research reports frequently criticize the choice of statistical methods. While some of these criticisms are well-founded, frequently the use of various parametric methods such as analysis of variance, regression, correlation are faulted because: (a) the sample size is too small, (b) the data may not be normally distributed, or (c) The data are from Likert scales, which are ordinal, so parametric statistics cannot be used. In this paper, I dissect these arguments, and show that many studies, dating back to the 1930s consistently show that parametric statistics are robust with respect to violations of these assumptions. Hence, challenges like those above are unfounded, and parametric methods can be utilized without concern for “getting the wrong answer”.
3,200 citations
TL;DR: I have recently used Likert-type rating scales to measure student views on various educational interventions, providing a range of responses to a given question or statement.
Abstract: Dipping my toe into the water of educational research, I have recently used Likert-type rating scales to measure student views on various educational interventions. Likert scales are commonly used to measure attitude, providing a range of responses to a given question or statement . Typically, there are 5 categories of response, from (for example) 1 1⁄4 strongly disagree to 5 1⁄4 strongly agree, although there are arguments in favour of scales with 7 or with an even number of response categories.
2,289 citations
TL;DR: Most recently in this journal, Jamieson outlined the view that ‘Likert scales’ are ordinal in character and that they must be analysed using non-parametric statistics, which are less sensitive and less powerful than parametric statistics and are more likely to miss weaker or emerging findings.
Abstract: How Likert type measurement scales should be appropriately used and analysed has been debated for over 50 years, often to the great confusion of students, practitioners, allied health researchers and educators. Basically, there are two major competing views that have evolved somewhat independently of one another and of the associated empirical research literature on this ‘great debate’. Most recently in this journal, Jamieson outlined the view that ‘Likert scales’ are ordinal in character (i.e., produce rank order data) and that they, therefore, must be analysed using non-parametric statistics. Non-parametric statistics, however, are less sensitive and less powerful than parametric statistics and are, therefore, more likely to miss weaker or emerging findings.
779 citations
TL;DR: The aim of the present editorial is to outline a systematic process for developing and collecting reliability and validity evidence for survey instruments used in GME and GME research.
Abstract: Surveys are frequently used in graduate medical education (GME). Examples include resident satisfaction surveys, resident work-hour questionnaires, trainee self-assessments, and end-of-rotation evaluations. Survey instruments are also widely used in GME research. A review of the last 7 issues of JGME indicates that of the 64 articles categorized as Original Research, 50 (77%) included surveys as part of the study design.
Despite the many uses of surveys in GME, the medical education literature provides limited guidance on survey design,1 and many surveys fail to use a rigorous methodology or best practices in survey design.2 As a result, the reliability and validity of many medical education surveys are uncertain. When surveys are not well designed, the data obtained from them may not be reproducible and may fail to capture the essence of the attitude, opinion, or behavior the survey developer is attempting to measure. A plethora of factors affecting reliability and validity in surveys includes, but is not limited to, poor question wording, confusing question layout, and inadequate response options. Ultimately, these problems negatively impact the reliability and validity of survey data, making it difficult to draw useful conclusions.3,4 With these problems in mind, the aim of the present editorial is to outline a systematic process for developing and collecting reliability and validity evidence for survey instruments used in GME and GME research.
The term survey is quite broad and could include questions used in a phone interview, the set of items used in a focus group, and the items on a self-administered patient survey. In this editorial, we limit our discussion to self-administered surveys, which are also sometimes referred to as questionnaires. The goals of any good questionnaire should be to develop a set of items that every respondent will interpret the same way, respond to accurately, and be willing and motivated to answer. The 6 questions below, although not intended to address all aspects of survey design, are meant to help guide the novice survey developer through the survey design process. Addressing each of these questions systematically will optimize the quality of GME surveys and improve the chances of collecting survey data with evidence of reliability and validity. A graphic depiction of the process described below is presented in the figure.
FIGURE
A Systematic Approach to Survey Design for Graduate Medical Education Research
123 citations