Intra-rater reliability

About: Intra-rater reliability is a research topic. Over the lifetime, 2073 publications have been published within this topic receiving 140968 citations.

The Definition and Estimation of Test Reliability

Abstract: empirical judgment, and different authors have given conflicting definitions and conflicting criteria to guide empirical judgment. Under several of these definitions, however, it can be shown that the reliability coefficient is the variance ratio of true scores to total scores on each of the equivalent forms. The present writer prefers to reverse this procedure, defining the reliability coefficient of any one form of a test as the variance ratio of true scores to total scores. This shifts the ground of a difficulty indicated by Kelly [6], that under the correlation definition we cannot discuss directly the reliability of one form of a test. Under the proposed definition such discussion is entirely possible so long as it remains formal, and we see that the second form is needed only when we come to the estimation of the

Interrater and intrarater reliability of the Knosp scale for pituitary adenoma grading.

Abstract: OBJECTIVE The goal of this study was to determine the interrater and intrarater reliability of the Knosp grading scale for predicting pituitary adenoma cavernous sinus (CS) involvement. METHODS Six independent raters (3 neurosurgery residents, 2 pituitary surgeons, and 1 neuroradiologist) participated in the study. Each rater scored 50 unique pituitary MRI scans (with contrast) of biopsy-proven pituitary adenoma. Reliabilities for the full scale were determined 3 ways: 1) using all 50 scans, 2) using scans with midrange scores versus end scores, and 3) using a dichotomized scale that reflects common clinical practice. The performance of resident raters was compared with that of faculty raters to assess the influence of training level on reliability. RESULTS Overall, the interrater reliability of the Knosp scale was "strong" (0.73, 95% CI 0.56-0.84). However, the percent agreement for all 6 reviewers was only 10% (26% for faculty members, 30% for residents). The reliability of the middle scores (i.e., average rated Knosp Grades 1 and 2) was "very weak" (0.18, 95% CI -0.27 to 0.56) and the percent agreement for all reviewers was only 5%. When the scale was dichotomized into tumors unlikely to have intraoperative CS involvement (Grades 0, 1, and 2) and those likely to have CS involvement (Grades 3 and 4), the reliability was "strong" (0.60, 95% CI 0.39-0.75) and the percent agreement for all raters improved to 60%. There was no significant difference in reliability between residents and faculty (residents 0.72, 95% CI 0.55-0.83 vs faculty 0.73, 95% CI 0.56-0.84). Intrarater reliability was moderate to strong and increased with the level of experience. CONCLUSIONS Although these findings suggest that the Knosp grading scale has acceptable interrater reliability overall, it raises important questions about the "very weak" reliability of the scale's middle grades. By dichotomizing the scale into clinically useful groups, the authors were able to address the poor reliability and percent agreement of the intermediate grades and to isolate the most important grades for use in surgical decision making (Grades 3 and 4). Authors of future pituitary surgery studies should consider reporting Knosp grades as dichotomized results rather than as the full scale to optimize the reliability of the scale.

Validation of a grading system for lateral nasal wall insufficiency.

Abstract: This study was designed to validate a grading scheme for lateral nasal wall insufficiency with interrater and intrarater reliability measures. Representative endoscopic videos depicting varied degrees of lateral nasal wall insufficiency were collated into a 30-clip video (15 clips in duplicate). This was rated by five reviewers for a total of 150 observations. Interrater and intrarater reliability were determined using Fleiss kappa and intraclass correlation coefficient (ICC) statistics, respectively. Good agreement was established between reviewers (interrater reliability), with a Fleiss kappa of 0.7733 (p < 0.01). Analysis of intrarater variability with the ICC revealed a very strong agreement (ICC = 0.88; p < 0.01). The proposed grading system is shown to have good interrater and intrarater reliability. It provides a reliable instrument for assessing lateral wall insufficiency.