Intra-rater reliability

About: Intra-rater reliability is a research topic. Over the lifetime, 2073 publications have been published within this topic receiving 140968 citations.

Simplified skeletal maturity scoring system: learning curve and methods to improve reliability.

Abstract: STUDY DESIGN Retrospective radiographical review by 5 independent observers. OBJECTIVE To validate the intra- and interobserver reliability of the simplified skeletal maturity scoring (SSMS) system in a large cohort for each stage and for the overall cohort. SUMMARY OF BACKGROUND DATA The SSMS has been used to successfully predict curve progression in idiopathic scoliosis. METHODS A total of 275 patients with scoliosis (8-16 yr) with 1 hand radiograph were included from 2005 to 2011. Five participants independently scored images on 2 separate occasions using the SSMS (stage, 1-8). Observers (listed in order of increasing SSMS experience) included orthopedic surgery resident, clinical fellow (CF), research fellow, and senior faculty. Intraobserver agreement between the 2 sets of scores was estimated using the Pearson and Spearman rank correlation coefficients. Interobserver agreement was estimated with the unweighted Fleiss κ coefficient for the overall cohort and for junior (orthopedic surgery resident, CF, research fellow) versus senior faculty. RESULTS Intrarater reliability for orthopedic surgery resident, CF, research fellow, senior faculty was 0.956, 0.967, 0.986, 0.991, and 0.998, respectively (Spearman). Intrarater agreement improved with greater familiarity using the SSMS. The inter-rater reliability for junior faculty (κ = 0.65), senior faculty (κ = 0.652), and the overall group (κ = 0.66) indicated agreement between all observers but no improved inter-rater agreement with experience. However, 98% of disagreements occurred only within 1 stage. Stages 2, 3, and 4 accounted for most of the variability; stage 3 was the most commonly scored stage, corresponding to peak growth velocity. CONCLUSION The SSMS has excellent intraobserver agreement with substantial interobserver agreement. Intraobserver--but not interobserver agreement--improves with familiarity using the SSMS. Expectancy bias may contribute to a higher likelihood of assigning an SSMS 3. Discrepancies when classifying stages 2 to 4 may be resolved by improved descriptions of epiphyseal capping in stages 2 and 3. LEVEL OF EVIDENCE 2.