Intra-rater reliability

About: Intra-rater reliability is a research topic. Over the lifetime, 2073 publications have been published within this topic receiving 140968 citations.

An assessment of the hopping strategy and inter-limb asymmetry during the triple hop test: a test–retest pilot study

Abstract: The aims of the present study are to: (1) determine within- and between-session reliability of multiple metrics obtained during the triple hop test; and (2) determine any systematic bias in both the test and inter-limb asymmetry scores for these metrics. Thirteen male young American football athletes performed three trials of a triple hop test on each leg on two separate occasions. In addition to the total distance hopped, manual detection of touch down and toe-off were calculated via video analysis, enabling flight time (for each hop), ground contact time (GCT), reactive strength index (RSI), and leg stiffness (between hops) to be calculated. Results showed all coefficient of variation (CV) values were ≤ 10.67% and intraclass correlation coefficients (ICC) ranged from moderate to excellent (0.53–0.95) in both test sessions. Intrarater reliability showed excellent reliability for all metrics (CV ≤ 3.60%, ICC ≥ 0.97). No systematic bias was evident between test sessions for raw test scores (g = −0.34 to 0.32) or the magnitude of asymmetry (g = −0.19 to 0.43). However, ‘real’ changes in asymmetry (i.e., greater than the CV in session 1) were evident on an individual level for all metrics. For the direction of asymmetry, kappa coefficients revealed poor-to-fair levels of agreement between test sessions for all metrics (K = −0.10 to 0.39), with the exception of the first hop (K = 0.69). These data show that, given the inherent limitations of distance jumped in the triple hop test, practitioners can confidently gather a range of reliable data when computed manually, provided sufficient test familiarization is conducted. In addition, although the magnitude of asymmetry appears to show only small changes between test sessions, limb dominance does appear to fluctuate between test sessions, highlighting the value of also monitoring the direction of the imbalance.

Intra- and Interobserver Reliability of Three Classification Systems for Hallux Rigidus

Abstract: BACKGROUND There are over ten classification systems currently used in the staging of hallux rigidus. This results in confusion and inconsistency with radiographic interpretation and treatment. The reliability of hallux rigidus classification systems has not yet been tested. The purpose of this study was to evaluate intra- and interobserver reliability using three commonly used classifications for hallux rigidus. METHODS Twenty-one plain radiograph sets were presented to ten ACFAS board-certified foot and ankle surgeons. Each physician classified each radiograph based on clinical experience and knowledge according to the Regnauld, Roukis, and Hattrup and Johnson classification systems. The two-way mixed single-measure consistency intraclass correlation was used to calculate intra- and interrater reliability. RESULTS The intrarater reliability of individual sets for the Roukis and Hattrup and Johnson classification systems was "fair to good" (Roukis, 0.62±0.19; Hattrup and Johnson, 0.62±0.28), whereas the intrarater reliability of individual sets for the Regnauld system bordered between "fair to good" and "poor" (0.43±0.24). The interrater reliability of the mean classification was "excellent" for all three classification systems. Conclusions Reliable and reproducible classification systems are essential for treatment and prognostic implications in hallux rigidus. In our study, Roukis classification system had the best intrarater reliability. Although there are various classification systems for hallux rigidus, our results indicate that all three of these classification systems show reliability and reproducibility.

Modeling Worker Performance Based on Intra-rater Reliability in Crowdsourcing : A Case Study of Speech Quality Assessment

Abstract: Crowdsourcing has become a convenient instrument for addressing subjective user studies to a large amounts of users. Data from crowdsourcing can be corrupted due to users’ neglect, and different mechanisms has been proposed to address the users’ reliability and to ensure valid experiments’ results. Users that are consistent in their answers or present a high intra-rater reliability score, are desired for subjective studies. This work investigates the relationship between the intra-rater reliability and the user performance in the context of a speech quality assessment task. To this end, a crowdsourcing study has been conducted in which users were requested to rate speech stimuli with respect to their overall quality. Ratings were collected on a 5-point scale in accordance with the ITU-T Rec. P.808. The speech stimuli were taken from the database ITU-T Rec. P.501 Annex D, and the results are to be contrasted with ratings collected in a laboratory experiment. Furthermore, a model as a function of intra-rater reliability, root-mean-squared-deviation between the listeners ratings and age, has been built to predict the listener performance. Such a model is intended to provide a measure of how valid the crowdsourcing results are, when there is no laboratory results to compare to.