Intra-rater reliability

About: Intra-rater reliability is a research topic. Over the lifetime, 2073 publications have been published within this topic receiving 140968 citations.

A new, validated instrument to evaluate competency in microsurgery: the University of Western Ontario Microsurgical Skills Acquisition/Assessment instrument [outcomes article].

Abstract: BACKGROUND The authors present a model for microsurgery learning as well as a validated instrument to evaluate microsurgical competency. METHODS Novice microsurgeons participated in three 3-hour sessions wherein they completed a number of increasingly complex, standardized microsurgical tasks. Performance was recorded and graded using a newly developed University of Western Ontario Microsurgery Skills Acquisition/Assessment (UWOMSA) instrument. The knot-tying and anastomosis modules contained three categories with five-point Likert scales. Each learner's performance was assessed by two blinded surgeons. Reznick's validated global rating scale for operative performance was utilized to establish criterion validity. Within-scale scores were compared via intraclass correlation and between-scale scores with Pearson correlation coefficient. Linear regression was used to evaluate the effect of various predictors on UWOMSA scores. RESULTS Thirty-seven videos (9.6 hours) were reviewed, including 20 knot-tying sessions and 17 anastomoses. Interrater reliability of UWOMSA was high, with an intraclass correlation coefficient of 0.75 (0.57, 0.87). The intraclass correlation of the global rating scale was 0.79 (0.62, 0.89). Intrarater reliability of the UWOMSA was also high, with an intraclass correlation of 0.69 (0.48, 0.83). The intraclass correlation of the global rating scale was 0.69 (0.47, 0.84). Measures of criterion validity demonstrated strong agreement between UWOMSA and the global rating scale (Pearson correlation coefficient, 0.96; p < 0.001). Measures of construct validity demonstrated that higher scores on the UWOMSA were associated with faster knot tying (p < 0.0001) and higher postgraduate year level (p = 0.05). CONCLUSIONS The UWOMSA instrument performed well in terms of reliability and validity. Further study is planned to assess the instrument's ability to predict microsurgical skills translation to the clinical setting.

Measurement of lower-limb muscle spasticity: intrarater reliability of Modified Modified Ashworth Scale.

Abstract: The Modified Modified Ashworth Scale (MMAS) is a clinical instrument for measuring spasticity. Few studies have been performed on the reliability of the MMAS. The aim of the present study was to investigate the intrarater reliability of the MMAS for the assessment of spasticity in the lower limb. We conducted a test-retest study on spasticity in the hip adductors, knee extensors, and ankle plantar flexors. Each patient was measured by a hospital-based clinical physiotherapist. Twenty-three patients with stroke or multiple sclerosis (fourteen women, nine men) and a mean +/- standard deviation age of 37.3 +/- 14.1 years participated. The weighted kappa was moderate for the hip adductors (weighted kappa = 0.45, standard error [SE] = 0.16, p = 0.007), good for the knee extensors (weighted kappa = 0.62, SE = 0.12, p < 0.001), and very good for the ankle plantar flexors (weighted kappa = 0.85, SE = 0.05, p < 0.001). The kappa value for overall agreement was very good (weighted kappa = 0.87, SE = 0.03, p < 0.001). The reliability for the ankle plantar flexors was significantly higher than that for the hip adductors. The intrarater reliability of the MMAS in patients with lower-limb muscle spasticity was very good, and it can be used as a measure of spasticity over time.

Interobserver Reliability of Quantitative Muscle Sonographic Analysis in the Critically Ill Population

Abstract: Objectives There is growing interest in the use of quantitative high-resolution neuromuscular sonography to evaluate skeletal muscles in patients with critical illness. There is currently considerable methodological variability in the measurement technique of quantitative muscle analysis. The reliability of muscle parameters using different measurement techniques and assessor expertise levels has not been examined in patients with critical illness. The primary objective of this study was to determine the interobserver reliability of quantitative sonographic measurement analyses (thickness and echogenicity) between assessors of different expertise levels and using different techniques for selecting the region of interest. Methods We conducted a cross-sectional observational study in neurocritical care and mixed surgical-medical intensive care units from 2 tertiary referral hospitals. Results Twenty diaphragm and 20 quadriceps images were evaluated. Images were obtained by using standardized imaging acquisition techniques. Quantitative sonographic measurements included muscle thickness and echogenicity analysis (either by the trace or square technique). All images were analyzed twice independently by 4 assessors of differing expertise levels. Excellent interobserver reliability was obtained for all measurement techniques regardless of expertise level (intraclass correlation coefficient, >0.75 for all comparisons). There was less variability between assessors for echogenicity values when the square technique was used for the quadriceps muscle and the trace technique for the diaphragm. Conclusions Excellent interobserver reliability exists regardless of expertise level for quantitative analysis of muscle parameters on sonography in the critically ill population. On the basis of these findings, it is recommended that echogenicity analysis be performed using the square technique for the quadriceps and the trace technique for the diaphragm.

A meta-analysis of job analysis reliability.

Abstract: Average levels of interrater and intrarater reliability for job analysis data were investigated using meta-analysis. Forty-six studies and 299 estimates of reliability were cumulated. Data were categorized by specificity (generalized work activity or task data), source (incumbents, analysts, or technical experts), and descriptive scale (frequency, importance, difficulty, time-spent, and the Position Analysis Questionnaire). Task data initially produced higher estimates of interrater reliability than generalized work activity data and lower estimates of intrarater reliability. When estimates were corrected for scale length and number of raters by using the Spearman-Brown formula, task data had higher interrater and intrarater reliabilities. Incumbents displayed the lowest reliabilities. Scales of frequency and importance were the most reliable. Implications of these reliability levels for job analysis practice are discussed.

Reliability of Assessing Percentage of Diffusion-Perfusion Mismatch

Abstract: Background and Purpose— Emergent neurovascular imaging holds promise in identifying new and optimum target populations for thrombolysis in stroke. Recent research has focused on patients with diffusion-weighted MRI (DWI)-perfusion-weighted MRI (PWI) mismatch as a marker of tissue at risk of infarction and a means to select the most suitable candidates for thrombolysis. The present study sought to estimate the reliability of assessing the percentage of DWI-PWI mismatch. Methods— Thirteen patients with acute strokes had DWI and PWI within 7 hours of symptom onset. Six raters independently created relative mean transit time (rMTT) maps and then compared them with DWI images to assess the percentage of mismatch (PWI>DWI) in 10% increments. The MR scans were reassessed by 4 raters, tracing around the lesions to calculate the volume percentage of mismatch. Results— Visual assessment had an interrater reliability of 0.68 (95% CI, 0.52 to 1.0; SEM=21.6%) and an intrarater reliability of 0.80 (95% CI, 0.47 to 1.0;...