Showing papers on "Intra-rater reliability published in 2009"

Reliability and Validity

Abstract: Reliability and its antonym unreliability are related to the consistency with which observations can be measured and recorded. Validity depends on reliable observations but addresses the user confidence in or credibility of a study. Both notions are evaluated with respect to a relative notion of ‘truth’ that requires a specification of against what reference we are evaluating observations or conclusions. Methods to evaluate reliability and validity depend on the research purpose, assumptions, and methodologies and may involve quantitative as well as qualitative assessments.

The reliability of an instrumented device for measuring components of the star excursion balance test.

Abstract: BACKGROUND The Star Excursion Balance Test (SEBT) is a dynamic test that requires strength, flexibility, and proprioception and has been used to assess physical performance, identify chronic ankle instability, and identify athletes at greater risk for lower extremity injury. In order to improve the repeatability in measuring components of the SEBT, the Y Balance Test™ has been developed. OBJECTIVE The purpose of this paper is to report the development and reliability of the Y Balance Test™. METHODS Single limb stance excursion distances were measured using the Y Balance Test™ on a sample of 15 male collegiate soccer players. Intraclass Correlation Coefficients (ICC) were used to determine the reliability of the test. RESULTS The ICC for intrarater reliability ranged from 0.85 to 0.91 and for interrater reliability ranged from 0.99 to 1.00. Composite reach score reliability was 0.91 for intrarater and 0.99 for interrater reliability. DISCUSSION This study demonstrated that the Y Balance Test™ has good to excellent intrarater and interrater reliability. The device and protocol attempted to address the common sources of error and method variation in the SEBT including whether touch down is allowed with the reach foot, where the stance foot is aligned, movement allowed of the stance foot, instantaneous measurement of furthest reach distance, standard reach height from the ground, standard testing order, and well defined pass/fail criteria. CONCLUSION The Y Balance Test™ is a reliable test for measuring single limb stance excursion distances while performing dynamic balance testing in collegiate soccer players.

Intrarater and interrater reliability of the Balance Error Scoring System (BESS).

Abstract: Objective The purpose of this study was to determine the intrarater and interrater reliability of the Balance Error Scoring System (BESS). Design A prospective observational study. Setting Academic sports medicine center. Participants Three scorers participated in this study. Methods Three scorers experienced in using the BESS viewed a videotape depicting 30 consecutive individuals performing the BESS stance positions. The 3 scorers independently scored each of the 30 videotaped individuals using the BESS scoring criteria. A week later, the same 3 subjects viewed and scored the videotape again. Main Outcome Measurements The interrater and intrarater reliability of the BESS was determined using intraclass correlation coefficients (ICC), reported with 95% confidence intervals. The minimum detectible change was also determined. Results The interrater and intrarater reliability ICCs for the total BESS scores were 0.57 and 0.74, respectively. The interrater reliability ICCs for the 6 different stance positions were between 0.44 and 0.83, while the intrarater reliability ICCs were between 0.50 and 0.88. The interrater and intrarater minimum detectible change for the total BESS score were 9.4 and 7.3 points, respectively. Conclusion This study suggests that certain subcategories of the BESS have sufficient reliability to be used in the evaluation of postural stability but that the total BESS score is not reliable. In addition, a change in score of greater than 9.4 (interrater) or 7.3 (intrarater) points is required before the change in postural stability can be attributed to the balancer rather than to the scorer.

Reliability and Validity of Observational Risk Screening in Evaluating Dynamic Knee Valgus

Abstract: Study Design Nonexperimental methodological study. Objectives To determine the interrater and intrarater reliability and validity of using observational risk-screening guidelines to evaluate dynamic knee valgus. Background A deficiency in the neuromuscular control of the hip has been identified as a key risk factor for noncontact anterior cruciate ligament (ACL) injury in postpubescent females. This deficiency can manifest itself as a valgus knee alignment during tasks involving hip and knee flexion. There are currently no scientifically tested methods to screen for dynamic knee valgus in the clinic or on the field. Methods Three physiotherapists used observational risk-screening guidelines to rate 40 adolescent female soccer players according to their risk of ACL injury. The rating was based on the amount of dynamic knee valgus observed on a drop-jump landing. Ratings were evaluated for intrarater and interrater agreement using kappa coefficients. Sensitivity and specificity of ratings were evaluated by ...

The Localized Scleroderma Skin Severity Index and Physician Global Assessment of Disease Activity: A Work in Progress Toward Development of Localized Scleroderma Outcome Measures

Abstract: Objective. To develop and evaluate a Localized Scleroderma (LS) Skin Severity Index (LoSSI) and global assessments’ clinimetric property and effect on quality of life (QOL). Methods. A 3-phase study was conducted. The first phase involved 15 patients with LS and 14 examiners who assessed LoSSI [surface area (SA), erythema (ER), skin thickness (ST), and new lesion/extension (N/E)] twice for inter/intrarater reliability. Patient global assessment of disease severity (PtGA-S) and Children’s Dermatology Life Quality Index (CDLQI) were collected for intrarater reliability evaluation. The second phase was aimed to develop clinical determinants for physician global assessment of disease activity (PhysGA-A) and to assess its content validity. The third phase involved 2 examiners assessing LoSSI and PhysGA-A on 27 patients. Effect of training on improving reliability/validity and sensitivity to change of the LoSSI and PhysGA-A was determined. Results. Interrater reliability was excellent for ER [intraclass correlation coefficient (ICC) 0.71], ST (ICC 0.70), LoSSI (ICC 0.80), and PhysGA-A (ICC 0.90) but poor for SA (ICC 0.35); thus, LoSSI was modified to mLoSSI. Examiners’ experience did not affect the scores, but training/practice improved reliability. Intrarater reliability was excellent for ER, ST, and LoSSI (Spearman’s rho = 0.71–0.89) and moderate for SA. PtGA-S and CDLQI showed good intrarater agreement (ICC 0.63 and 0.80). mLoSSI correlated moderately with PhysGA-A and PtGA-S. Both mLoSSI and PhysGA-A were sensitive to change following therapy. Conclusion. mLoSSI and PhysGA-A are reliable and valid tools for assessing LS disease severity and show high sensitivity to detect change over time. These tools are feasible for use in routine clinical practice. They should be considered for inclusion in a core set of LS outcome measures for clinical trials.

Inter- and intra-rater reliability of neck circumference measurements in children.

Abstract: Summary. Rationale: Increased neck circumference is a risk factor for obstructive sleep apnea in adults. With rising obesity prevalence in children, it may be an important identifier of obstructive sleep apnea in children. The reliability of measuring neck circumference in children has not been systematically evaluated. Objective: To determine the inter- and intra-rater reliability of neck circumference measurements in children aged 2‐16 years. Methods: Children aged 2‐16 years with limb fractures were recruited. Neck circumference was measured by three investigators each using two separate unmarked paper tapes in the 2‐5 year age group (N ¼ 43), and three separate tapes in the 6‐10 and 11‐16 year age groups (N ¼ 18 and 40). Results: Neck circumference measurements showed excellent inter-rater reliability for children 6‐10 and 11‐16 years (ICC ¼ 0.952 and 0.989). Substantial variation was observed for the 2‐5 year age group (ICC ¼ 0.701). Good intra-rater reliability was demonstrated for the three groups (ICC range: 0.776, 0.950). Repeatability coefficients were 2.5‐3.4 cm in the youngest age group and were 1.2‐1.4 cm in the 6‐16 year age group. Conclusion: In children 6‐16 years old, neck circumference shows very good inter and good intra-rater reliability. Multiple measurements are not required for precision and reliability. Pediatr Pulmonol. 2009; 44:64–69. 2008 Wiley-Liss, Inc.

The knee pain map: reliability of a method to identify knee pain location and pattern.

Abstract: Objective To describe the location and pattern of knee pain in patients with chronic, frequent knee pain using the Knee Pain Map, and to evaluate the inter- and intrarater reliability of the map. Methods A cohort of 799 participants from the University of Pittsburgh Osteoarthritis Initiative Clinical Center who had knee pain in the last 12 months were studied. Trained interviewers assessed and recorded participant-reported knee pain patterns into 8 local areas, 4 regional areas, or as diffuse. Inter- and intrarater reliability were assessed using Fleiss' kappa. Results Participants most often reported localized (69%) followed by regional (14%) or diffuse (10%) knee pain. In those with localized pain, the most commonly reported locations were the medial (56%) and lateral (43%) joint lines. In those with regional pain, the most commonly reported regions were the patella (44%) and medial region (38%). There was excellent interrater reliability for the identification of localized and regional pain patterns (κ = 0.7–0.9 and 0.7–0.8, respectively). The interrater reliability for specific locations was also excellent (κ = 0.7–1.0) when the number of participants with pain in a location was >4. For regional pain, the kappa for specific regions varied from 0.7–1.0. Conclusion The majority of participants could identify the location of their knee pain, and trained interviewers could reliably record those locations. The variation in locations suggests that there are multiple sources of pain in knee OA. Additional studies are needed to determine whether specific knee pain patterns correlate with discrete pathologic findings on radiographs or magnetic resonance images.

A Modified Reliability Index Approach for Reliability-Based Design Optimization

Abstract: RBDO problems have been intensively studied for many decades. Since Hasofer and Lind defined a measure of the second-moment reliability index, many RBDO methods utilizing the concept of reliability index have been introduced as the Reliability Index Approach (RIA). In the RIA, a reliability analysis problem is formulated to find the reliability index for each performance constraint and the solutions are used to evaluate the failure probability. However, the traditional RIA suffers from inefficiency and convergence problems. In this paper, we revisited the definition of the reliability index and revealed the convergence problem in the traditional RIA. Furthermore, a new definition of the reliability index is proposed to correct this problem and a modified Reliability Index Approach based on this definition is developed. Numerical examples using both the traditional RIA and the modified RIA are compared and discussed.Copyright © 2009 by ASME

Reliability of the Perceive, Recall, Plan and Perform System of Task Analysis: A criterion-referenced assessment

Abstract: Objective: To conduct preliminary examination of the rater and test-reliability of the Perceive, Recall, Plan and Perform (PRPP) System of Task Analysis, an ecological measure designed to assess task-embedded information processing capacity during occupational therapy assessment of confused and agitated adults following traumatic brain injury. Methods: Occupational therapists observed and scored client performance using the PRPP System of Task Analysis. Correlational analysis and measures of agreement were performed to determine interrater and intrarater reliability. Test procedures were examined for reliability and internal consistency. Results: Interrater and test reliability considered three factors: therapists, clients and tasks. A moderate level of interrater reliability was achieved between trained therapists (intraclass correlation coefficient (ICC) = 0.60). Test procedures were highly reliable (ICC = 0.88). Across two measurement occasions, therapists showed a tendency towards harder rating on the second test occasion ( -4.5%; 95% confidence interval for : -10.67%→ 3.17%). Conclusion: The findings of this study support the use of criterion-referenced tests in the area of occupational performance measurement. Occupational therapists achieved moderate interrater reliability when measuring the performance of adults with brain injury on various activities of daily living. Test procedures were found to be highly reliable in measuring the occupational performance of adults demonstrating confusion and agitation typical to the stage of post-traumatic amnesia following head injury. Language: en

Lack of agreement between rheumatologists in defining digital ulceration in systemic sclerosis

Abstract: OBJECTIVE: To test the intra- and interobserver variability, among clinicians with an interest in systemic sclerosis (SSc), in defining digital ulcers. METHODS: Thirty-five images of finger lesions, incorporating a wide range of abnormalities at different sites, were duplicated, yielding a data set of 70 images. Physicians with an interest in SSc were invited to take part in the Web-based study, which involved looking through the images in a random sequence. The sequence differed for individual participants and prevented cross-checking with previous images. Participants were asked to grade each image as depicting "ulcer" or "no ulcer," and if "ulcer," then either "inactive" or "active." Images of a range of exemplar lesions were available for reference purposes while participants viewed the test images. Intrarater reliability was assessed using a weighted kappa coefficient with quadratic weights. Interrater reliability was estimated using a multirater weighted kappa coefficient. RESULTS: Fifty individuals (most of them rheumatologists) from 15 countries participated in the study. There was a high level of intrarater reliability, with a mean weighted kappa value of 0.81 (95% confidence interval [95% CI] 0.77, 0.84). Interrater reliability was poorer (weighted kappa = 0.46 [95% CI 0.35, 0.57]). CONCLUSION: The poor interrater reliability suggests that if digital ulceration is to be used as an end point in multicenter clinical trials of SSc, then strict definitions must be developed. The present investigation also demonstrates the feasibility of Web-based studies, for which large numbers of participants can be recruited over a short time frame.

Reliability of the anterior drawer and talar tilt tests using the LigMaster joint arthrometer.

Abstract: Context: Joint arthrometers have been developed to help determine the severity of ligament sprains. Objective: To establish intratester and intertester reliability of the talar inversion and anterior drawer tests using the LigMaster. Design: Intratester reliability was investigated using a repeated-measures design. Intertester reliability was investigated using 2 different clinicians testing subjects on the same day. Setting: Athletic training research laboratory. Participants: Thirty participants volunteered for this study. Main Outcome Measurements: Anterior displacement and talar inversion were measured using the LigMaster. Results: Intrarater reliability was .74 for the talar inversion test and .65 for the anterior drawer test. Interrater reliabilities for the talar inversion and anterior drawer tests were .76 and .81, respectively. Conclusions: The LigMaster joint arthrometer is a reliable tool for measuring talar inversion and anterior displacement at the ankle.

Dental Hygiene Faculty Calibration in the Evaluation of Calculus Detection

Abstract: The purpose of this pilot study was to explore the impact of faculty calibration training on intra- and interrater reliability regarding calculus detection. After IRB approval, twelve dental hygiene faculty members were recruited from a pool of twenty-two for voluntary participation and randomized into two groups. All subjects provided two pre- and two posttest scorings of calculus deposits on each of three typodonts by recording yes or no indicating if they detected calculus. Accuracy and consistency of calculus detection were evaluated using an answer key. The experimental group received three two-hour training sessions to practice a prescribed exploring sequence and technique for calculus detection. Participants immediately corrected their answers, received feedback from the trainer, and reconciled missed areas. Intra- and interrater reliability (pre- and posttest) was determined using Cohen's Kappa and compared between groups using repeated measures (split-plot) ANOVA. The groups did not differ from pre- to posttraining (intrarater reliability p=0.64; interrater reliability p=0.20). Training had no effect on reliability levels for simulated calculus detection in this study. Recommendations for future studies of faculty calibration when evaluating students include using patients for assessing rater reliability, employing larger samples at multiple sites, and assessing the impact on students' attitudes and learning outcomes.

The Reliability of Computed Tomography and Magnetic Resonance Imaging Grading of Lumbar Facet Arthropathy in Total Disc Replacement Patients

Abstract: Study Design. Prospective inter-rater and intrarater reliability analysis. Objective. To compare the inter-rater and intrarater reliability of magnetic resonance imaging (MRI) and computed tomography (CT) for grading of facet arthropathy as well as determining whether there is a contraindication to total disc replacement (TDR). Summary of Background Data. Several classifications attempted to evaluate lumbar facet joints and their degree of arthropathy. The Fujiwara-MRI and Pathrea–CT classifications remain the most commonly used. Methods. A total of 10 fellowship-trained orthopedic spine surgeons and 3 orthopedic spine fellows evaluated 50 levels from L3–L4 through L5–S1 on parallel axial MRI (T1 and T2) and CT images. The degree of osteoarthritis was graded on a 4-point scale (Fujiwara-MRI and Pathrea-CT). Surgeons evaluated whether the degree of facet disease represented a contraindication to TDR. Grading was performed during 2 sessions. Weighted kappa statistics were used to describe inter- and intraobserver agreement. Results. The inter-rater reliability for MRI was 0.21 and 0.07 (fair to slight) among attendings and fellows, respectively. inter-rater reliability for CT was 0.33 and 0.27 (fair), respectively. The mean intrarater reliability for MRI was fair, 0.36 (attendings) and 0.26 (fellows). The mean intrarater reliability for CT was moderate, 0.52 (attendings) and 0.51 (fellows). For possible TDR contraindication, the inter-rater reliability for MRI was 0.22 and 0.01 (fair to slight) among attendings and fellows, respectively. Inter-rater reliability for CT was 0.33 and 0.45 (fair), respectively. The mean intrarater reliability for MRI was fair, 0.36 (attendings) and 0.26 (fellows). The mean intrarater reliability for CT was moderate, 0.52 (attendings) and 0.51 (fellows). Conclusion. The current grading system for facet arthropathy has only fair agreement. CT is slightly more reliable for grading. Intrarater reliability was only fair for MRI and moderate for CT. Only limited agreement existed between surgeons as to the extent of facet disease that would pose as a contraindication for TDR.

Palmar abduction: reliability of 6 measurement methods in healthy adults.

Abstract: Purpose The aim of the current study was to assess reliability of 6 palmar thumb abduction measurement methods: conventional goniometry, the Inter Metacarpal Distance, the method described by the American Medical Association, the method described by the American Society of Hand Therapists, and 2 new methods: the Pollexograph-thumb and the Pollexograph-metacarpal. Methods An experienced hand therapist and a less-experienced examiner (trainee in plastic surgery) measured the right hands of 25 healthy subjects. Palmar abduction was measured both passively and actively. Means and ranges for palmar abduction were calculated, and intrarater and interrater reliability was expressed in intraclass correlation coefficients, standard errors of measurement, and smallest detectable differences. Results Mean active and passive angles measured with goniometry resembled values measured with the Pollexograph-thumb method (approximately 60°). Mean angles found with the Pollexograph-metacarpal method were approximately 48°. Mean active and passive distances for the Inter Metacarpal Distance were 64 mm. Mean active and passive distances found with the American Society of Hand Therapists method were 97 to 101 mm, and mean distances found with the American Medical Association method were 67 to 70 mm for active and passive measurements. Intraclass correlation coefficients for the Pollexograph-thumb, Pollexograph-metacarpal, and the Inter Metacarpal Distance indicated good and significantly higher intrarater agreement for active and passive measurements than intraclass correlation coefficients of conventional goniometry, the American Society of Hand Therapists method, and the American Medical Association method, which showed only moderate agreement. For interrater reliability, the same measurement methods were found to be most reliable: the Pollexograph-thumb, Pollexograph-metacarpal, and the Inter Metacarpal Distance. Conclusions We found that the Pollexograph-thumb, Pollexograph-metacarpal, and the Inter Metacarpal Distance are the most reliable measurement methods for palmar abduction.

Reproducibility and reference values of inclinometric balance and isometric trunk muscle strength measurements in Finnish young adults.

Abstract: The objectives of the study were to evaluate the reproducibility of the maximal isometric trunk muscle strength testing and inclinometric method to measure body sway and to establish reference values for these measurements in young Finnish adults. Reproducibility was assessed with 2 repeated measurements. Reference values are based on the cross-sectional cohort data. Maximal isometric trunk extension, flexion, and rotations were measured in standing posture. Body sway was measured with inclinometric method. The subjects (N = 874) belonged to a subcohort of the Northern Finland Birth Cohort 1986. Intrarater reliability was assessed with 19 volunteers from the birth cohort, and interrater reliability was assessed with 15 young healthy adults. Intrarater reliability intraclass correlation coefficients (ICCs) for body sway measurements ranged from 0.39 to 0.74 and for trunk muscle strength measurements from 0.84 to 0.94. Interrater reliability ICC for body sway ranged from 0.61 to 0.85 and for trunk muscle strength from 0.84 to 0.88. Maximal muscle strength was better in men than in women, but extensor to flexor ratio was higher in women. Women had significantly better values in all body sway measurements than men. Our study showed a remarkable biological variation in isometric trunk muscle strength and inclinometric body sway. Although ICC analysis indicated good reliability, the Bland and Altman analysis revealed quite wide range of measurement error. However, reproducibility of isometric trunk muscle testing is comparable to other measurements of trunk muscle function in use. Reference values collected in our study offer valuable normative data for future studies and can be used as a tool when assessing physical capacity of healthy subjects or patients with different pathological conditions.

Development and reliability of an observational gait analysis tool for children with Down syndrome.

Abstract: Purpose To develop an observational gait analysis tool for children with Down syndrome and to examine its reliability. Methods Two physical therapists and 5 physical therapy students participated in this study. Sixty videos were examined to determine interrater reliability. Four months later, 15 of those 60 videos were reexamined twice (2 weeks apart) by the students for intrarater reliability. Intraclass correlation coefficients for interrater and intrarater reliability, percentage of agreement, and standard error of measurement were calculated. Results Interrater reliability was 0.663, intrarater reliability was 0.616-0.877, standard error of measurement was 1.89, and absolute agreement was found for 35.56% of the trials. Conclusion Good reliability was found, but further study is needed to determine validity and clinical relevance.

Test-retest intra-rater reliability of vaginal measurement of pelvic floor muscle strength using Myomed 932.

Abstract: Objectives. To investigate test–retest intra-rater reliability of measurements of pelvic floor muscle (PFM) strength using Myomed 932® (Enraf-Nonius, Delft, the Netherlands). Design. Test–retest intra-rater reliability test with a 2–7 day interval. Setting. Physiotherapy outpatient clinic in association with the Department of Physiotherapy, University of Iceland. Population. Twenty healthy female volunteers in Reykjavik metropolitan area, Iceland, with a mean age of 43.8 (SD 9.8) years, mean body mass index (BMI) 26.2 (SD 5.3), and mean parity 2.1 children (SD 1.0). Main outcome measures. PFM strength during two attempts of maximal voluntary contraction (MVC) measured as vaginal squeeze pressure in hectoPascals (hPa). Results. Mean MVC was 34.45 hPa (95% CI: 26.07–42.83) on Day 1 and 34.05 hPa (95% CI: 26.66–41.44) on Day 2. The measurements ranged from 5–74 hPa interindividually on Day 1 and 5–62 hPa on Day 2. Intraclass Correlation Coefficient (ICC) was 0.97; p<0.001 and coefficient of variation (CV%) w...

Intrarater and interrater reliability of Myotonometer measurements of muscle tone in children.

Abstract: The aim of this study was to investigate reliability of Myotonometer measurements, by assessing muscle tone in muscle rectus femoris in children with cerebral palsy and children without disabilities. Differences in muscle tone between groups and conditions, contracted or relaxed, were also investigated. A total of 15 children with cerebral palsy and 15 children without disabilities participated. Intrarater reliability of Myotonometer measurements showed high to very high intraclass correlation, under both conditions and in both groups. Interrater reliability showed high to very high intraclass correlation under both conditions in the control group and in the cerebral palsy group under contracted condition and moderate under relaxed condition. There were slight differences in tissue displacement between groups, although not statistically significant. Myotonometer measurements are reliable when assessing muscle tone in children. However, there is still a need to investigate this method for objectively quantifying increased muscle tone in children with cerebral palsy.