One of the principal characteristics of an outcome measure in a clinical trial, and any measurement in general, is its reliability. Reliability refers to the reproducibility of the measurement when repeated at random in the same subject or specimen. Reliability is often confused with validity, which refers to the extent to which the variable properly measures the underlying trait of interest. The coefficient of reliability is an estimate of the proportion of all variation that is not due to measurement error and is readily estimated from replicate measurements. The reliability of a measurement determines its maximal correlation or R2 and slope (or effect size) in regression models, its sensitivity and specificity when used for classifications or predictions, and the power of a statistical test employing the measurement. All decline as the reliability of the measure declines. The reliability of a measurement is an important consideration in the choice of the primary outcome measure for a clinical trial and in the choice of measures used for assessment of eligibility and exclusion. Reliability of measures should be assessed and assured by a quality control program based on randomly selected duplicate assessments. Just as the power of a study is reported in a final publication, so also should the reliability of the outcome and eligibility measurements so as to allow the authors to better describe, and readers to better understand, the sources of imprecision in study results, and those who follow to improve the design of future trials.

The role of measurement reliability in clinical trials.

Reliability studies are widely used to assess the measurement reproducibility of human observers, laboratory assays or diagnostic tests. For quantitative measures, intra-class correlation coefficient (ICC) is the principal measurement of reliability. In this paper, a SAS macro is provided to calculate the ICC and its confidence limits. The application of the macro is demonstrated in a clinical reliability case study, the comparison of two ICCs is discussed.

/pdf/reliability-analysis-calculate-and-compare-intra-class-50ru8bj0ex.pdf

Reliability analysis: Calculate and Compare Intra-class Correlation Coefficients (ICC) in SAS

The variability among data may have quite complex structure and the splitting of variation into its individual components is important in many areas of applications. In particular, the study of variation through a class of linear models known as random and mixed models is a central topic in statistics with wide ramifications in both theory and applications. Sometimes these models are called components of variance models since the interest in the model lies in investigating the variances of the effects of factor levels which are assumed to be randomly selected from a large population of levels. These variances are also known as variance components since the total variance can be expressed as the sum of these variances and the error variance. Knowledge of the variance components, i.e., the information on the relative importance and magnitude of the sources of variability in the data is of critical importance in many problems since the isolation of the total variance into the individual components of variance would result in the physical elimination of the sources of variability in question.

/pdf/a-bibliography-on-variance-components-an-introduction-and-an-26dsdcssx9.pdf

A bibliography on variance components an introduction and an update: 1984-2002

Statistical methods and scientific inference (3rd edition), by R. A. Fisher. Pp viii, 182. £4·95. 1973. SBN 0 02 844740 9 (Collier-Macmillan)

Linear discrimination with equicorrelated training vectors

The problem of estimating the intraclass correlation when the sampling design is unbalanced is discussed. The method of moments is used to derive an approximation to the distribution of the estimate of the intraclass correlation obtained by the variance components approach. The maximum likelihood estimator is also presented, along with a simple procedure due to Richard (1961), for numerically maximizing a likelihood function of several parameters. Finally, the issue of optimal study design is considered for both balanced and unbalanced situations. For given power we determine the number of sets required to detect different values of the intraclass correlation.

M.M. Shoukri

Papers

On the estimation of the intraclass correlation