Reliability, the consistency of a test or measurement, is frequently quantified in the movement sciences literature. A common metric is the intraclass correlation coefficient (ICC). In addition, the SEM, which can be calculated from the ICC, is also frequently reported in reliability studies. However, there are several versions of the ICC, and confusion exists in the movement sciences regarding which ICC to use. Further, the utility of the SEM is not fully appreciated. In this review, the basics of classic reliability theory are addressed in the context of choosing and interpreting an ICC. The primary distinction between ICC equations is argued to be one concerning the inclusion (equations 2,1 and 2,k) or exclusion (equations 3,1 and 3,k) of systematic error in the denominator of the ICC equation. Inferential tests of mean differences, which are performed in the process of deriving the necessary variance components for the calculation of ICC values, are useful to determine if systematic error is present. If so, the measurement schedule should be modified (removing trials where learning and/or fatigue effects are present) to remove systematic error, and ICC equations that only consider random error may be safely used. The use of ICC values is discussed in the context of estimating the effects of measurement error on sample size, statistical power, and correlation attenuation. Finally, calculation and application of the SEM are discussed. It is shown how the SEM and its variants can be used to construct confidence intervals for individual scores and to determine the minimal difference needed to be exhibited for one to be confident that a true change in performance of an individual has occurred.

Quantifying test-retest reliability using the intraclass correlation coefficient and the SEM.

The authors conducted a meta-analysis of 150 studies in which the risk-taking tendencies of male and female participants were compared. Studies were coded with respect to type of task (e.g., self-reported behaviors vs. observed behaviors), task content (e.g., smoking vs. sex), and 5 age levels. Results showed that the average effects for 14 out of 16 types of risk taking were significantly larger than 0 (indicating greater risk taking in male participants) and that nearly half of the effects were greater than .20. However, certain topics (e.g., intellectual risk taking and physical skills) produced larger gender differences than others (e.g., smoking). In addition, the authors found that (a) there were significant shifts in the size of the gender gap between successive age levels, and (b) the gender gap seems to be growing smaller over time. The discussion focuses on the meaning of the results for theories of risk taking and the need for additional studies to clarify age trends.

Gender differences in risk taking: A meta-analysis.

Self-report measures of adult attachment are typically scored in ways (e.g., averaging or summing items) that can lead to erroneous inferences about important theoretical issues, such as the degree of continuity in attachment security and the differential stability of insecure attachment patterns. To determine whether existing attachment scales suffer from scaling problems, the authors conducted an item response theory (IRT) analysis of 4 commonly used self-report inventories: Experiences in Close Relationships scales (K. A. Brennan, C. L. Clark, & P. R. Shaver, 1998), Adult Attachment Scales (N. L. Collins & S. J. Read, 1990), Relationship Styles Questionnaire (D. W. Griffin & K. Bartholomew, 1994) and J. Simpson's (1990) attachment scales. Data from 1,085 individuals were analyzed using F. Samejima's (1969) graded response model. The authors' findings indicate that commonly used attachment scales can be improved in a number of important ways. Accordingly, the authors show how IRT techniques can be used to develop new attachment scales with desirable psychometric properties.

An item response theory analysis of self-report measures of adult attachment.

Procedures previously recommended by various authors for the estimation of "change" scores, "residual" or "basefree" measures of change, and other kinds of difference scores are examined. A procedure proposed by Lord is extended to obtain more precise estimates, and an alternative to the Tucker-Damarin-Messick procedure is offered. A consideration of the purposes for which change measures have been sought in the past leads to a series of recommended procedures which solve research and personnel-decision problems without estimation of change scores for individuals.

How we should measure "change": Or should we?

Statistical power analysis for the behavioral sciences: (revised edition), by Jacob Cohen. New York: Academic Press, 1977, xv + 474 pp., $24.50.

What were the historic origins of classical test theory? What have been major milestones in its development?

/pdf/classical-test-theory-in-historical-perspective-471xpr6gpj.pdf

Classical Test Theory in Historical Perspective

Two sets of mathematical reasoning and two sets of verbal comprehension items were cast into each of three formats—constructed response, standard multiple-choice, and Coombs multiple- choice—in order to assess whether tests with iden tical content but different formats measure the same attribute, except for possible differences in error variance and scaling factors. The resulting 12 tests were administered to 199 eighth-grade stu dents. The hypothesis of equivalent measures was rejected for only two comparisons: the con structed-response measure of verbal comprehen sion was different from both the standard and the Coombs multiple-choice measures of this ability. Maximum likelihood factor analysis confirmed the hypothesis that a five-factor structure will give a satisfactory account of the common variance among the 12 tests. As expected, the two major factors were mathematical reasoning and verbal comprehension. Contrary to expectation, only one of the other three factors bore a (weak) resem blance to a fo...

/pdf/on-the-equivalence-of-constructed-response-and-multiple-4g3l4c75rf.pdf

On the equivalence of constructed-response and multiple-choice tests

NCME Instructional Module: Understanding Reliability.

Formula Scoring, Number-Right Scoring, and Test Taking Strategy.

Although Birnbaum's logistic models have been known since 1957, there have been few applications to empirical data reported in the literature. In this study, the one- and two-parameter logistic models were compared with respect to their capacity to predict the distribution of statistics for estimating ability. The results of this study suggest that the two-parameter model will provide greatest improvement over the one-parameter model when applied to data from short tests where the variability of the discrimination parameters is substantial. Whether the gains are worth the increased cost of solving for the parameters of the more complex model is a question which requires investigation.

https://files.eric.ed.gov/fulltext/ED042816.pdf

Ross E. Traub

Papers

Classical Test Theory in Historical Perspective

On the equivalence of constructed-response and multiple-choice tests

NCME Instructional Module: Understanding Reliability.

Formula Scoring, Number-Right Scoring, and Test Taking Strategy.

Analysis of empirical data using two logistic latent trait models