Publisher Summary This chapter discusses reliability coefficients as developed in the framework of the classical test theory (CTT) and describes the way the conception and estimation of reliability has been broadened in generalizability theory (G). It also discusses the foundations of CTT and focuses on the traditional methods of estimating reliability. It reviews generalizability theory, its applications, and recent theoretical contributions. In addition to computational procedures, G theory provides a conceptual framework and notational system for characterizing alternative intended universes of generalization, distinguishing fixed versus random facets, and designing efficient measurement procedures. G theory has also clarified the use and interpretation of coefficients and estimated standard errors from CTT. This is of value because, despite the greater sophistication of G theory and the important applications, CTT remains the dominant tool for estimating reliability. However, reliability is just one consideration in the responsible use and interpretation of tests, and not the most important consideration.

4 Reliability Coefficients and Generalizability Theory

Part 1: The State of the Art of Educational Effectiveness: Challenges For Research Methodology 1: Background of Eer 2: Methodological Issues in Educational Effectiveness Research 3: Theory And Research: The Problem of Causality 4: Theory-Driven Evaluation Studies: Establishing Links Between Research, Policy and Practice Part 2: The Contribution of Different Methodological Orientations to the Development of Research and Evaluation on Educational Effectiveness 5: Longitudinal Designs 6: Experimental Studies in Education 7: Applications of Mixed Methods to the Field of Educational Effectiveness Research 8: Using Irt to Measure Outcomes and Factors: An Overview of Irt Models 9: Irt Models: Parameter Estimation, Statistical Testing and Application in Eer 10: Using Generalizability Theory 11: Multilevel Modelling 12: Structural Equation Modelling Techniques 13: Meta-Analyses of Effectiveness Studies Part 3: Methodological Advances And Eer: Retrospect & Prospect 14: Conclusions for the Development of Educational Effectiveness Research

/pdf/methodological-advances-in-educational-effectiveness-5886p2sgkx.pdf

Methodological Advances in Educational Effectiveness Research

Methodological advances in educational effectiveness research, by Bert P.M. Creemers, Leonidas Kyriakides, and Pam Sammons, Abingdon, UK, Routledge, 2010, 362 pp., £30.99 (paperback), ISBN 978-0-41...

Methodological advances in educational effectiveness research

The paper describes the small sample stability of least square estimates of variance components within the context of generalizability theory. Monte Carlo methods are used to generate data conforming to some selected multifacet generalizability designs to illustrate the sampling behavior of variance component estimates. Based on the findings, recommendations are made concerning the design of efficient small sample generalizability studies.

Sampling Errors of Variance Components in Small Sample Multifacet Generalizability Studies

Generalizability theory provides a framework for examining the dependability of behavioral measurements. When designing generalizability studies, two important statistical issues are generally considered: power and measurement error. Control over power and error of measurement can be obtained by manipulation of sample size and/or test reliability. In generalizability theory, the mean error variance is an estimate that takes into account both these statistical issues. When limited resources are available, determining an optimal measurement design is not a simple task. This article presents a methodology for minimizing mean error variance in generalizability studies when resource constraints are imposed.

Maximizing Power in Generalizability Studies under Budget Constraints.

For random-model, fully-crossed, two- and three-facet experimental designs the following two problems were considered. First, equations were developed for determining the optimal number of conditions of a facet for maximizing the coefficient of generalizability under the constraint that the total number of observations per subject is constant. Second, the problem of determining the minimum number of observations per subject for a specified generalizability coefficient is solved for the two-facet crossed design.

Maximizing The Coefficient Of Generalizability In Multi-Facet Decision Studies.

sor Henry F. Kaiser, provides means, standard deviations, correlation matrix, root-mean-square correlation of each variable with the remaining variables, overall root-mean-square correlation, squared multiple correlation of each variable with the remaining variables, a measure of sampling adequacy of each variable, overall measure of sampling adequacy, number of factors, a model-free factor pattern matrix, index of factorial simplicity, relative contributions of the factors, factor intercorrelation matrix, factor structure matrix, and factor scores.

Kaiser's Little Jiffy, Mark IV for a Small Computer Using Basic Fortran IV.

An approximate confidence interval is developed for the maximum coefficient alpha reliability of a scale where the scale is considered fixed an persons are sampled.

J. Arthur Woodward

Papers

Maximizing The Coefficient Of Generalizability In Multi-Facet Decision Studies.

Kaiser's Little Jiffy, Mark IV for a Small Computer Using Basic Fortran IV.

An Approximate Confidence Interval for Maximum Coefficient Alpha.

Cronb: a Fortran Iv Program to Compute Variance Components for Various Experimental Designs