Primary, Secondary, and Meta-Analysis of Research

TLDR: The meta-analysis of research as discussed by the authors is an important feature of the research and evaluation enterprise, and it has been widely used in the field of computer science and computer engineering, especially in the context of education.

Abstract: M y subject is data analysis at three levels. Primary analysis is the original analysis of data in a research study. It is what one typically imagines as the application of statistical methods. Secondary analysis is the re-analysis of data for the purpose of answering the original research question with better statistical techniques, or answering new questions with old data. Secondary analysis is an important feature of the research and evaluation enterprise. Tom Cook (1974) at Northwestern University has written about its purposes and methods. Some of our best methodologists have pursued secondary analyses in such grand style that its importance has eclipsed that of the primary analysis. We can cite with pride some state of the art documents: the MostellerMoynihan secondary analysis of the Coleman study; the Campbell-Erlebacher analysis of the Ohio-Westinghouse Headstart evaluation; and the Elashoff-Snow secondary analysis of Pygmalion in the Classroom, to name three. About all that can effectively be done to insure that secondary analyses of important studies are carried out is to see that the data from the original studies are preserved and that secondary analyses are funded. The preservation of original data could improve. Last month, one of our graduate students, Karl White, spent 15 hours and made 30 phone calls attempting to obtain from the government a copy of the data tapes for the Coleman study only to learn in the end that they had been irretrievably filed in unmarked tape cannisters with some 2,000 other unmarked data tapes. Tom Cook remarked in an Annual Meeting symposium on secondary analysis that you can get the data if you have chutzpah or if you're socio metrically well-connected. The whole business is too important to be treated so casually. On the other extreme, one can point with satisfaction to the ready availability to any researcher of the data tapes from Project TALENT or the National Assessment of Educational Progress. Others are advancing the practice of secondary analysis. My major interest currently is in what we have come to call—not for want of a less pretentious name—the meta-analysis of research. The term is a bit grand, but it is precise, and apt, and in the spirit of "metamathematics," "meta-psychology," and "meta-evaluation." Meta-analysis refers to the analysis of analyses. I use it to refer to the statistical analysis of a large collection of analysis results from individual studies for the purpose of integrating the findings. It connotes a rigorous alternative to the casual, narrative discussions of research studies which typify our attempts to make sense of the rapidly expanding research literature. The need for the meta-analysis of research is clear. The literature on dozens of topics in education is growing at an astounding rate. In five years time, researchers can produce literally hundreds of studies on IQ and creativity, or impulsive vs. reflective cognitive styles, or any other topic.