In the light of continuing debate over the applications of significance testing in psychology journals and following the publication of Cohen's (1994) article, the Board of Scientific Affairs (BSA) of the American Psychological Association (APA) convened a committee called the Task Force on Statistical Inference (TFSI) whose charge was "to elucidate some of the controversial issues surrounding applications of statistics including significance testing and its alternatives; alternative underlying models and data transformation; and newer methods made possible by powerful computers" (BSA, personal communication, February 28, 1996). Robert Rosenthal, Robert Abelson, and Jacob Cohen (cochairs) met initially and agreed on the desirability of having several types of specialists on the task force: statisticians, teachers of statistics, journal editors, authors of statistics books, computer experts, and wise elders. Nineindividuals were subsequently invited to join and all agreed. These were Leona Aiken, Mark Appelbaum, Gwyneth Boodoo, David A. Kenny, Helena Kraemer, Donald Rubin, Bruce Thompson, Howard Wainer, and Leland Wilkinson. In addition, Lee Cronbach, Paul Meehl, Frederick Mosteller and John Tukey served as Senior Advisors to the Task Force and commented on written materials.

/pdf/statistical-methods-in-psychology-journals-guidelines-and-2dxv2y8x6m.pdf

Statistical Methods in Psychology Journals: Guidelines and Explanations

Quasi-experimentation - design and analysis issues for field settings - cook,td, campbell,dt

Combining nonoverlap and trend for single-case research: Tau-U.

Missing data analyses have received considerable recent attention in the methodological literature, and two “modern” methods, multiple imputation and maximum likelihood estimation, are recommended. The goals of this article are to (a) provide an overview of missing-data theory, maximum likelihood estimation, and multiple imputation; (b) conduct a methodological review of missing-data reporting practices in 23 applied research journals; and (c) provide a demonstration of multiple imputation and maximum likelihood estimation using the Longitudinal Study of American Youth data. The results indicated that explicit discussions of missing data increased substantially between 1999 and 2003, but the use of maximum likelihood estimation or multiple imputation was rare; the studies relied almost exclusively on listwise and pairwise deletion.

Missing Data in Educational Research: A Review of Reporting Practices and Suggestions for Improvement

Practices of data analysis in psychology and related disciplines are changing. This is evident in the longstanding controversy about statistical tests in the behavioral sciences and the increasing number of journals requiring effect size information. Beyond Significance Testing offers integrative and clear presentations about the limitations of statistical tests and reviews alternative methods of data analysis, such as effect size estimation (at both the group and case levels) and interval estimation (i.e., confidence intervals). Written in a clear and accessible style, the book is intended for applied researchers and students who may not have strong quantitative backgrounds. Readers will learn how to measure effect size on continuous or dichotomous outcomes in comparative studies with independent or dependent samples. They will also learn how to calculate and correctly interpret confidence intervals for effect sizes. Numerous research examples from a wide range of areas illustrate the application of these principles and how to estimate substantive significance instead of just statistical significance.

Beyond significance testing : reforming data analysis methods in behavioral research

Articles published in several prominent educational journals were examined to investigate the use of data analysis tools by researchers in four research paradigms: between-subjects univariate designs, between-subjects multivariate designs, repeated measures designs, and covariance designs. In addition to examining specific details pertaining to the research design (e.g., sample size, group size equality/inequality) and methods employed for data analysis, the authors also catalogued whether (a) validity assumptions were examined, (b) effect size indices were reported, (c) sample sizes were selected on the basis of power considerations, and (d) appropriate textbooks and/or articles were cited to communicate the nature of the analyses that were performed. The present analyses imply that researchers rarely verify that validity assumptions are satisfied and that, accordingly, they typically use analyses that are nonrobust to assumption violations. In addition, researchers rarely report effect size statistics, ...

/pdf/statistical-practices-of-educational-researchers-an-analysis-2k4nst3lzk.pdf

Statistical Practices of Educational Researchers: An Analysis of their ANOVA, MANOVA, and ANCOVA Analyses:

The research content of interest herein is that of comparison of means. It is generally recognized that statistical test p values do not adequately reflect mean comparison assessments. What is desirable is some effect-size assessment. The typical effect-size indexes used in mean comparisons are restricted to the variance homogeneity condition. What is proposed here is the use of the group-overlap concept. Group overlap may be assessed via prediction of group assignment, that is, using predictive discriminant analysis. The effect-size index proposed is that of improvement-over-chance classification (I). The I index may be used in situations that are univariate, multivariate, homogeneous, heterogeneous, or any combination thereof. Some very tentative suggestions for cutoffs of I values to define index magnitude for some data situations are made.

Group Overlap as a Basis for Effect Size

http://link.springer.com/content/pdf/10.1007/978-94-011-3971-7_6.pdf

Discriminant Analysis in Higher Education Research

The two aspects of discriminant analysis are briefly reviewed: predictive discriminant analysis (PDA) and descriptive discriminant analysis (DDA). Use of three popular statistical packages (BMDP, SAS, and SPSS) to obtain computational results for a PDA and for a DDA is generally reviewed. Results yielded by two BMDP procedures (7M and SM) are discussed, as are four SAS procedures (DISCRIM, STEPDISC, CANDISC, and GLM), and two SPSS procedures (DISCRIMINANT and MANOVA). It is pointed out which procedures are used to obtain PDAresults and which are used for PDAresults. Examples of printout information are given. Some of these examples pertain to misleading information-one to incorrect information, and some, of course, to very useful information. The need for nonpackage analyses to generate specific PDA and DDA information is also mentioned.

Laureen L. Lowman

Papers

Statistical Practices of Educational Researchers: An Analysis of their ANOVA, MANOVA, and ANCOVA Analyses:

Group Overlap as a Basis for Effect Size

Discriminant Analysis in Higher Education Research

Discriminant Analysis via Statistical Packages.