Alison Black

Bio: Alison Black is an academic researcher from MDRC. The author has contributed to research in topics: Disability benefits & Academic achievement. The author has an hindex of 8, co-authored 14 publications receiving 1764 citations.

Empirical Benchmarks for Interpreting Effect Sizes in Research

Abstract: There is no universal guideline or rule of thumb for judging the practical importance or substantive significance of a standardized effect size estimate for an intervention. Instead, one must develop empirical benchmarks of comparison that reflect the nature of the intervention being evaluated, its target population, and the outcome measure or measures being used. This approach is applied to the assessment of effect size measures for educational interventions designed to improve student academic achievement. Three types of empirical benchmarks are illustrated: (a) normative expectations for growth over time in student achievement, (b) policy-relevant gaps in student achievement by demographic group or school performance, and (c) effect size results from past research for similar interventions and target populations. The findings can be used to help assess educational interventions, and the process of doing so can provide guidelines for how to develop and use such benchmarks in other fields.

Performance Trajectories and Performance Gaps as Achievement Effect-Size Benchmarks for Educational Interventions

Abstract: Two complementary approaches to developing empirical benchmarks for achievement effect sizes in educational interventions are explored. The first approach characterizes the natural developmental progress in achievement made by students from one year to the next as effect sizes. Data for seven nationally standardized achievement tests show large annual gains in the early elementary grades followed by gradually declining gains in later grades. A given intervention effect will therefore look quite different when compared to the annual progress for different grade levels. The second approach explores achievement gaps for policy-relevant subgroups of students or schools. Data from national- and district-level achievement tests show that, when represented as effect sizes, student gaps are relatively small for gender and much larger for economic disadvantage and race/ethnicity. For schools, the differences between weak schools and average schools are surprisingly modest when expressed as student-level e...

Using Covariates to Improve Precision for Studies that Randomize Schools to Evaluate Educational Interventions.

Abstract: This article examines how controlling statistically for baseline covariates, especially pretests, improves the precision of studies that randomize schools to measure the impacts of educational inte...

The Impact of Enhancing Students’ Social and Emotional Learning: A Meta‐Analysis of School‐Based Universal Interventions

Abstract: This article presents findings from a meta-analysis of 213 school-based, universal social and emotional learning (SEL) programs involving 270,034 kindergarten through high school students. Compared to controls, SEL participants demonstrated significantly improved social and emotional skills, attitudes, behavior, and academic performance that reflected an 11-percentile-point gain in achievement. School teaching staff successfully conducted SEL programs. The use of 4 recommended practices for developing skills and the presence of implementation problems moderated program outcomes. The findings add to the growing empirical evidence regarding the positive impact of SEL programs. Policy makers, educators, and the public can contribute to healthy development of children by supporting the incorporation of evidence-based SEL programming into standard educational practice.

Multiple Inference and Gender Differences in the Effects of Early Intervention: A Reevaluation of the Abecedarian, Perry Preschool, and Early Training Projects

Abstract: The view that the returns to educational investments are highest for early childhood interventions is widely held and stems primarily from several influential randomized trials—Abecedarian, Perry, and the Early Training Project—that point to super-normal returns to early interventions. This article presents a de novo analysis of these experiments, focusing on two core issues that have received limited attention in previous analyses: treatment effect heterogeneity by gender and overrejection of the null hypothesis due to multiple inference. To address the latter issue, a statistical framework that combines summary index tests with familywise error rate and false discovery rate corrections is implemented. The first technique reduces the number of tests conducted; the latter two techniques adjust the p values for multiple inference. The primary finding of the reanalysis is that girls garnered substantial short- and long-term benefits from the interventions, but there were no significant long-term benefits fo...

How to Select, Calculate, and Interpret Effect Sizes

Abstract: The objective of this article is to offer guidelines regarding the selection, calculation, and interpretation of effect sizes (ESs). To accomplish this goal, ESs are first defined and their important contribution to research is emphasized. Then different types of ESs commonly used in group and correlational studies are discussed. Several useful resources are provided for distinguishing among different types of effects and what modifications might be required in their calculation depending on a study's purpose and methods. This article should assist producers and consumers of research in understanding the role, importance, and meaning of ESs in research reports.

Promoting Positive Youth Development through School-Based Social and Emotional Learning Interventions: A Meta-Analysis of Follow-Up Effects.

Abstract: This meta-analysis reviewed 82 school-based, universal social and emotional learning (SEL) interventions involving 97,406 kindergarten to high school students (Mage = 11.09 years; mean percent low socioeconomic status = 41.1; mean percent students of color = 45.9). Thirty-eight interventions took place outside the United States. Follow-up outcomes (collected 6 months to 18 years postintervention) demonstrate SEL's enhancement of positive youth development. Participants fared significantly better than controls in social-emotional skills, attitudes, and indicators of well-being. Benefits were similar regardless of students’ race, socioeconomic background, or school location. Postintervention social-emotional skill development was the strongest predictor of well-being at follow-up. Infrequently assessed but notable outcomes (e.g., graduation and safe sexual behaviors) illustrate SEL's improvement of critical aspects of students’ developmental trajectories.

How Big Is “Big”? Interpreting Effect Sizes in L2 Research

Abstract: The calculation and use of effect sizes—such as d for mean differences and r for correlations—has increased dramatically in second language (L2) research in the last decade. Interpretations of these effects, however, have been rare and, when present, have largely defaulted to Cohen's levels of small (d = .2, r = .1), medium (.5, .3), and large (.8, .5), which were never intended as prescriptions but rather as a general guide. As Cohen himself and many others have argued, effect sizes are best understood when interpreted within a particular discipline or domain. This article seeks to promote more informed and field-specific interpretations of d and r by presenting a description of L2 effects from 346 primary studies and 91 meta-analyses (N > 604,000). Results reveal that Cohen's benchmarks generally underestimate the effects obtained in L2 research. Based on our analysis, we propose a field-specific scale for interpreting effect sizes, and we outline eight key considerations for gauging relative magnitude and practical significance in primary and secondary studies, such as theoretical maturity in the domain, the degree of experimental manipulation, and the presence of publication bias.