Showing papers in "Educational Researcher in 2014"

Facts Are More Important Than Novelty Replication in the Education Sciences

Abstract: Despite increased attention to methodological rigor in education research, the field has focused heavily on experimental design and not on the merit of replicating important results. The present study analyzed the complete publication history of the current top 100 education journals ranked by 5-year impact factor and found that only 0.13% of education articles were replications. Contrary to previous findings in medicine, but similar to psychology, the majority of education replications successfully replicated the original studies. However, replications were significantly less likely to be successful when there was no overlap in authorship between the original and replicating articles. The results emphasize the importance of third-party, direct replications in helping education research improve its ability to shape education policy and practice.

What’s Past Is Prologue: Relations Between Early Mathematics Knowledge and High School Achievement

Abstract: Although previous research has established the association between early-grade mathematics knowledge and later mathematics achievement, few studies have measured mathematical skills prior to school entry, nor have they investigated the predictive power of early gains in mathematics ability. The current paper relates mathematical skills measured at 54 months to adolescent mathematics achievement using multi-site longitudinal data. We find that preschool mathematics ability predicts mathematics achievement through age 15, even after accounting for early reading, cognitive skills, and family and child characteristics. Moreover, we find that growth in mathematical ability between age 54 months and first grade is an even stronger predictor of adolescent mathematics achievement. These results demonstrate the importance of pre-kindergarten mathematics knowledge and early math learning for later achievement.

Changing “Course” Reconceptualizing Educational Variables for Massive Open Online Courses

Abstract: In massive open online courses (MOOCs), low barriers to registration attract large numbers of students with diverse interests and backgrounds, and student use of course content is asynchronous and unconstrained. The authors argue that MOOC data are not only plentiful and different in kind but require reconceptualization—new educational variables or different interpretations of existing variables. The authors illustrate this by demonstrating the inadequacy or insufficiency of conventional interpretations of four variables for quantitative analysis and reporting: enrollment, participation, curriculum, and achievement. Drawing from 230 million clicks from 154,763 registrants for a prototypical MOOC offering in 2012, the authors present new approaches to describing and understanding user behavior in this emerging educational context.

Relevance to Practice as a Criterion for Rigor

Abstract: The authors argue for a reconceptualization of rigor that requires sustained, direct, and systematic documentation of what takes place inside programs to document how students and teachers change and adapt interventions in interactions with each other in relation to their dynamic local contexts. Building on promising new programs at the Institute of Education Sciences, they call for the formulation of collaborative research standards that must require researchers to provide evidence that they have engaged in a process to surface and negotiate the focus of their joint work, and to document the ways participation in this process was structured to include district and school leaders, teachers, parents, community stakeholders, and, wherever possible, children and youth. They close by describing how this new criterion—“relevance to practice”—can ensure the longevity and efficacy of educational research.

Moving Through MOOCs: Understanding the Progression of Users in Massive Open Online Courses

Abstract: This paper reports on the progress of users through 16 Coursera courses taught by University of Pennsylvania faculty for the first time between June 2012 and July 2013. Using descriptive analyses, this study advances knowledge by considering two definitions of massive open online course (MOOC) users (registrants and starters), comparing two approaches to measuring student progress through a MOOC course (sequential versus user driven), and examining several measures of MOOC outcomes and milestones. The patterns of user progression found in this study may not describe current or future patterns given the continued evolution of MOOCs. Nonetheless, the findings provide a baseline for future studies.

The Expanding Role of Philanthropy in Education Politics

Abstract: Philanthropic involvement in education politics has become bolder and more visible. Have foundations changed funding strategies to enhance their political influence? Using data from 2000, 2005, and 2010, we investigate giving patterns among the 15 largest education foundations. Our analyses show growing support for national-level advocacy organizations. Furthermore, we find that foundations increasingly fund organizations that operate as “jurisdictional challengers” by competing with traditional public sector institutions. We apply social network analysis to demonstrate the growing prevalence of convergent grant-making—multiple foundations supporting the same organizations. These results suggest that a sector once criticized for not leveraging its investments now increasingly seeks to maximize its impact by supporting alternative providers, investing concurrently, and supporting grantees to engage in policy debates.

Inaccurate Estimation of Disparities Due to Mischievous Responders Several Suggestions to Assess Conclusions

Abstract: This article introduces novel sensitivity-analysis procedures for investigating and reducing the bias that mischievous responders (i.e., youths who provide extreme, and potentially untruthful, responses to multiple questions) often introduce in adolescent disparity estimates based on data from self-administered questionnaires (SAQs). Mischievous responders affect a wide range of disparity estimates, including those between adoptees and nonadoptees, sexual minorities and nonminorities, and individuals with and without disabilities. Thus, the procedures introduced here have broad relevance to research and can be widely, and easily, implemented. The sensitivity-analysis procedures are illustrated with SAQ data from youths in Grades 9–12 (N = 11,829) to examine between-group disparities based on sexual identity, gender identity, and physical disability. Sensitivity analyses revealed that each disparity estimated with these data was extremely sensitive to the presence of potentially mischievous responders. Pat...

Teacher Evaluation Policy and Conflicting Theories of Motivation

Abstract: Current interest in teacher evaluation focuses disproportionately on measurement issues and performance-based pay without an overarching theory of how evaluation works. To develop such a theory, I contrast two motivation theories often used to guide thinking about teacher evaluation. External motivation theory relies on economics and extrinsic incentives. Internal motivation uses psychology and intrinsic incentives. These theories and available evidence raise doubts about performance-based pay, but not the use of other extrinsic incentives. These theories also suggest that to maintain effective intrinsic incentives, policies to remove ineffective teachers should not reduce autonomy or trust among effective teachers and that evaluations should provide teachers with useful feedback and policy makers with information on the conditions that facilitate good teaching.

Employing Multilevel Intersectionality in Educational Research Latino Identities, Contexts, and College Access

Abstract: The theoretical framework of intersectionality shows much promise in exploring how multiple social identities and their relationships with interlocking systems of power influence educational equity, particularly for historically underserved groups in education. Yet, social scientists have critiqued this framework for not adequately specifying how these dimensions shape life opportunities. This essay draws on the work of sociologist Floya Anthias to advance a conceptual model of intersectionality for educational research. This model addresses how different levels of analysis, types of practices, and relationships between social categories separately or together affect educational opportunities. To illustrate the model’s utility in research, policy, and practice, I apply this model to understand contextual influences on Latino im/migrant students’ college access.

“I Am Working-Class”: Subjective Self-Definition as a Missing Measure of Social Class and Socioeconomic Status in Higher Education Research

Abstract: This review provides a critical appraisal of the measurement of students’ social class and socioeconomic status (SES) in the context of widening higher education participation. Most assessments of social class and SES in higher education have focused on objective measurements based on the income, occupation, and education of students’ parents, and they have tended to overlook diversity among students based on factors such as age, ethnicity, indigeneity, and rurality. However, recent research in psychology and sociology has stressed the more subjective and intersectional nature of social class. The authors argue that it is important to consider subjective self-definitions of social class and SES alongside more traditional objective measures. The implications of this dual measurement approach for higher education research are discussed.

“Teaching to the Test” in the NCLB Era How Test Predictability Affects Our Understanding of Student Performance

Abstract: What is “teaching to the test,” and can one detect evidence of this practice in state test scores? This paper unpacks this concept and empirically investigates one variant of it by analyzing test item–level data from three states’ mathematics and reading tests. We show that NCLB-era state tests predictably emphasized some state standards while consistently excluding others; a small number of standards typically accounted for a substantial fraction of test points. We find that students performed better on items testing frequently assessed standards—those that composed a larger fraction of the state test in prior years—which suggests that teachers targeted their instruction towards these predictably tested skills. We conclude by describing general principles that should guide high-stakes test construction if a policy goal is to ensure that test score gains accurately represent gains in student learning.

The Test Matters The Relationship Between Classroom Observation Scores and Teacher Value Added on Multiple Types of Assessment

Abstract: In this study, we examined how the relationships between one observation protocol, the Protocol for Language Arts Teaching Observation (PLATO), and value-added measures shift when different tests are used to assess student achievement. Using data from the Measures of Effective Teaching Project, we found that PLATO was more strongly related to the Stanford Achievement Test (SAT-9), the alternative assessment used by MET to assess more ambitious outcomes. We also found that the SAT-9 is more instructionally sensitive to the PLATO factor of Cognitive and Disciplinary Demand than the state tests used in MET study. This difference suggests that PLATO factors designed specifically to identify ambitious instructional practices are especially sensitive to which test is used to construct value-added scores.

What Makes for Powerful Classrooms, and How Can We Support Teachers in Creating Them? A Story of Research and Practice, Productively Intertwined:

Abstract: This article, and my career as an educational researcher, are grounded in two fundamental assumptions: (1) that research and practice can and should live in productive synergy, with each enhancing the other; and (2) that research focused on teaching and learning in a particular discipline can, if carefully framed, yield insights that have implications across a broad spectrum of disciplines. This article begins by describing in brief two bodies of work that exemplify these two fundamental assumptions. I then elaborate on a third example, the development of a new set of tools for understanding and supporting powerful mathematics classroom instruction—and by extension, powerful instruction across a wide range of disciplines.

Toward Teacher Education Research That Informs Policy

Abstract: This article investigates the extent to which researchers are currently engaged in a shared research program that offers systematic evidence of the classroom impact of organized venues (preservice as well as inservice) for teacher professional learning. The article stems from concern about policies rooted in suspicion that teacher education is either ineffective or tangential to improving outcomes for students, as well as earlier findings that far too little teacher education research has been designed to address that suspicion with data (Cochran-Smith & Zeichner, 2005; Yoon, Duncan, Lee, Scarloss, and Shapely, 2007). An analysis of 196 articles published in 2012 in four leading teacher education journals internationally found only 1% to report large-scale mixed-methods studies, only 6% to examine the impact of teacher education on teaching practice and/or student learning, and most of the rest to be conducted within rather than across silos. Three recommendations for strengthening teacher education resea...

Tenth Annual Brown Lecture in Education Research A New Civil Rights Agenda for American Education

Abstract: This article reviews the impacts of the civil rights policies framed in the 1960s and the anti–civil rights political and legal movements that reversed them. It documents rising segregation by race and poverty. The policy reversals and transformation of U.S. demography require a new civil rights strategy. Vast immigrations, the sinking White birthrate and massive suburban change means it must be multiracial and metropolitan and reflect the huge increase in students from language-minority homes. School policy must be linked with social and economic policy. Housing integration is critical since residence is often destiny for children of color. Researchers are key participants in developing new policies and explaining possibilities for positive change within a stalemated political and legal system. The article outlines essential components of a new civil rights policy.

Beyond a Definition Toward a Framework for Designing and Specifying Mentoring Models

Abstract: More than three decades of mentoring research has yet to converge on a unifying definition of mentoring; this is unsurprising given the diversity of relationships classified as mentoring. This article advances beyond a definition toward a common framework for specifying mentoring models. Sixteen design elements were identified from the literature and tested through specification of two different mentoring models from higher education contexts. This framework provides researchers and practitioners with a detailed yet concise method of communicating exactly what they mean when using the word mentoring; it may also act as a useful set of prompts for educators designing new mentoring interventions.

Learning to Think Critically: A Visual Art Experiment

Abstract: This article examines whether exposure to the arts has an effect on the ability of students to engage in critical thinking. We conduct a randomized controlled trial involving 3,811 students who were assigned by lottery to participate in a School Visit Program at the newly opened Crystal Bridges Museum of American Art. Students who participated in the School Visit Program demonstrated significantly stronger critical thinking skills when analyzing a new painting. These effects were larger for students from more disadvantaged backgrounds. In light of recent declines in the availability of the arts for disadvantaged populations, our results have important policy implications for efforts to restore and expand access to the arts.

The Impact of Open Textbooks on Secondary Science Learning Outcomes

Abstract: Given the increasing costs associated with commercial textbooks and decreasing financial support of public schools, it is important to better understand the impacts of open educational resources on student outcomes. The purpose of this quantitative study is to analyze whether the adoption of open science textbooks significantly affects science learning outcomes for secondary students in earth systems, chemistry, and physics.This study uses a quantitative quasi-experimental design with propensity score matched groups and multiple regression to examine whether student learning was influenced by the adoption of open textbooks instead of traditional publisher-produced textbooks. Students who used open textbooks scored .65 points higher on end-of-year state standardized science tests than students using traditional textbooks when controlling for the effects of 10 student and teacher covariates. Further analysis revealed statistically significant positive gains for students using the open chemistry textbooks, w...

Evidence-Based Practices in a Changing World: Reconsidering the Counterfactual in Education Research

Abstract: Experimental and quasi-experimental designs are used in educational research to establish causality and develop effective practices. These research designs rely on a counterfactual model that, in simple form, calls for a comparison between a treatment group and a control group. Developers of educational practices often assume that the population from which control groups are drawn is unchanging in its behavior or performance. This is not always the case. Populations and study samples can change over time—sometimes dramatically so. We illustrate this important point by presenting data from 5 randomized control trials of the efficacy of Kindergarten Peer-Assisted Learning Strategies, a supplemental, peer-mediated reading program. The studies were conducted across 9 years and involved 2,591 students. Findings demonstrate a dramatic increase in the performance of control students over time, and suggest the need for a more nuanced understanding of the counterfactual model and its role in establishing evidence-...

Promoting Human Capital Development: A Typology of International Scholarship Programs in Higher Education

Abstract: This article sheds light on the availability and characteristics of international scholarship programs that are sponsored by national and federal governments worldwide and that are intended to promote student mobility. Utilizing descriptive and cluster analyses, the article produces a framework for organizing the population of these programs. The analyses take into account both the central characteristics of programs and economic and political characteristics of the nations sponsoring the program. The typology produced in this analysis may be used by policy makers and researchers to facilitate cross-national comparisons of program design, implementation, and outcomes.

Dynamic Diversity Toward a Contextual Understanding of Critical Mass

Abstract: Through an analysis of relevant social science evidence, this article provides a deeper understanding of critical mass, a concept that has become central in litigation efforts related to affirmative action admissions policies that seek to further the educational benefits of diversity. We demonstrate that the concept of critical mass requires an understanding of the conditions needed for meaningful interactions and participation among students, given the particular institutional context. To highlight this contextual definition of critical mass and to avoid further obfuscations in the legal debate, we offer the term dynamic diversity and outline four main components of dynamic diversity that institutions can attend to. By thinking of dynamic diversity as the goal, institutions and lawyers should be better poised to answer the question of how much diversity is necessary for leveraging its educational benefits.

Using Educative Curriculum Materials to Support the Development of Prospective Teachers’ Knowledge

Abstract: Building on the work of Ball and Cohen and that of Davis and Krajcik, as well as more recent research related to teacher learning from and about curriculum materials, we seek to answer the question, How can prospective teachers (PTs) learn to read and use educative curriculum materials in ways that support them in acquiring the knowledge needed for teaching? We present two extended conceptual examples of ways in which educative curriculum materials might be used to support PTs in developing the knowledge needed for teaching. We follow these examples with a set of empirically based design principles and conclude with a brief consideration of unanswered questions related to the use of educative curriculum materials in teacher education.

Mitigating the Dangers of a Single Story Creating Large-Scale Writing Assessments Aligned With Sociocultural Theory

Abstract: The dangers of a single story in current U.S. large-scale writing assessment are that assessment practice does not align with theory and this practice has negative effects on instruction and students. In this article, I analyze the connections or lack of connections among writing theory, writing assessment, and writing instruction, critique the construct and consequential validity of direct writing assessment and portfolio assessment, and reframe reliability as local consensus among experts. A new vision of large-scale sociocultural writing portfolios in K–12 education is offered that builds on the practices of past large-scale portfolio assessment but also encourages students to write in multiple languages/dialects and modes for multiple purposes. Another key feature of sociocultural portfolios is that students are encouraged to write for impact. These additional components will mitigate the dangers of a single story by ensuring that (a) assessment practice matches sociocultural writing theory, resulting...

Common Structural Design Features of Rubrics May Represent a Threat to Validity

Abstract: Rubrics for assessing student performance are often seen as providing rich information about complex skills. Despite their widespread usage, however, little empirical research has focused on whether it is possible for rubrics to validly meet their intended purposes. The authors examine a rubric used to assess students’ writing in a large-scale testing program. They present empirical evidence for the existence of a potentially widespread threat to the validity of rubric assessments that arose due to design features. In this research, an iterative tryout-redesign-tryout approach was adopted. The research casts doubt on whether rubrics with structurally aligned categories can validly assess complex skills. A solution is proposed that involves rethinking the structural design of the rubric to mitigate the threat to validity. Broader implications are discussed.

The Waive of the Future? School Accountability in the Waiver Era:

Abstract: Forty-two states and the District of Columbia have recently received waivers to the school accountability requirements of the No Child Left Behind Act (NCLB). As the prospects for reauthorizing the Act in the near term are dim, these new accountability systems will be law for at least several years. Drawing on a four-part framework from the measurement literature, we describe and critique the approved waiver accountability plans, comparing them to the NCLB accountability rules. We find a mixed bag—some states have made large improvements and others have not. Overall we conclude that states missed opportunities to design more effective school accountability systems that might minimize negative unintended consequences of these policies. The article concludes with suggestions for state and federal policy in light of the available literature.

Patterns and Trends in Grade Retention Rates in the United States, 1995–2010:

Abstract: Although grade retention may be consequential for a number of important educational and socioeconomic outcomes, we know surprisingly little about the actual rate at which students are made to repeat grades. We build on Hauser, Frederick, and Andrew's (2007) measure of grade retention using data from the 1995 through 2010 Current Population Surveys. We make technical improvements to their measure; provide more recent estimates; and validate the measure against external criteria. Our measure describes large disparities in grade retention rates by sex, race/ethnicity, geographic locale, and students' socioeconomic circumstances. However, both absolute retention rates and disparities in retention rates have declined markedly since 2005. We conclude by describing how our measures might be used to model the impact of economic and policy contexts on grade retention rates.

Conceptual and Methodological Problems in Research on College Undermatch

Abstract: Access to the nation’s most selective colleges remains starkly unequal, with students in the lowest income quartile constituting less than 4% of enrollment. A popular explanation for this phenomenon is that low-income students undermatch by attending less selective colleges when their credentials predict admission to more highly selective colleges. We identify three problematic assumptions in research on undermatching: (a) that researchers can differentiate colleges at the “margin that matters” for student outcomes; (b) that researchers can accurately predict who will be admitted at colleges that use holistic admission processes; and (c) that using achievement measures like SAT (Scholastic Assessment Test) scores to match students to colleges will reduce postsecondary inequality. We discuss the implications of these assumptions for future research on college choice and stratification.

Research in the Hard Sciences, and in Very Hard “Softer” Domains

Abstract: The author of this commentary argues that physical scientists are attempting to advance knowledge in the so-called hard sciences, whereas education researchers are laboring to increase knowledge and understanding in an “extremely hard” but softer domain. Drawing on the work of Popper and Dewey, this commentary highlights the relative similarities between hard sciences and education research in their rhetorical nature, while acknowledging the divergent paths of these two fields of inquiry with regard to prediction and generalizability. The author suggests that given the highly contextualized nature of educational processes, embedded in shifting complex social settings, and the relevance of all variables, very little education research is able to pursue predictive power.

Who Enters Teaching? Encouraging Evidence That the Status of Teaching Is Improving:

Abstract: The relatively low status of teaching as a profession is often given as a factor contributing to the difficulty of recruiting teachers, the middling performance of American students on international assessments, and the well-documented decline in the relative academic ability of teachers through the 1990s. Since the turn of the 21st century, however, a number of federal, state, and local teacher accountability policies have been implemented toward improving teacher quality over the objections of some who argue the policies will decrease quality. In this article, we analyze 25 years of data on the academic ability of teachers in New York State and document that since 1999 the academic ability of both individuals certified and those entering teaching has steadily increased. These gains are widespread and have resulted in a substantial narrowing of the differences in teacher academic ability between high- and low-poverty schools and between White and minority teachers. We interpret these gains as evidence th...

Teachers Choosing Rich Tasks The Moderating Impact of Technology on Student Learning, Enjoyment, and Preparation

Abstract: This article reports on the pioneering use in education of Discrete Choice Experiments (DCEs) to assess teachers’ decisions regarding deployment of rich tasks. The incorporation of this quantitative method into what is usually considered the domain of qualitative researchers is an innovative feature of this study. The DCEs enabled rigorous, reliable, and efficient investigation of the relationships between attributes of the complex environment in which teachers operate. The findings articulate the choices made by teachers related to rich task pedagogy, technology use, and other resources. Understanding the influences on these choices will inform the adoption and adaptation of productive technologies, improve dissemination of good practices, and enhance the design of digital technologies, resulting in better student learning outcomes.