Showing papers in "Educational Researcher in 2015"

Measurement Matters: Assessing Personal Qualities Other Than Cognitive Ability for Educational Purposes

Abstract: There has been perennial interest in personal qualities other than cognitive ability that determine success, including self-control, grit, growth mindset, and many others. Attempts to measure such qualities for the purposes of educational policy and practice, however, are more recent. In this article, we identify serious challenges to doing so. We first address confusion over terminology, including the descriptor "non-cognitive." We conclude that debate over the optimal name for this broad category of personal qualities obscures substantial agreement about the specific attributes worth measuring. Next, we discuss advantages and limitations of different measures. In particular, we compare self-report questionnaires, teacher-report questionnaires, and performance tasks, using self-control as an illustrative case study to make the general point that each approach is imperfect in its own way. Finally, we discuss how each measure's imperfections can affect its suitability for program evaluation, accountability, individual diagnosis, and practice improvement. For example, we do not believe any available measure is suitable for between-school accountability judgments. In addition to urging caution among policymakers and practitioners, we highlight medium-term innovations that may make measures of these personal qualities more suitable for educational purposes.

Uneven Playing Field? Assessing the Teacher Quality Gap Between Advantaged and Disadvantaged Students

Abstract: Policymakers aiming to close the well-documented achievement gap between advantaged and disadvantaged students have increasingly turned their attention to issues of teacher quality. A number of studies have demonstrated that teachers are inequitably distributed across student subgroups by input measures, like experience and qualifications, as well as output measures, like value-added estimates of teacher performance, but these tend to focus on either individual measures of teacher quality or particular school districts. In this study, we present a comprehensive, descriptive analysis of the inequitable distribution of both input and output measures of teacher quality across various indicators of student disadvantage across all school districts in Washington State. We demonstrate that in elementary school, middle school, and high school classrooms, virtually every measure of teacher quality we examine—experience, licensure exam scores, and value added—is inequitably distributed across every indicator of stu...

The Role of Schooling in Perpetuating Educational Inequality An International Perspective

Abstract: In this paper, student-level indicators of opportunity to learn (OTL) included in the 2012 Programme for International Student Assessment are used to explore the joint relationship of OTL and socioeconomic status (SES) to student mathematics literacy. Using multiple methods, we find consistent evidence that (a) OTL has a significant relationship to student outcomes, (b) a positive relationship exists between SES and OTL, and (c) roughly a third of the SES relationship to literacy is due to its association with OTL. These relationships hold across most countries and both within and between schools within countries. Our findings suggest that in most countries, the organization and policies defining content exposure may exacerbate educational inequalities.

Minorities Are Disproportionately Underrepresented in Special Education: Longitudinal Evidence Across Five Disability Conditions

Abstract: We investigated whether minority children attending U.S. elementary and middle schools are disproportionately represented in special education. We did so using hazard modeling of multiyear longitudinal data and extensive covariate adjustment for potential child-, family-, and state-level confounds. Minority children were consistently less likely than otherwise similar White, English-speaking children to be identified as disabled and so to receive special education services. From kindergarten entry to the end of middle school, racial- and ethnic-minority children were less likely to be identified as having (a) learning disabilities, (b) speech or language impairments, (c) intellectual disabilities, (d) health impairments, or (e) emotional disturbances. Language-minority children were less likely to be identified as having (a) learning disabilities or (b) speech or language impairments.

Thinking Together and Alone

Abstract: Collaborative intellectual engagement is held in high regard in contemporary educational thought as a pedagogical practice of broad value to K–12 students. To what extent is this enthusiasm warranted? Is the practice uniformly productive, or does variability exist in the contexts in which collaboration is effective, the mechanisms involved, and the objectives achieved? In addition to examining these questions, this article suggests further questions that might be addressed with the objective of establishing a more comprehensive base of evidence to substantiate the practice of collaborative learning. Finally, the article reconsiders why collaborative cognition should be a critical concern.

What Is Improvement Science? Do We Need It in Education?

Abstract: The theory and tools of “improvement science” have produced performance improvements in many organizational sectors. This essay describes improvement science and explores its potential and challenges within education. Potential contributions include attention to the knowledge-building and motivational systems within schools, strategies for learning from variations in practice, and focus on improvement (rather than on program adoption). Two examples of improvement science in education are examined: the Community College Pathways Networked Improvement Community and lesson study in Japan. To support improvement science use, we need to recognize the different affordances of experimental and improvement science, the varied types of knowledge that can be generalized, the value of practical measurement, and the feasibility of learning across boundaries.

2014 AERA Distinguished Lecture Accelerating How We Learn to Improve

Abstract: A chasm is growing between our rapidly rising aspirations for our educational systems and what schools can routinely accomplish. Education needs an improvement paradigm—one that recognizes the complexity of the work of education and the wide variability in outcomes that our systems currently produce. This article sketches out such a paradigm. It joins together the discipline of improvement science with the power of structured networked communities to accelerate learning to improve. These networked improvement communities (NICs) combine analytic thinking and systematic methods to develop and test changes that can achieve better outcomes more reliably. NICs are inclusive in drawing together the expertise of practitioners, researchers, designers, technologists, and many others. And they organize their activities in ways akin to a scientific community. They develop practice-based evidence as an essential complement to findings from other forms of educational research. The point is not just to know what can ma...

Make Room Value Added Principals’ Human Capital Decisions and the Emergence of Teacher Observation Data

Abstract: Increasingly, states and districts are combining student growth measures with rigorous, rubric-aligned teacher observations in constructing teacher evaluation measures. Although the student growth or value-added components of these measures have received much research and policy attention, the results of this study suggest that the data generated by high-quality observation systems have potential to inform principals’ use of data for human capital decisions. Interview and survey data from six school districts that have recently implemented new evaluation systems with classroom observations provide evidence that principals tend to rely less on test scores in their human capital decisions. The consistency, transparency, and specificity of observation data may provide benefits for principals seeking to use these data to inform their decision making.

Can Value Added Add Value to Teacher Evaluation

Abstract: –137 DOI: 10.3102/0013189X15575346 © 2015 AERA. http://er.aera.net 132 EDUCATIONAL RESEARCHER The five thoughtful papers included in this issue of Educational Researcher (ER) raise new questions about the use of value-added methods (VAMs) to estimate teachers’ contributions to students’ learning as part of personnel evaluation. The papers address both technical and implementation concerns, considering potential effects on teachers’ behavior and on the resulting quality of teaching. In this response, I reflect on articles’ findings in light of other work in this field, and I offer my own thoughts about whether and how VAMs may add value to teacher evaluation.

Using Student Test Scores to Measure Teacher Performance: Some Problems in the Design and Implementation of Evaluation Systems.

Abstract: Our aim in this article is to draw attention to some underappreciated problems in the design and implementation of evaluation systems that incorporate value-added measures. We focus on four: (1) taking into account measurement error in teacher assessments, (2) revising teachers’ scores as more information becomes available about their students, and (3) and (4) minimizing opportunistic behavior by teachers during roster verification and the supervision of exams.

Experience-Sampling Research Methods and Their Potential for Education Research:

Abstract: Experience-sampling methods (ESM) enable us to learn about individuals’ lives in context by measuring participants’ feelings, thoughts, actions, context, and/or activities as they go about their daily lives. By capturing experience, affect, and action in the moment and with repeated measures, ESM approaches allow researchers access to expand the areas and aspects of participants’ experiences they can investigate and describe and to better understand how people and contexts shape these experiences. We argue ESM approaches can be particularly enriching for education research by enabling us to ask new and interesting questions about how students, teachers, and school leaders engage with education as they are living their lives and thus help us to better understand how education contexts shape learning and other outcomes. In this article, we highlight the value of these approaches for addressing new and exciting questions they may help education researchers to answer as they allow us to uncover experience in ...

The “Representative Bureaucracy” in Education: Educator Workforce Diversity, Policy Outputs, and Outcomes for Disadvantaged Students

Abstract: Bureaucratic representation—the idea that a governmental organization is better situated to serve its clients when its employee composition reflects that of its client population—has received considerable scholarly attention in the study of public institutions in the fields of political science and public administration. In a wide variety of settings, this research has demonstrated important connections between the racial, ethnic, and gender composition of the public sector workforce and how different groups—particularly traditionally underserved groups—interact with street-level bureaucrats and benefit from public services. Although scholars in those fields long ago recognized that the public school system is a large bureaucracy with diverse street-level bureaucrats (teachers) and clients (students and parents) and thus began studying bureaucratic representation in the context of schools, the concept and the causal mechanisms it hypothesizes remain largely unfamiliar to education researchers. This articl...

Science Achievement Gaps by Gender and Race/Ethnicity in Elementary and Middle School: Trends and Predictors

Abstract: Research on science achievement disparities by gender and race/ethnicity often neglects the beginning of the pipeline in the early grades. We address this limitation using nationally representative data following students from Grades 3 to 8. We find that the Black–White science test score gap (–1.07 SD in Grade 3) remains stable over these years, the Hispanic–White gap narrows (–.85 to –.65 SD), and the Asian–White Grade 3 gap (–.31 SD) closes by Grade 8. The female–male Grade 3 gap (–.23 SD) may narrow slightly by eighth grade. Accounting for prior math and reading achievement, socioeconomic status, and classroom fixed effects, Grade 8 racial/ethnic gaps are not statistically significant. The Grade 8 science gender gap disappears after controlling for prior math achievement.

Teacher Perspectives on Evaluation Reform: Chicago’s REACH Students

Abstract: In the 2012–2013 school year, Chicago Public Schools unveiled its new teacher evaluation system in all of its almost 600 schools. This study draws on 32 interviews from a random sample of teachers and 2 years of survey data from more than 12,000 teachers per year to measure their perceptions of the clarity, practicality, and cost of the new system. Relationships between these measures and teacher characteristics and indicators of leadership and school community are also explored. We find teachers are positive about the new system—especially the observation process. However, they have concerns about the inclusion of student growth in their evaluation. We find teacher perceptions about evaluation are positively correlated with their perceptions of leadership and professional community.

Toxic Rain in Class Classroom Interpersonal Microaggressions

Abstract: In this article we share exploratory findings from a study that captures microaggressions (MAs) in vivo to shed light on how they occur in classrooms. These brief and commonplace indignities communicate derogatory slights and insults toward individuals of underrepresented status contributing to invalidating and hostile learning experiences. Our aim is to expand the ways in which we research and think about MAs in educational settings. Our data are drawn from structured observations of 60 diverse classrooms on three community college campuses. Our findings provide evidence that classroom MAs occur frequently—in nearly 30% of the observed community college classrooms. Although cultural/racial as well as gendered MAs were observed, the most frequent types of MAs were those that undermined the intelligence and competence of students. MAs were more likely to be delivered on campuses with the highest concentration of minority students and were most frequently delivered by instructors. We conclude by reflecting ...

2014 Wallace Foundation Distinguished Lecture Rigor and Realism: Doing Educational Science in the Real World

Abstract: Transcending the low status of educational research will require demonstrating its relevance to improvements in practice. Educational progress is most likely to emerge from approaches to research that create an equal footing for practitioners and researchers, recognizing that though these groups accumulate and curate knowledge in different ways, they both have a role in creating tools (curricula, practices, professional development approaches) that can be used to forge lasting improvements. A brief history of the ongoing shift toward practice-embedded educational research (PEER) demonstrates its increasing acceptance and popularity and suggests modifications to the future selection of research topics, funding mechanisms, and professional preparation of both practitioners and researchers.

Exploring the Potential of Value-Added Performance Measures to Affect the Quality of the Teacher Workforce:

Abstract: The past decade has seen a tremendous amount of research on the use of value-added modeling to assess individual teachers, and a significant number of states and districts are now using, or plan to use, value added as a component of a teacher’s summative performance evaluation. In this article, I explore the various mechanisms through which the use of value added might affect teacher quality and describe what we know empirically about the potential of each mechanism. Given that many of these mechanisms work through the behavioral response of teachers to the high-stakes uses of evaluation, I argue that the jury is still out on how the use of value added will affect the quality of the teacher workforce.

Editors’ Introduction: The Use of Teacher Value-Added Measures in Schools: New Evidence, Unanswered Questions, and Future Prospects

Abstract: Teacher accountability based on teacher value-added measures could have far-reaching effects on classroom instruction and student learning, for good and for ill. To date, however, research has focused almost entirely on the statistical properties of the measures. While a useful starting point, the validity and reliability of the measures tell us very little about the effects on teaching and learning that come from embedding value added into policies like teacher evaluation, tenure, and compensation. We pose dozens of unanswered questions, not only about the net effects of these policies on measurable student outcomes, but about the numerous, often indirect ways in which these and less easily observed effects might arise. Drawing in part on other articles in the special issue, we consider perspectives from labor economics, sociology of organizations, and psychology. Some of the pathways of these policy effects directly influence teaching and learning and in intentional ways, while other pathways are indire...

Will VAMS Reinforce the Walls of the Egg-Crate School?

Abstract: Throughout the United States there is an increasing trend toward using value-added methods (VAMs) for high-stakes decisions. When policymakers use VAMs to identify, reward, and dismiss teachers, they may perpetuate the egg-crate model of schooling and undermine efforts to build instructional capacity schoolwide. At any time, in any school, some teachers are more knowledgeable, experienced, and skilled than others. Schools function best when they continuously leverage teachers’ expertise so that all students in all classrooms are well served. Drawing from research about the incentives and norms that influence teachers’ work within schools, this article illustrates what can happen when these methodologies are used to make job decisions and it identifies the hazards of using VAMs for this purpose. Contextualizing this within the larger discussion about performance evaluation systems, the article suggests how VAMs can be used productively as one source of information to promote improvement schoolwide.

Fifty Ways to Leave a Child Behind Idiosyncrasies and Discrepancies in States’ Implementation of NCLB

Abstract: The No Child Left Behind (NCLB) Act required states to adopt accountability systems measuring student proficiency on state-administered exams. The federal legislation contained several strict requirements for NCLB implementation, such as escalating student proficiency targets that reach 100% proficiency by 2014. But it also gave states considerable flexibility to interpret and implement components of NCLB. Using a data set we constructed, this paper is the first national study examining which schools failed during the early years of NCLB and which performance targets they failed to meet. We explore how states’ NCLB implementation decisions were related to their schools’ failure rates, which ranged from less than 1% to more than 80% across states. Wide cross-state variation in failure rates resulted from how states’ decisions interacted with each other and with school characteristics, like enrollment size, grade span, and ethnic diversity. Subtle differences in policy implementation may cause dramatic diff...

Reinventing the Role of the University Researcher

Abstract: This study examines the structuring of university–community research partnerships that facilitate theoretically grounded research while also generating findings that community partners find actionable. We analyze one partnership that positions university-based researchers as members of a team working to create, maintain, and use a longitudinal multiagency data source. Through our focus on the evolution of this university–community collaboration, we show how researchers established their commitment to a mutually beneficial exchange and how data-driven action emerged when community agencies assumed ownership and prioritized action throughout the research process.

Describing Profiles of Instructional Practice A New Approach to Analyzing Classroom Observation Data

Abstract: The authors outline the application of latent class analysis (LCA) to classroom observational instruments. LCA offers diagnostic information about teachers’ instructional strengths and weaknesses, along with estimates of measurement error for individual teachers, while remaining relatively straightforward to implement and interpret. It is discussed how the methodology can support formative feedback to educators and facilitate research into the associations between instructional practices and student outcomes. The approach is illustrated with a secondary analysis of data from the Measures of Effective Teaching study, focusing on middle school literacy instruction.

Performance Screens for School Improvement The Case of Teacher Tenure Reform in New York City

Abstract: Tenure is intended to protect teachers with demonstrated teaching skills against arbitrary or capricious dismissal. Critics of typical tenure processes argue that tenure assessments are superficial and rarely discern whether teachers in fact have the requisite teaching skills. A recent reform of the tenure process in New York City provides an unusual opportunity to learn about the role of tenure in teachers’ career outcomes. We find the reform led to many fewer teachers receiving tenure. Those not receiving tenure typically had their probationary periods extended to allow them an opportunity to demonstrate teaching effectiveness. These “extended” teachers were much more likely to leave their schools and be replaced by a teacher who was judged to be more effective.

Grade Assignments and the Teacher Pipeline A Low-Cost Lever to Improve Student Achievement?

Abstract: Research on teacher stability typically focuses on the extent to which teachers remain in the same school, district, or the teaching profession from one year to the next. I investigate another facet of stability—whether teachers remain in the grade they teach. Drawing on administrative data from a large district in California, I find that high shares of teachers switch grades. Disproportionately, these are early career teachers who come from low-achieving or high-minority schools. Teachers who switch grades leave schools at higher rates than their colleagues and exhibit lower impacts on their students’ achievement. For teachers who switch to a nonadjacent grade, these negative effects can wipe out any gains due to increased experience and can persist in the year after the switch occurs.

Guidance for Schools Selecting Antibullying Approaches Translating Evidence-Based Strategies to Contemporary Implementation Realities

Abstract: This article synthesizes the current research on bullying prevention and intervention in order to provide guidance to schools seeking to select and implement antibullying strategies. Evidence-based best practices that are shared across generally effective antibullying approaches are elucidated, and these strategies are grounded in examples garnered from model antibullying programs as implemented in contemporary schools. Future directions for practice, research, and policy are also explicated.

Seriously Considering Design in Educational Games

Abstract: Research suggests that well-designed games can be good for learning under the right conditions. How such games are designed remains poorly understood, as studies have focused more on whether games can produce learning than on how such games work or how they can be reliably developed. That is, though the design of a game is considered essential to its effectiveness, educational games lack a theory-informed definition and have predominantly shared design in terms of “principles” or “heuristics.” The aim of this paper is to discuss how we define and share educational game design and why design is important for improving educational game research and development.

Understanding the Gap in Special Education Enrollments Between Charter and Traditional Public Schools: Evidence From Denver, Colorado

Abstract: A widely cited report by the federal Government Accountability Office found that charter schools enroll a significantly smaller percentage of students with disabilities than do traditional public schools. However, thus far no hard evidence exists to definitively explain or quantify the disparity between special education enrollment rates in charter and traditional public schools. This article uses student-level data from Denver, Colorado, to map the creation and growth of the special education gap in elementary and middle school grades. The gap begins because students with disabilities are less likely to apply to charter schools in gateway grades than are nondisabled students. However, the special education gap in Denver elementary schools more than doubles as students progress between kindergarten and the fifth grade. About half of the growth in the gap in elementary grades (46%) occurs because of classification differences across sectors. The remaining 54% of the growth in the gap in elementary grades i...

Person-Organization Fit and Research on Instruction:

Abstract: Research from industrial and organizational (I-O) psychology indicates that outside of K–12 education, employees’ sense of fit with their organizations is often associated with job satisfaction, performance, commitment, and retention. Person-organization (P-O) fit has been conceptualized as the degree of congruence between an individual’s values, goals, and/or cognitive skills and abilities and the characteristics or requirements of their workplace. This essay reviews research from I-O psychology on how P-O fit predicts key outcomes for workers outside of K–12 education and discusses recent studies of P-O fit and teacher commitment and retention. We then theorize ways in which P-O fit can be used in research on teachers’ instruction, using research on teachers’ enactment of ambitious mathematics instruction as an example. Finally, the essay concludes by identifying directions for future research.

Syntactic Complexity as an Aspect of Text Complexity

Abstract: Students’ ability to read complex texts is emphasized in the Common Core State Standards (CCSS) for English Language Arts and Literacy. The standards propose a three-part model for measuring text complexity. Although the model presents a robust means for determining text complexity based on a variety of features inherent to a text as well as considerations outside the text, the grammar used in a text is not an overt component of the model. In this essay, we argue that the grammar of a text—especially, the syntactic complexity of sentences in a text—should be included as an explicit and distinct component in a text complexity model due to the fact that grammar contributes to the meaning of text and grammatical meaning impacts reading comprehension. We summarize findings from linguistics research on academic English to support this argument.

Gender Gaps in High School Students’ Homework Time:

Abstract: Gender differences in human capital investments made outside of the traditional school day suggest that males and females consume, respond to, and form habits relating to education differently. We document robust, statistically significant one-hour weekly gender gaps in secondary students’ non-school study time using time diary data from the 2003–2012 waves of the American Time Use Survey (ATUS) and transcript data from the Educational Longitudinal Study of 2002 (ELS). These complementary data sets provide consistent evidence of gender gaps that favor females and are not explained by gender differences in after-school time use, parental involvement, educational expectations, course taking, past academic achievement, or cognitive ability.