Showing papers in "Educational Evaluation and Policy Analysis in 2015"
TL;DR: This article found that students who begin post-secondary education at a community college are less likely to earn a bachelor's degree than otherwise similar undergraduates who begin at a 4-year school, but there is less consensus over the mechanisms generating this disparity.
Abstract: It is well established that students who begin post-secondary education at a community college are less likely to earn a bachelor’s degree than otherwise similar undergraduates who begin at a 4-year school, but there is less consensus over the mechanisms generating this disparity. We explore these using national longitudinal transcript data and propensity-score methods. Inferior academic preparation does not seem to be the main culprit: We find few differences between students’ academic progress at each type of institution during the first 2 years of college and (contrary to some earlier scholarship) students who do transfer have BA graduation rates equal to similar students who begin at 4-year colleges. However, after 2 years, credit accumulation diverges in the two kinds of institutions, due in part to community college students’ greater involvement in employment, and a higher likelihood of stopping out of college, after controlling for their academic performance. Contrary to some earlier claims, we fin...
230 citations
TL;DR: In this paper, the authors explore the promises and pitfalls of using National Student Clearinghouse (NSC) data to measure a variety of postsecondary outcomes and explore two potentially less salient sources of non-coverage: suppressed student records due to privacy laws and matching errors due to typographic inaccuracies in student names.
Abstract: This article explores the promises and pitfalls of using National Student Clearinghouse (NSC) data to measure a variety of postsecondary outcomes. We first describe the history of the NSC, the basic structure of its data, and recent research interest in using NSC data. Second, using information from the Integrated Postsecondary Education Data System (IPEDS), we calculate enrollment coverage rates for NSC data over time, by state, institution type, and demographic student subgroups. We find that coverage is highest among public institutions and lowest (but growing) among for-profit colleges. Across students, enrollment coverage is lower for minorities but similar for males and females. We also explore two potentially less salient sources of non-coverage: suppressed student records due to privacy laws and matching errors due to typographic inaccuracies in student names. To illustrate how this collection of measurement errors may affect estimates of the levels and gaps in postsecondary attendance and persist...
155 citations
TL;DR: In 2007, Washington adopted the Student Achievement Initiative (SAI), a statewide performance accountability system designed to improve retention rates and degree productivity among community colleges in the state as mentioned in this paper. But it was not widely used.
Abstract: In 2007, Washington adopted the Student Achievement Initiative, a statewide performance accountability system designed to improve retention rates and degree productivity among community colleges. U...
151 citations
TL;DR: The authors found that earning an associate degree or a long-term certificate is associated with increased wages, particularly for women, and that there is greater variation in returns to wages by the field of study than by degree type.
Abstract: This study provides one of the first estimates of the returns to different types of community college credentials—short-term certificates, long-term certificates, and associate degrees—across different fields of study. We exploit a rich data set that includes matched, longitudinal college transcripts and Unemployment Insurance records for students who entered a Washington State community college in 2001–2002. Our findings based on an individual fixed effect model suggest that earning an associate degree or a long-term certificate is associated with increased wages, particularly for women. We find that there is greater variation in returns to wages by the field of study than by degree type.
146 citations
TL;DR: The use of student test score data to measure teacher performance has fueled recent policy interest in using those data for measuring the effects of school administrators as well as discussed by the authors, however, the use of such data has been controversial.
Abstract: Expansion of the use of student test score data to measure teacher performance has fueled recent policy interest in using those data to measure the effects of school administrators as well. However...
144 citations
TL;DR: The authors investigated the differences in academic achievement trajectories from elementary through middle school among English Learner (EL) students in four different instructional programs: English Immersion (EI), Transitional Bilingual (TB), developmental bilingual (DB), and dual immersion (DI) and found that the English Language Arts (ELA) test scores of ELs in all bilingual programs grow at least as fast as, if not faster than, those in EI.
Abstract: This article investigates the differences in academic achievement trajectories from elementary through middle school among English Learner (EL) students in four different instructional programs: English Immersion (EI), Transitional Bilingual (TB), Developmental Bilingual (DB), and Dual Immersion (DI). Comparing students with the same parental preferences but who attend different programs, we find that the English Language Arts (ELA) test scores of ELs in all bilingual programs grow at least as fast as, if not faster than, those in EI. The same is generally true of math, with the exception of DB programs, where average student scores grow more slowly than those of students in EI. Furthermore, Latino ELs perform better longitudinally in both subjects when in bilingual programs than their Chinese EL counterparts. We find no differences in program effectiveness by ELs’ initial English proficiency.
140 citations
TL;DR: In this article, the primary aims of choice policies is to introduce competition between schools, and when parents can choose where to send their children, there is pressure on schools to improve to attract and retain students.
Abstract: One of the primary aims of choice policies is to introduce competition between schools. When parents can choose where to send their children, there is pressure on schools to improve to attract and ...
111 citations
TL;DR: The authors leveraged the random assignment of teachers to classrooms from the Measures of Effective Teaching (MET) study to identify teacher effectiveness using scores from the Framework for Teaching (FFT) instrument, one of the most widely used classroom observation protocols.
Abstract: Despite policy efforts to encourage multiple measures of performance in newly developing teacher evaluation systems, practical constraints often result in evaluations based predominantly on formal classroom observations. Yet there is limited knowledge of how these observational measures relate to student achievement. This article leverages the random assignment of teachers to classrooms from the Measures of Effective Teaching (MET) study to identify teacher effectiveness using scores from the Framework for Teaching (FFT) instrument, one of the most widely used classroom observation protocols. While our evidence suggests that teacher performance, as measured by the FFT, is correlated with student achievement, noncompliance with randomization and the modest year-to-year correlation of a teacher’s FFT scores constrain our ability to causally identify effective teachers. Implications for policy and practice are discussed.
103 citations
TL;DR: The test-optional movement in the United States emerged largely in response to criticism of standardized admissions tests as inadequate and potentially biased measures of postsecondary promise as mentioned in this paper, and it has been widely accepted as an alternative to the traditional admission test.
Abstract: The test-optional movement in the United States emerged largely in response to criticism of standardized admissions tests as inadequate and potentially biased measures of postsecondary promise. Alt...
93 citations
TL;DR: This article used a randomized design to assess the impact of the Balanced Leadership program on principal leadership, instructional climate, principal efficacy, staff turnover, and student achievement in a sample of rural northern Michigan schools.
Abstract: This study uses a randomized design to assess the impact of the Balanced Leadership program on principal leadership, instructional climate, principal efficacy, staff turnover, and student achievement in a sample of rural northern Michigan schools. Participating principals report feeling more efficacious, using more effective leadership practices, and having a better instructional climate than control group principals. However, teacher reports indicate that the instructional climate of the schools did not change. Furthermore, we find no impact of the program on student achievement. There was an impact of the program on staff turnover, with principals and teachers in treatment schools significantly more likely to remain in the same school over the 3 years of the study than staff in control schools.
79 citations
TL;DR: In this article, the effect of beginning at a community college on baccalaureate success in science, technology, engineering, and mathematics (STEM) fields was examined, drawing upon a new concept of "STEM momentum".
Abstract: This study examines the effect of beginning at a community college on baccalaureate success in science, technology, engineering, and mathematics (STEM) fields. In addition, drawing upon a new concept of “STEM momentum,” this research explores whether and how STEM baccalaureate success is related to early STEM momentum, as well as whether and how such momentum is shaped differently in community colleges as compared with public 4-year institutions. Results indicate that, overall, students beginning at community colleges were less likely to achieve STEM baccalaureate success. However, this negative effect was reduced to some extent by the positive influence of community colleges on a key STEM momentum indicator, quality points students achieved in STEM courses during the first term, which in turn positively impacted STEM baccalaureate success.
TL;DR: First-grade teachers in the United States may need to increase their use of teacher-directed instruction if they are to raise the mathematics achievement of students with MD.
Abstract: We used population-based, longitudinal data to investigate the relation between mathematics instructional practices used by 1st grade teachers in the U.S. and the mathematics achievement of their students. Factor analysis identified four types of instructional activities (i.e., teacher-directed, student-centered, manipulatives/calculators, movement/music) and eight types of specific skills taught (e.g., adding two-digit numbers). First-grade students were then classified into five groups on the basis of their fall and/or spring of kindergarten mathematics achievement-three groups with mathematics difficulties (MD) and two without MD. Regression analysis indicated that a higher percentage of MD students in 1st grade classrooms was associated with greater use by teachers of manipulatives/calculators and movement/music to teach mathematics. Yet follow-up analysis for each of the MD and non-MD groups indicated that only teacher-directed instruction was significantly associated with the achievement of students with MD (covariate-adjusted ESs = .05-.07). The largest predicted effect for a specific instructional practice was for routine practice and drill. In contrast, for both groups of non-MD students, teacher-directed and student-centered instruction had approximately equal, statistically significant positive predicted effects (covariate-adjusted ESs = .03-.04). First-grade teachers in the U.S. may need to increase their use of teacher-directed instruction if they are to raise the mathematics achievement of students with MD.
TL;DR: For example, the authors found that enrolling more students in advanced courses has negative average effects on students' achievement, driven by negative effects in large districts, and that the effects of increasing 8th-grade Algebra enrollment rates on a 10th grade mathematics achievement measure was not well understood.
Abstract: The United States is in the midst of an effort to intensify middle school mathematics curricula by enrolling more 8th graders in Algebra. California is at the forefront of this effort, and in 2008, the state moved to make Algebra the accountability benchmark test for 8th-grade mathematics. This article takes advantage of this unevenly implemented policy to understand the effects of curricular intensification in middle school mathematics. Using district-level panel data from all California K–12 public school districts, we estimate the effects of increasing 8th-grade Algebra enrollment rates on a 10th-grade mathematics achievement measure. We find that enrolling more students in advanced courses has negative average effects on students’ achievement, driven by negative effects in large districts.
TL;DR: This paper used a natural experiment to estimate the effects of need-based aid policies on first-year college persistence rates and found that the new state financial aid policy increased the likelihood that students attend 4-year campuses and increased their first year grade point averages.
Abstract: This article exploits a natural experiment to estimate the effects of need-based aid policies on first-year college persistence rates. In fall 2006, Ohio abruptly adopted a new state financial aid policy that was significantly more generous than the previous plan. Using student-level data and very narrowly defined sets of students, I estimate a difference-in-differences model to identify the program effects. Students who benefited from the program received awards about US$800 higher than they would have received under the prior program. These students’ drop-out rates fell by 2% as a result of the program. The new program also increased the likelihood that students attend 4-year campuses and increased their first-year grade point averages. The program may not have been cost-effective given the combination of its generosity and inability to target the marginal students who would be most sensitive to financial aid.
TL;DR: In this article, the authors examine from both perspectives why more of these research partnerships are needed, why they are not more common, and how these barriers can be overcome, using an example of a recently formed partnership with one of the nation's largest school districts.
Abstract: Research partnerships between school districts and universities can be extremely beneficial to both institutions, but these partnerships require many skills that academics and district leaders generally do not have, making these collaborations challenging to set up and difficult to maintain. Co-authored by a university professor and a school district leader, this article examines from both perspectives why more of these research partnerships are needed, why they are not more common, and how these barriers can be overcome, using an example of a recently formed partnership with one of the nation's largest school districts.
TL;DR: The authors examined the communitywide effects of investments in two early childhood initiatives in North Carolina (Smart Start and More at Four) on the likelihood of a student being placed into early childhood programs.
Abstract: This study examines the community-wide effects of investments in two early childhood initiatives in North Carolina (Smart Start and More at Four) on the likelihood of a student being placed into sp...
TL;DR: In this article, the authors present findings from a lottery-based study of the impacts of a broad set of 33 charter middle schools across 13 states on student achievement, comparing test score outcomes of students admitted to these schools through the randomized admissions lotteries with outcomes of applicants who were not admitted.
Abstract: This article presents findings from a lottery-based study of the impacts of a broad set of 33 charter middle schools across 13 states on student achievement. To estimate charter school impacts, we compare test score outcomes of students admitted to these schools through the randomized admissions lotteries with outcomes of applicants who were not admitted. We find that impacts varied considerably across schools and students, with more positive impacts for more disadvantaged schools and students and more negative impacts for the more advantaged. On average across the schools in the study, the impacts of charter middle schools on student achievement were negative but not statistically significant, regardless of whether we examined the impact of the offer of admission or actual attendance at these schools.
TL;DR: This paper used a difference-in-differences analytic approach to estimate postsecondary consequences from Maine's mandate that all public school juniors take the SAT® and found that, overall, the policy increased 4-year college-going rates by 2- to 3-percentage points.
Abstract: We use a difference-in-differences analytic approach to estimate postsecondary consequences from Maine’s mandate that all public school juniors take the SAT®. We find that, overall, the policy increased 4-year college-going rates by 2- to 3-percentage points and that 4-year college-going rates among induced students increased by 10-percentage points.
TL;DR: The authors found that being required to take a remedial course either in math or in English significantly reduces a student's probability of success in college and also the probability that a student ever passes a college-level math or English course.
Abstract: This article contributes to the empirical literature on remediation in community colleges by using policy variation across North Carolina’s community colleges to examine how remediation affects various outcomes for traditional-age college students. We find that being required to take a remedial course (as we define it in this article) either in math or in English significantly reduces a student’s probability of success in college and also the probability that a student ever passes a college-level math or English course. Among students who are required to take a remedial course in their first semester, however, we find no adverse effects on the probability of returning for another semester. We also find differential effects by a student’s prior achievement level, family income, and gender. Despite methodological differences, our main findings are generally consistent with, albeit somewhat more negative, than those from prior studies based on regression discontinuity designs.
TL;DR: Community colleges play a vital role in postsecondary education, enrolling more than one in every three postsecondary students as mentioned in this paper, while their importance has grown over the past 50 years, their studen...
Abstract: Community colleges play a vital role in postsecondary education, enrolling more than one in every three postsecondary students. While their importance has grown over the past 50 years, their studen...
TL;DR: This study analyzed the 12 state RttT grantees’ applications and revealed that states designed large implementation networks with potential to bring a wide range of resources to bear on reform efforts, particularly through participation of numerous nonsystem actors.
Abstract: This study sought to understand the opportunities and challenges associated with the implementation of state designed Race to the Top (RttT) funded reform networks. Drawing on a conceptual framework developed from the networked governance literature, we analyzed the 12 state RttT grantees’ applications. Our analysis revealed that states designed large implementation networks with potential to bring a wide range of resources to bear on reform efforts, particularly through participation of numerous nonsystem actors. However, coordinating large and diverse networks places state education agencies (SEAs) in a new and challenging role. The extent to which networks extend state capacity to support educational improvement or further complicate the work of SEAs remains an open question. We propose a model including a set of theoretical propositions to guide future research.
TL;DR: In this article, the effect of transitory changes in school size on the academic achievement of fourth and fifth-grade students in North Carolina using student-level longitudinal administrative data is estimated.
Abstract: Evidence on optimal school size is mixed. We estimate the effect of transitory changes in school size on the academic achievement of fourth- and fifth-grade students in North Carolina using student-level longitudinal administrative data. Estimates of value-added models that condition on school-specific linear time trends and a variety of teacher-by-school, student, and school-by-year fixed effects suggest that, on average, there is no causal relationship between school size and academic performance. However, two subgroups of interest are significantly harmed by school size: socioeconomically disadvantaged students and students with learning disabilities. The largest effects are observed among students with learning disabilities: A 10-student increase in grade size is found to decrease their math and reading achievement by about 0.015 test-score standard deviations.
TL;DR: This paper found that teacher coaching can improve teaching practices and student achievement, but little is known about specific features of effective coaching programs, and they did not estimate the effectiveness of coaching programs.
Abstract: Although previous research has shown that teacher coaching can improve teaching practices and student achievement, little is known about specific features of effective coaching programs. We estimat...
TL;DR: English as a second language (ESL) courses seek to address a primary barrier to college success for language minority students: second language issues that can inhibit their success in college-level courses as mentioned in this paper.
Abstract: English as a second language (ESL) courses seek to address a primary barrier to college success for language minority students: second language issues that can inhibit their success in college-leve...
TL;DR: However, there are overlooked methodological problems in the way this design has typically been applied that have the potential to produce biased effect estimates as mentioned in this paper, and they deserve more attention from researchers using this design than they have received.
Abstract: Much of the currently available evidence on the causal effects of public prekindergarten programs on school readiness outcomes comes from studies that use a regression-discontinuity design (RDD) with the age cutoff to enter a program in a given year as the basis for assignment to treatment and control conditions. Because the RDD has high internal validity when its key assumptions are met, these studies appear to provide strong evidence for the effectiveness of these programs. However, there are overlooked methodological problems in the way this design has typically been applied that have the potential to produce biased effect estimates. We describe these problems, argue that they deserve more attention from researchers using this design than they have received, and offer suggestions for improving future studies.
TL;DR: This paper found that teachers successfully changed their practice toward reform-aligned instructional goals only in schools demonstrating high growth in providing reformaligned professional development (both in the content and process of their PD).
Abstract: The research on professional development (PD) consists mostly of studies utilizing cross-sectional data. We examined effects of change in school-level PD on change in teachers’ practice longitudinally. Using survey reports from 1,722 teachers in 31 schools implementing a popular comprehensive school reform (America’s Choice), we found that although schools were implementing the same reform model, they varied in their implementation of PD. Teachers successfully changed their practice toward reform-aligned instructional goals only in schools demonstrating high growth in providing reform-aligned PD (both in the content and process of their PD). Our findings have implications for understanding interventions in schools and for thinking about the rigor of PD effects on teaching.
TL;DR: In 2010, the Wake County Public Schools began assigning middle school students to accelerated math and eighth grade algebra based on a defined prior achievement metric, which reduced the relationship between course assignment and student characteristics such as income and race/ethnicity while increasing its relationship to academic skill.
Abstract: Taking algebra by eighth grade is considered an important milestone on the pathway to college readiness. We highlight a collaboration to investigate one district’s effort to increase middle school algebra course-taking. In 2010, the Wake County Public Schools began assigning middle school students to accelerated math and eighth-grade algebra based on a defined prior achievement metric. This policy reduced the relationship between course assignment and student characteristics such as income and race/ethnicity, while increasing its relationship to academic skill. The policy increased the share of students on track for algebra by eighth grade. Students placed in accelerated math were exposed to higher-skilled peers but larger classes. Future work will assess impacts on subsequent achievement and course-taking outcomes.
TL;DR: In this paper, the authors examined the impact of charters on student achievement and found that on average, they were modestly less effective than traditional public schools in raising student achievement in some subjects.
Abstract: Arizona enrolls a larger share of its students in charter schools than any other state in the country, but no comprehensive examination exists of the impact of those schools on student achievement. Using student-level data covering all Arizona students from 2006 to 2012, we find that the performance of charter schools in Arizona in improving student achievement varies widely, and more so than that of traditional public schools (TPS). On average, charter schools at every grade level have been modestly less effective than TPS in raising student achievement in some subjects. But charter schools that closed during this period have been lower performing than schools that remained open, a pattern that is not evident in the traditional public sector.
TL;DR: In this paper, the estimation of racial test score gap trends plays an important role in monitoring educational equality, however, it is complex and estimates can differ depending on the metr.
Abstract: The estimation of racial test score gap trends plays an important role in monitoring educational equality Documenting gap trends is complex, however, and estimates can differ depending on the metr
TL;DR: In this paper, state longitudinal data systems (SLDSs) have created more opportunities than ever before for rigorous research to influence education policy decisions, but the data in SLDSs will not answer many of our most pressing research questions, nor will the presence of these systems create meaningful collaboration between researchers and practitioners that we feel is needed to inform our states' policy landscapes.
Abstract: State longitudinal data systems (SLDSs) have created more opportunities than ever before for rigorous research to influence education policy decisions. As state practitioners who play central roles in building and using our states’ longitudinal data systems, we are excited about their promise for supporting policymaking and research. Yet, we also recognize that the data in SLDSs will not answer many of our most pressing research questions, nor will the presence of these systems create the meaningful collaboration between researchers and practitioners that we feel is needed to inform our states’ policy landscapes. The barriers to the kinds of research we need are mostly unrelated to the promises of SLDSs. We outline the challenges we have experienced in developing research agendas, building our internal capacity for research, and working with external partners, and we identify the research questions we need to answer that are not easily addressed with SLDS data.