Does School Autonomy Make Sense Everywhere? Panel Estimates from PISA *

TL;DR: In this paper, the authors construct a panel dataset from the four waves of international PISA tests spanning 20002009, comprising over one million students in 42 countries and identify the effect of school autonomy from within-country changes in the average share of schools with autonomy over key elements of school operations.

About: This article is published in Journal of Development Economics.The article was published on 2013-09-01 and is currently open access. It has received 268 citations till now. The article focuses on the topics: Autonomy & Developing country.

Summary

1. Introduction

• Virtually every country in the world accepts the importance of human capital investment as an element of economic development, but this has introduced a set of important policy questions about how best to pursue such investments.
• With local autonomy comes the possibility that individual schools pursue goals other than achievement maximization and a potential threat to maintaining common standards across the nation.
• But dealing with these issues through international comparisons – where institutional variation can be found – brings other identification issues related to variations in culture, governmental institutions, and other things that are difficult to measure.
• The authors develop a panel of international test results from the Programme for International Student Assessment (PISA), covering 42 countries and four waves that span a time period of ten years.

2. Conceptual Framework

• Substantial research has gone into understanding the determinants of educational achievement.
• The authors are specifically interested in investigating the decision-making institutions of different countries.
• In fact, economic models of school governance often suggest that greater autonomy can lead to increased efficiency of public schools (e.g., Hoxby (1999); Nechyba (2003)), because autonomy offers the possibility of using superior local knowledge, with positive consequences for outcomes.
• Furthermore, the quality of decision-making may also be inferior at the local level when the technical capabilities of local decision-makers to provide high-quality services are limited and when local communities lack the ability to ensure highquality services (see Galiani, Gertler, and Schargrodsky (2008)).
• The impact of autonomy on performance thus depends on the level of development, being positive in well-functioning systems but possibly even negative in dysfunctional systems.

3.1 Building a PISA Panel Database

• The authors empirical analysis relies on the Programme for International Student Assessment (PISA), an internationally standardized assessment conducted by the Organisation for Economic Co-operation and Development (OECD).
• PISA’s target population is the 15-year-old students in each country, regardless of the institution and grade they currently attend.
• In addition to the achievement data, PISA also provides a rich array of background information on each student and her school.
• In their analyses below, year fixed effects take account of this.
• But there is also one low-income country, seven lower-middle-income countries, and nine upper-middle-income countries in the sample.

3.2 Measuring School Autonomy

• The authors construct their measures of school autonomy for each country from the background questionnaires of the four PISA studies.
• In 2000 and 2003, principals were asked, “In your school, who has the main responsibility for …” For each of the enumerated areas, principals had to tick whether decisions were mainly a responsibility of the school’s governing board, the principal, department heads, or teachers as opposed to not being a responsibility of the school.
• In all four waves, respondents were explicitly allowed to tick as many options as appropriate in each area.
• At a higher level of achievement, Germany has increased school autonomy, particularly in course offerings, whereas countries such as Great Britain, Australia, Denmark, Ireland, and Sweden have all seen slight decreases in the autonomy measures.
• Obviously, the three autonomy areas on decisions that are related to academic content – namely courses offered, course content, and textbooks used – are highly correlated among each other, both in levels and in changes.

3.3 Descriptive Statistics

• Table 1 presents country-level means of the three autonomy measures, as well as mean PISA math scores, in 2000 and 2009.
• Figure 3 plots this measure of initial economic development against initial educational achievement, measured as the PISA math score in 2000.
• Comparing these achievement trends to the autonomy trends seen in Figure 2, there are many examples where the combined achievement and autonomy trends are consistent with increased autonomy, particularly over academic content, being bad in low-performing but good in high-performing countries.
• Great Britain, Australia, Denmark, Ireland, and Sweden all slightly reduced their autonomy, which is mirrored by slightly decreasing achievement.

4. Empirical Model

• To test the effect of autonomy on student achievement and its dependence on a country’s development level more formally, the authors make use of the education production function framework 13 introduced above.
• The formulation in equation (4) shows the main elements of their approach.
• But, by aggregating over all schools in the country and measuring autonomy by the proportion of schools with local autonomy, the authors eliminate the selection bias from school choice.
• A central component of the analysis is the possibility of significant interactions of institutional factors with other institutions or country-specific elements such as school accountability systems or level of capacity and stage of development.

5.1 Main Results

• Conventional estimation identifies the effect of autonomy from the cross-sectional variation.
• There is little indication that this association differs across levels of development, although the positive association of academic-content autonomy seems to increase slightly with a country’s development level, measured by the initial GDP per capita in 2000 (column 2).
• Thus, Table 4, which shows their main results, adds an interaction term of autonomy with initial GDP per capita to the panel specification with country fixed effects and with autonomy measured at the country level.
• For budget autonomy, the negative autonomy effect does not reach statistical significance for even the poorest country in their sample.
• The substantial correlation between the different categories of autonomy limits the extent to which the authors can distinguish among the three categories, but Table 5 presents models with pairs of two autonomy variables, as well as all three of them, combined.

5.2 Robustness Tests

• Several extended tests confirm the robustness of their main finding.
• The authors main model includes measures of school characteristics, but the final columns of Table 6 show that results are robust to alternative treatments of school controls.
• Column 5 of Table 7 replicates their analysis for the fully 22 As reviewed in Hanushek and Woessmann (2011), international comparative studies of the impact of expenditures provide mixed results but tend to indicate no consistent relationships between spending and international test scores.
• Additional robustness tests show that results also do not hinge on any specific country being included in the estimation.
• Results are qualitatively the same as in their preferred one-step specification, and they do not differ depending on whether the model does or does not already include country fixed effects in the estimation of the first step.

5.3 Specification Tests

• The authors identification derives from country-level variation in autonomy over time and its interaction with initial development levels in a panel model with country fixed effects.
• Given that the tests corroborate their main specification mostly by producing the result of insignificant alternative effects, the authors simply summarize the findings here.
• 20 concern, the authors estimate several models where the changes in autonomy that identify their results are regressed on initial PISA scores.
• Some of these measures may also be interpreted as possible channels through which the level of economic development may matter for the impact of autonomy on student achievement.
• 22 differential impacts by social background would narrow or widen the performance gap between well-off and disadvantaged families.

5.4 Further Results

• While the results so far relate to math achievement, which is most readily tested comparably across countries, PISA also tested students in reading and science.
• This is particularly interesting because reading scores have been psychometrically scaled to be comparable over all four PISA waves.
• While the interaction of autonomy with the initial per-capita GDP level remains qualitatively unaffected, there is also some indication that academic-content autonomy is more beneficial in more equal societies.
• This pattern is not confirmed by distributional measures of family background taken from the PISA dataset that directly relate to the parents of the tested students.
• Both negative and positive effects of autonomy are reduced when external education authorities may also have a say in decision-making.

6. Adding Accountability and Educational Development

• The prior analysis presumes that a country’s income level can sufficiently characterize the set of institutional features that are complementary to local autonomy in schools – including, for example, experience with general economic structures, the importance of the rule of law as seen in economic operations, generally functioning governmental institutions, and the like.
• It has the potential disadvantages of ignoring specific educational institutions and the overall development of the educational sector.
• Thus, the first column of Table 9 adds an interaction term between autonomy and central exit exams to their basic model.
• There is a sizeable positive interaction between (time-variant) school autonomy and the (time-invariant) measure of central exit exams, statistically significant in the case of academic-content autonomy.
• Similarly, results hold when measuring initial achievement by a dummy for countries scoring higher than the OECD mean of 500 PISA points (column 4).

7. Conclusions

• Decentralization of decision-making has been hotly debated in many countries of the world, and prior research has left considerable uncertainty about the expected impact of giving more autonomy to schools.
• Empirically, the main result proves highly consistent across a series of robustness and specification checks.
• Within this framework, the authors can exploit the pattern of policy changes within countries to obtain cleaner estimates of the institutional differences.
• The authors results indicate that the impact of school autonomy on student achievement is highly heterogeneous, varying by the level of development of a country.
• This overall result may have broader implications for the generalizability of findings across countries and education systems.

Figures

Table 1: Descriptive statistics by country

Table 4: Panel fixed-effects results on the effect of school autonomy on student achievement by development level

Figure 3: Development level and PISA performance, 2000

Table 8: Further results: Other subjects and joint authority

Table 3: Conventional cross-sectional estimation of the effect of school autonomy on student achievement

Table 7: Robustness: Including expenditure per student and different sub-samples of waves and countries

Table 6: Robustness: Different forms of measuring initial GDP per capita and different school controls

Figure 1: Performance on the PISA math tests, 2000-2009

Table 10: Alternative measure of development level: Initial level of student achievement

Table 5: Robustness: Impact of including several autonomy measures together in the same estimation

Table 2: Country-level correlation matrix of autonomy measures

Figure 2: School autonomy over courses and over hiring, 2000-2009

Table 9: Extended model: Including central exit exams

Frequently asked questions

Q1. What are the contributions in this paper?

The authors suggest that autonomy may be conducive to student achievement in well-developed systems but detrimental in low-performing systems. 

Q2. What are the future works in this paper?

Nonetheless, these specific issues require further research and confirmation. Within this framework, the authors can exploit the pattern of policy changes within countries to obtain cleaner estimates of the institutional differences. It suggests that lessons from educational policies in developed countries may not translate directly into advice for developing countries, and vice versa. 

Q3. What is the key to identification of, the parameter of interest?

The key to identification of α, the parameter of interest, is that ɛcti is orthogonal to theincluded explanatory factors and, importantly, to the measure of local autonomy. 

Q4. What is the effect of autonomy on math achievement?

In such a country, going from no autonomy to full autonomy over academic content would reduce math achievement by 0.34 standard deviations according to this model. 

Q5. Why do the authors not use the individual school measures of autonomy in the modeling of achievement?

As will be made explicit in the next section, the authors do not use the individual school measures of autonomy in the modeling of achievement because of concerns about introducing selection bias and because of the possibility of general-equilibrium effects. 

Q6. What are some of the factors that may affect the quality of local monitoring?

In addition, parental human capital may moderate the quality of local monitoring, their ability to pay for private schooling may affect the incentives of autonomous schools, and autonomous schools may use specific local policies. 

Q7. What is the effect of dropping all student observations with missing values?

Since the authors consider a large set of explanatory variables and since a portion of these variables is missing for some students, dropping all student observations with missing values would result in substantial sample reduction.