scispace - formally typeset

Journal ArticleDOI

A comment on "School choice: An experimental study" [J. Econ. Theory 127 (1) (2006) 202-231]

01 Jan 2011-Journal of Economic Theory (Academic Press)-Vol. 146, Iss: 1, pp 392-396

TL;DR: It is shown that one of the main results in Chen and Sonmez (2006, 2008) does no longer hold when the number of recombinations is sufficiently increased to obtain reliable conclusions.

AbstractWe show that one of the main results in Chen and Sonmez (2006, 2008) [6] , [7] does no longer hold when the number of recombinations is sufficiently increased to obtain reliable conclusions. No school choice mechanism is significantly superior in terms of efficiency.

Topics: Robustness (economics) (55%)

...read more

Content maybe subject to copyright    Report

A Comment On:
School Choice: An Experimental Study
Caterina Calsamiglia Guillaume Haeringer
Flip Klijn
August 8, 2009
Abstract
We show that one of the main results in Chen and onmez (2006, 2008) does no longer ho ld
when the number of recombinations is sufficiently increased to obtain reliable conclusions.
No school choice mechanism is significantly superior in terms of efficiency.
JEL classification: C70, C13, C91.
Keywords: school choice, efficiency, recombinant estimato r, robustness.
We consider the experimental study of Chen and onmez (2006,2008 —henceforth CS for
short).
1
CS’s experiment was intended to assess the relative performance of three school choice
mechanisms: Boston (BOS), Gale-Shapley (GS), and Top Trading Cycles (TTC). Their exper-
imental study complemented the mechanism design approach of Abdulkadiro˘glu and onmez
(2003) to study the assignment of children to public schools in the US. As such it played an im-
portant role to convince the Boston school district authorities to replace the previous mechanism
(BOS) by one of the other mechanisms.
2
The choice between GS and TTC mainly depended on
the relative weight that the authorities assigned to stability versus efficiency. Abdulkadiro˘glu
and onmez’s (2003) (theoretical) results are very clear: GS is stable (but not Pareto-efficient)
and TTC is Pareto-efficient (but not stable). However, CS’s “perhaps most surprising result ...
concerns th e efficiency comparison of the three mechanisms, as [their] experimental results do
not s upport theory” (CS, 2006, concluding discussion on p age 229). In particular, they find that
GS is significantly more efficient th an TTC. In this n ote we show that CS’s claim does no longer
hold when the number of recombinations is sufficiently increased to obtain robust conclusions.
More precisely, we will see that no school choice mechanism is significantly sup erior in terms of
efficiency.
We thank Yan Chen and Tayfun onmez for kindly sharing with us their data and for their comments which
helped us improve our understanding of their analysis. Comments from David Reiley and Charles Mullin are also
gratefully acknowledged. This research was supported through the Spanish Plan Nacional I+D+I (SEJ2005-01481
and SEJ2008-04784), the Generalitat de Catalunya (SGR2008-01142 and the Barcelona GSE Research Network),
and th e Consolider-Ingenio 2010 (CSD2006-00016) program.
Departament d’Economia i d’Hist`oria Econ`omica, Universitat Aut`onoma de Barcelona, Spain;
caterina.calsamiglia@uab.es, guillaume.haeringer@uab.es
Institute for Economic Analysis (CSIC), Campus UAB, 08193 Bellaterra (Barcelona), Spain;
flip.klijn@iae.csic.es
1
See onmez and
¨
Unver (2008) for a survey on recent developments about school choice mechanisms.
2
See Abdulkadiro˘glu et al. (2005) for a first report on the recent redesign of the Bost on Public S chool match.
1

CS considered two environments (one based on a designed preference profile, and the other
based on a randomly generated preference profile) and thus obtained 6 treatments.
3
For each
treatment they ran two sessions (i.e., n = 2), with k = 36 students in each session. CS employed
a recombinant estimation technique with r = 200 recombinations to obtain a refined analysis
of the relative efficiency of th e mechanisms. Their statistical analysis was based on t-tests. We
describe the recombinant technique as well as the statistical estimators in Section 1. Finally, in
Section 2, we show that when the number of recombinations is sufficiently increased in order to
obtain stable conclusions CS’s result that GS outperforms TTC can no longer be sustained.
1 Recombinant Technique and Estimators
Recombinant techniques are a useful tool to an alyze data obtained from laboratory experiments
based on normal form games.
4
The idea behind recombinant techniques is that as long as one is
interested in the analysis of th e outcome of the game (i.e., payoffs, n ot the strategies) running
the experiment a “few” times suffices to obtain more experimental data. More precisely, for the
game in CS one can generate up to n
k
= 2
36
“virtual” data sets by picking each of the k players’
strategies from either of the n sessions.
To avoid the computationally impossible task to calculate the outcomes induced by all virtual
data, CS employed the recombinant estimator proposed in Mullin and Reiley (2006), which
requires run ning fewer recombinations. In the case of the experimental data of CS the procedure
boils down to the following. One starts by picking the strategy of the first subject from the first
session, and then choosing randomly the strategies of player 2 up to 36 from either of the two
sessions. For this strategy profile the outcome of th e game is computed. Next, one repeats
the procedure by picking the strategy of the first su bject from the second s ession, and so on,
until one has done so for all subjects from both sessions. As a general guideline, Mullin and
Reiley (2006, page 177) recommend to repeat the procedure at least r
= 100 times for each of
the n × k subjects. CS opted for r = 200 recombinations.
Given the virtual data sets, CS compared the estimated mean payoff in each of the treat-
ments in order to evaluate the efficiency of the different mechanisms. To d etermine whether
the differences are statistically significant CS used t-tests based on the following estimators.
Consider any of the 6 treatments. For each of its n × k × r = 2 × 36 × 200 recombinations, let
Y (i, j, l) be the mean payoff of the l-th artificial session created by fixing player j from session
i. The estimated mean payoff over all recombinations is given by
ˆµ =
1
14400
2
X
i=1
36
X
j=1
200
X
l=1
Y (i, j, l) .
The estimated variance in payoffs is then given by
σ
2
=
1
14400
2
X
i=1
36
X
j=1
200
X
l=1
[Y (i, j, l) ˆµ]
2
.
3
See Chen and onmez (2006) for further details.
4
See for example Engelbrecht-Wiggans, List and Reiley (2006), Apesteguia, Dufwenberg and Selten (2007) or
Dufwenberg, Gneezy, Goeree and Nagel (2007) for recent applications of such techniques.
2

To compute th e covariance, CS split each of the 200 recombinations (i, j, ·) in two sets of 100
recombinations, an d compute the covariance across these two sets, i.e.,
φ =
1
7200
2
X
i=1
36
X
j=1
100
X
l=1
[Y (i, j, l) ˆµ] × [Y (i, j, l + 100) ˆµ] .
The asymptotic variance can then be estimated using Eq. (6.5) of Mullin and Reiley (2006),
5
var (ˆµ)
σ
2
36 × 200 × 2
+
36φ
2
.
2 Statistical Tests, Robustness, and Discrepancies
CS’s choice to generate 200 recombinations per subject-session follows Mullin and Reiley’s (2006)
suggestion to use at least 100 recombinations (per subject-session). Nevertheless, it turns out
that 200 recombinations is not sufficient to obtain robust statistics in such a rich game as the
one representing each treatment. The r esults we obtained when we carried out multiple series of
200 recombinations vary considerably from one series to another. For each of the 6 treatments
the mean payoff ˆµ and its variance σ
2
and covariance φ do not depend very much on the number
of recombinations. But the asymptotic variance, which p uts a higher weight on the covariance
as we increase the number of recombinations, decreas es with the number of recombinations,
thereby affecting the results of the tests.
6
To give an idea of this variation we generated 150
series of r = 200 (resp. 2000, 10000, and 100000) recombinations (per subject-session) for each
of the 6 treatments. Thus, in each case we obtained 150 × 150 = 22500 hypothesis tests for
the 6 pairs of treatments. If the percentage of acceptance is 0% or 100% then the associated
conclusion may be considered robust since all 22500 hypothesis tests led to the same conclusion.
Table 1 summarizes the proportions of acceptance rates.
x vs. y \ r 200 2000 10000 100000
GS
d
vs. BOS
d
99.87% 100.00% 100.00% 100.00%
TTC
d
vs. BOS
d
28.37% 0.20% 0.07% 0.00%
GS
d
vs. TTC
d
33.67% 40.78% 44.36% 33.28%
BOS
r
vs. GS
r
0.00% 0.00% 0.00% 0.00%
BOS
r
vs. TTC
r
28.11% 25.49% 6.98% 0.06%
GS
r
vs. TTC
r
17.52% 18.89% 7.11% 0.01%
Table 1: Acceptance rates of H
0
: ˆµ
x
> ˆµ
y
and H
1
: ˆµ
x
= ˆµ
y
.
5
Abrevaya (2008) provides evidence that Mullin and Reiley’s (2006) variance estimation can be downward
biased and provides a method to avoid this b ias. Our findings in the next Section are based on Mullin and
Reiley’s (2006) asymptotic variance but the qualitative results are also true with Abrevaya’s (2008) method. If
the result is that the difference between TTC and GS is not statistically significant with a downward biased
variance, the difference will be even less significant with a larger, less biased variance.
6
A first problem we encountered is that in many instances the estimated covariance from a given recombination
was negative. That implied that the estimated asymmetric variance was negative. This problem disappears when
the number of recombinations is increased.
3

Figure 1 additionally depicts the distributions of the p-values for 4 relevant cases. We
omitted the cases GS
d
vs. BOS
d
and BOS
r
vs. GS
r
since it is clear fr om Table 1 that the
associated conclusions are already very robust for r = 200. Note that in the remaining 4 cases
the distribution of the p-values has a high variance when the number of recomb inations is small.
When r = 2000, TTC
d
vs. BOS
d
also becomes robust. For r = 100000 all results are robust
except for GS
d
vs. TTC
d
. However, when r = 200000 this latter result also becomes (almost)
robust.
How should we rank the mechanisms in terms of efficiency? For the designed environment,
CS’s corrigendum on Result 6 concluded that TTC
d
BOS
d
, GS
d
>BOS
d
, and GS
d
> TTC
d
.
7
However, the values in Table 1 and the distributions in Figure 1 strongly suggest that in fact GS
d
does not outperform TTC
d
, i.e., GS
d
TTC
d
. For the random environment, CS’s corrigendum
on Result 6 concluded that GS
r
BOS
r
, GS
r
TTC
r
, and BOS
r
> TTC
r
. However, Table 1
and Figure 1 provide evidence that in fact BOS
r
TTC
r
.
As we have pointed out, we can no longer conclude that GS is superior to TTC in the
designed environment (which in contrast to the random environment was specifically constructed
to mimic a realistic environment
8
). In other words, our findings do not provide support to part
of CS’s “perhaps most surprising result ... [w hich] ... concerns the efficiency comparison of the
three mechanisms, as [their] experimental results do not support theory” (CS, 2006, concludin g
discussion on page 229).
7
Following CS’s notation, x > y denotes that x has a higher p er capita p ayo than y at the 5% significance
level or less, and x y denotes that x does not have a higher per capita payoff than y at the 5% significance level.
8
See CS for details.
4

0 50 100 150 200
0 .05 .1
p−value
TTCd vs. BOSd
0 50 100 150
0 .05 .1
p−value
TTCr vs. BOSr
0 50 100 150 200
0 .05 .1
p−value
GSd vs. TTCd
0 50 100
0 .05 .1
p−value
GSr vs. TTCr
kdensity recomb_200 kdensity recomb_2000
kdensity recomb_10000 kdensity recomb_100000
kdensity recomb_200000
Figure 1: kernel densities of p-values
5

Citations
More filters

Journal ArticleDOI

199 citations


Journal ArticleDOI
Abstract: The literature on school choice assumes that families can submit a preference list over all the schools they want to be assigned to. However, in many real-life instances families are only allowed to submit a list containing a limited number of schools. Subjects' incentives are drastically affected, as more individuals manipulate their preferentes. Including a safety school in the constrained list explains most manipulations. Competitiveness across schools plays an important role. Constraining choices increases segregation and affects the stability and efficiency of the final allocation. Remarkably, the constraint reduces significantly the proportion of subjects playing a dominated strategy.

189 citations


Journal ArticleDOI
Abstract: 31 pages, 18 tables.-- JEL classification: C72, C78, D78, I20.-- Trabajo publicado como articulo en American Economic Review 100(4): 1860-1874 (2010).-- http://dx.doi.org/10.1257/aer.100.4.1860

147 citations


Journal ArticleDOI
TL;DR: This work shows that even in simple environments with ample feedback and repetition, agents fail to reach non-truthtelling equilibria, and offers another way forward: implementing truth-telling as an ordinal Bayes–Nash equilibrium rather than as a dominant strategy equilibrium, showing that this weaker solution concept can allow for more efficient mechanisms in theory and in practice.
Abstract: While much of the school choice literature advocates strategyproofness, recent research has aimed to improve efficiency using mechanisms that rely on non-truthtelling equilibria. We address two issues that arise from this approach. We first show that even in simple environments with ample feedback and repetition, agents fail to reach non-truthtelling equilibria. We offer another way forward: implementing truth-telling as an ordinal Bayes–Nash equilibrium rather than as a dominant strategy equilibrium. We show that this weaker solution concept can allow for more efficient mechanisms in theory and provide experimental evidence that this is also the case in practice. In fact, truth-telling rates are basically the same whether truthtelling is implemented as an ordinal Bayes–Nash equilibrium or a dominant strategy equilibrium. This provides a proof-of-concept that ordinal Bayes–Nash design might provide a middle path, achieving efficiency gains over strategy-proof mechanisms without relying on real-life agents playing a non-truth-telling equilibrium.

52 citations


01 Jan 2013
Abstract: We characterize a parametric family of application-rejection school choice mechanisms, including the Boston and Deferred Acceptance mechanisms as special cases, and spanning the parallel mechanisms for Chinese college admissions, the largest centralized matching in the world. Moving from one extreme member to the other results in systematic changes in manipulability, stability and welfare properties. Neither the ex-post dominance of the DA over the Boston equilibria, nor the ex-ante dominance of the Boston equilibria over the DA in stylized settings extends to the parallel mechanisms. In the laboratory, participants are most likely to reveal their preferences truthfully under the DA mechanism, followed by the Chinese parallel and then the Boston mechanisms. Furthermore, while the DA is significantly more stable than the Chinese parallel mechanism, which is more stable than Boston, efficiency comparisons vary across environments.

50 citations


References
More filters

Journal ArticleDOI
Abstract: A central issue in school choice is the design of a student assignment mechanism. Education literature provides guidance for the design of such mechanisms but does not offer specific mechanisms. The flaws in the existing school choice plans result in appeals by unsatisfied parents. We formulate the school choice problem as a mechanism design problem and analyze some of the existing school choice plans including those in Boston, Columbus, Minneapolis, and Seattle. We show that these existing plans have serious shortcomings, and offer two alternative mechanisms each of which may provide a practical solution to some critical school choice issues.

1,241 citations


Journal ArticleDOI
Abstract: After the publication of “School Choice: A Mechanism Design Approach” by Abdulkadiroglu and Sonmez (2003), a Boston Globe reporter contacted us about the Boston Public Schools (BPS) system for assigning students to schools. The Globe article highlighted the difficulties that Boston’s system may give parents in strategizing about applying to schools. Briefly, Boston tries to give students their firstchoice school. But a student who fails to get her first choice may find her later choices filled by students who chose them first. So there is a risk in ranking a school first if there is a chance of not being admitted; other schools that would have been possible had they been listed first may also be filled. Valerie Edwards, then Strategic Planning Manager at BPS, and her colleague Carleton Jones invited us to a meeting in October 2003. BPS agreed to a study of their assignment system and provided us with micro-level data sets on choices and characteristics of students in the grades at which school choices are made (K, 1, 6, and 9), and school characteristics. Based on the pending results of this study, the Superintendent has asked for our advice on the design of a new assignment mechanism. This paper describes some of the difficulties with the current mechanism and some elements of the design and evaluation of possible replacement mechanisms. School choice in Boston has been partly shaped by desegregation. In 1974, Judge W. Arthur Garrity ordered busing for racial balance. In 1987, the U.S. Court of Appeals freed BPS to adopt a new, choice-based assignment plan. In 1999 BPS eliminated racial preferences in assignment and adopted the current mechanism.

517 citations


Journal ArticleDOI
TL;DR: The results suggest that replacing the Boston mechanism with either Gale-Shapley or Top Trading Cycles mechanism might significantly improve efficiency, however, the efficiency gains are likely to be more profound when parents are educated about the incentive compatibility of these mechanisms.
Abstract: We present an experimental study of three school choice mechanisms. The Boston mechanism is influential in practice, while the Gale–Shapley and Top Trading Cycles mechanisms have superior theoretical properties. Consistent with theory, this study indicates a high preference manipulation rate under Boston. As a result, efficiency under Boston is significantly lower than that of the two competing mechanisms in the designed environment. However, contrary to theory, Gale–Shapley outperforms Top Trading Cycles and generates the highest efficiency. Our results suggest that replacing the Boston mechanism with either Gale–Shapley or Top Trading Cycles mechanism might significantly improve efficiency.

307 citations


"A comment on "School choice: An exp..." refers result in this paper

  • ...We show that one of the main results in Chen and Sönmez (2006, 2008) [6,7] does no longer hold when the number of recombinations is sufficiently increased to obtain reliable conclusions....

    [...]


Journal ArticleDOI

199 citations


Journal ArticleDOI
Abstract: The literature on school choice assumes that families can submit a preference list over all the schools they want to be assigned to. However, in many real-life instances families are only allowed to submit a list containing a limited number of schools. Subjects' incentives are drastically affected, as more individuals manipulate their preferentes. Including a safety school in the constrained list explains most manipulations. Competitiveness across schools plays an important role. Constraining choices increases segregation and affects the stability and efficiency of the final allocation. Remarkably, the constraint reduces significantly the proportion of subjects playing a dominated strategy.

189 citations


Frequently Asked Questions (1)
Q1. What have the authors contributed in "A comment on: school choice: an experimental study" ?

The authors show that one of the main results in Chen and Sönmez ( 2006, 2008 ) does no longer hold when the number of recombinations is sufficiently increased to obtain reliable conclusions.