Journal of Economic Theory 146 (2011) 392–396
www.elsevier.com/locate/jet
A comment on “School choice: An experimental study”
[J. Econ. Theory 127 (1) (2006) 202–231]
✩
Caterina Calsamiglia
a,∗
, Guillaume Haeringer
b
, Flip Klijn
c
a
Departament d’Economia i d’Història Econòmica, Universitat Autònoma de Barcelona, Spain
b
Departament d’Economia i d’Història Econòmica, Universitat Autònoma de Barcelona, Spain
c
Institute for Economic Analysis (CSIC), Campus UAB, 08193 Bellaterra (Barcelona), Spain
Received 8 September 2009; accepted 7 February 2010
Available online 30 March 2010
Abstract
We show that one of the main results in Chen and Sönmez (2006, 2008) [6,7] does no longer hold when
the number of recombinations is sufficiently increased to obtain reliable conclusions. No school choice
mechanism is significantly superior in terms of efficiency.
© 2010 Elsevier Inc. All rights reserved.
JEL classification: C70; C13; C91
Keywords: School choice; Efficiency; Recombinant estimator; Robustness
✩
We thank Yan Chen and Tayfun Sönmez for kindly sharing with us their data and for their comments which
helped us improve our understanding of their analysis. Comments from David Reiley and Charles Mullin are also
gratefully acknowledged. Caterina Calsamiglia gratefully acknowledges support from the Spanish Ministry of Education
and Science through the Ramon y Cajal contract and grant SEJ2006-27589-E and FEDER. Caterina Calsamiglia and
Guillaume Haeringer gratefully acknowledge support from MOVE, the Fundacion Ramon Areces, the Spanish Ministry
of Science and Innovation through grant “Consolidated Group-C” ECO2008-04756, and by the Generalitat de Catalunya,
Departament d’Universitats, Recerca i Societat de la Informació through grant SGR2009-0419. Flip Klijn gratefully
acknowledges support from Plan Nacional I+D+i (ECO2008-04784), Generalitat de Catalunya (SGR2009-01142), and
the Consolider-Ingenio 2010 (CSD2006-00016) program. Flip Klijn is visiting Harvard Business School for academic
year 2009–10 and gratefully acknowledges a research fellowship from HBS. Support from the Barcelona GSE Research
Network is gratefully acknowledged.
DOI of original article: 10.1016/j.jet.2004.10.006.
*
Corresponding author.
E-mail addresses: caterina.calsamiglia@uab.es (C. Calsamiglia), guillaume.haeringer@uab.es (G. Haeringer),
flip.klijn@iae.csic.es (F. Klijn).
0022-0531/$ – see front matter © 2010 Elsevier Inc. All rights reserved.
doi:10.1016/j.jet.2010.03.004
C. Calsamiglia et al. / Journal of Economic Theory 146 (2011) 392–396 393
We consider the experimental study of Chen and Sönmez [6,7]—henceforth CS for short.
1
CS’s experiment was intended to assess the relative performance of three school choice mecha-
nisms: Boston (BOS), Gale–Shapley (GS), and Top Trading Cycles (TTC). Their experimental
study complemented the mechanism design approach of Abdulkadiro
˘
glu and Sönmez [2] to study
the assignment of children to public schools in the US. As such it played an important role to con-
vince the Boston school district authorities to replace the previous mechanism (BOS) by one of
the other mechanisms.
2
The choice between GS and TTC mainly depended on the relative weight
that the authorities assigned to stability versus efficiency. Abdulkadiro
˘
glu and Sönmez’s [2] (the-
oretical) results are very clear: GS is stable (but not Pareto-efficient) and TTC is Pareto-efficient
(but not stable). However, CS’s “perhaps most surprising result . .. concerns the efficiency com-
parison of the three mechanisms, as [their] experimental results do not support theory” (CS [6],
concluding discussion on p. 229). In particular, they find that GS is significantly more efficient
than TTC. In this note we show that CS’s claim does no longer hold when the number of recom-
binations is sufficiently increased to obtain robust conclusions. More precisely, we will see that
no school choice mechanism is significantly superior in terms of efficiency.
CS considered two environments (one based on a designed preference profile, and the other
based on a randomly generated preference profile) and thus obtained 6 treatments.
3
For each
treatment they ran two sessions (i.e., n = 2), with k = 36 students in each session. CS employed
a recombinant estimation technique with r = 200 recombinations to obtain a refined analysis
of the relative efficiency of the mechanisms. Their statistical analysis was based on t -tests. We
describe the recombinant technique as well as the statistical estimators in Section 1. Finally, in
Section 2, we show that when the number of recombinations is sufficiently increased in order to
obtain stable conclusions CS’s result that GS outperforms TTC can no longer be sustained.
1. Recombinant technique and estimators
Recombinant techniques are a useful tool to analyze data obtained from laboratory experi-
ments based on normal form games.
4
The idea behind recombinant techniques is that as long
as one is interested in the analysis of the outcome of the game (i.e., payoffs, not the strategies)
running the experiment a “few” times suffices to obtain more experimental data. More precisely,
for the game in CS one can generate up to n
k
= 2
36
“virtual” data sets by picking each of the k
players’ strategies from either of the n sessions.
To avoid the computationally impossible task to calculate the outcomes induced by all virtual
data, CS employed the recombinant estimator proposed in Mullin and Reiley [10], which requires
running fewer recombinations. In the case of the experimental data of CS the procedure boils
down to the following. One starts by picking the strategy of the first subject from the first session,
and then choosing randomly the strategies of player 2 up to 36 from either of the two sessions.
For this strategy profile the outcome of the game is computed. Next, one repeats the procedure
by picking the strategy of the first subject from the second session, and so on, until one has done
so for all subjects from both sessions. As a general guideline, Mullin and Reiley [10, p. 177]
1
See Sönmez and Ünver [11] for a survey on recent developments about school choice mechanisms.
2
See Abdulkadiro
˘
glu, Pathak, Roth, and Sönmez [1] for a first report on the recent redesign of the Boston Public
School match.
3
See Chen and Sönmez [6] for further details.
4
See for example Apesteguia, Dufwenberg, and Selten [4], Calsamiglia, Haeringer, and Klijn [5], Dufwenberg,
Gneezy, Goeree, and Nagel [8], or Engelbrecht-Wiggans, List, and Reiley [9] for recent applications of such techniques.
394 C. Calsamiglia et al. / Journal of Economic Theory 146 (2011) 392–396
recommend to repeat the procedure at least r
∗
= 100 times for each of the n × k subjects. CS
opted for r = 200 recombinations.
Given the virtual data sets, CS compared the estimated mean payoff in each of the treatments
in order to evaluate the efficiency of the different mechanisms. To determine whether the differ-
ences are statistically significant CS used t-tests based on the following estimators. Consider any
of the 6 treatments. For each of its n × k × r = 2 × 36 × 200 recombinations, let Y(i,j,l)be the
mean payoff of the l-th artificial session created by fixing player j from session i. The estimated
mean payoff over all recombinations is given by
ˆμ =
1
14400
2
i=1
36
j=1
200
l=1
Y(i,j,l).
The estimated variance in payoffs is then given by
σ
2
=
1
14400
2
i=1
36
j=1
200
l=1
Y(i,j,l)−ˆμ
2
.
To compute the covariance, CS split each of the 200 recombinations (i, j, ·) in two sets of 100
recombinations, and compute the covariance across these two sets, i.e.,
φ =
1
7200
2
i=1
36
j=1
100
l=1
Y(i,j,l)−ˆμ
×
Y(i,j,l+ 100) −ˆμ
.
The asymptotic variance can then be estimated using Eq. (6.5) of Mullin and Reiley [10],
5
var( ˆμ) ≈
σ
2
36 × 200 × 2
+
36φ
2
.
2. Statistical tests, robustness, and discrepancies
CS’s choice to generate 200 recombinations per subject-session follows Mullin and Reiley’s
[10] suggestion to use at least 100 recombinations (per subject-session). Nevertheless, it turns
out that 200 recombinations is not sufficient to obtain robust statistics in such a rich game as the
one representing each treatment. The results we obtained when we carried out multiple series of
200 recombinations vary considerably from one series to another. For each of the 6 treatments the
mean payoff ˆμ and its variance σ
2
and covariance φ do not depend very much on the number of
recombinations. But the asymptotic variance, which puts a higher weight on the covariance as we
increase the number of recombinations, decreases with the number of recombinations, thereby
affecting the results of the tests.
6
To give an idea of this variation we generated 150 series of
5
Abrevaya [3] provides evidence that Mullin and Reiley’s [10] variance estimation can be downward biased and
provides a method to avoid this bias. Our findings in the next section are based on Mullin and Reiley’s [10] asymptotic
variance but the qualitative results are also true with Abrevaya’s [3] method. If the result is that the difference between
TTC and GS is not statistically significant with a downward biased variance, the difference will be even less significant
with a larger, less biased variance.
6
A first problem we encountered is that in many instances the estimated covariance from a given recombination was
negative. That implied that the estimated asymmetric variance was negative. This problem disappears when the number
of recombinations is increased.
C. Calsamiglia et al. / Journal of Economic Theory 146 (2011) 392–396 395
Fig. 1. Kernel densities of p-values.
r = 200 (resp. 2000, 10 000, and 100 000) recombinations (per subject-session) for each of the 6
treatments. Thus, in each case we obtained 150 × 150 = 22 500 hypothesis tests for the 6 pairs
of treatments. If the percentage of acceptance is 0% or 100% then the associated conclusion
may be considered robust since all 22 500 hypothesis tests led to the same conclusion. Table 1
summarizes the proportions of acceptance rates.
Fig. 1 additionally depicts the distributions of the p-values for 4 relevant cases. We omitted
the cases GS
d
vs. BOS
d
and BOS
r
vs. GS
r
since it is clear from Table 1 that the associated con-
clusions are already very robust for r = 200. Note that in the remaining 4 cases the distribution of
396 C. Calsamiglia et al. / Journal of Economic Theory 146 (2011) 392–396
Table 1
Acceptance rates of H
0
: ˆμ
x
> ˆμ
y
and H
1
: ˆμ
x
=ˆμ
y
.
x vs. y Number of recombinations (r)
200 2000 10 000 100 000
GS
d
vs. BOS
d
99.87% 100.00% 100.00% 100.00%
TTC
d
vs. BOS
d
28.37% 0.20% 0.07% 0.00%
GS
d
vs. TTC
d
33.67% 40.78% 44.36% 33.28%
BOS
r
vs. GS
r
0.00% 0.00% 0.00% 0.00%
BOS
r
vs. TTC
r
28.11% 25.49% 6.98% 0.06%
GS
r
vs. TTC
r
17.52% 18.89% 7.11% 0.01%
the p-values has a high variance when the number of recombinations is small. When r = 2000,
TTC
d
vs. BOS
d
also becomes robust. For r = 100 000 all results are robust except for GS
d
vs.
TTC
d
. However, when r = 200 000 this latter result also becomes (almost) robust.
How should we rank the mechanisms in terms of efficiency? For the designed environment,
CS’s corrigendum on Result 6 concluded that TTC
d
BOS
d
,GS
d
> BOS
d
, and GS
d
> TTC
d
.
7
However, the values in Table 1 and the distributions in Fig. 1 strongly suggest that in fact GS
d
does not outperform TTC
d
, i.e., GS
d
TTC
d
. For the random environment, CS’s corrigendum
on Result 6 concluded that BOS
r
GS
r
,GS
r
TTC
r
, and BOS
r
> TTC
r
. However, Table 1
and Fig. 1 provide evidence that in fact BOS
r
TTC
r
.
As we have pointed out, we can no longer conclude that GS is superior to TTC in the de-
signed environment (which in contrast to the random environment was specifically constructed
to mimic a realistic environment
8
). In other words, our findings do not provide support to part
of CS’s “perhaps most surprising result . . . [which] . . . concerns the efficiency comparison of
the three mechanisms, as [their] experimental results do not support theory” (CS [6], concluding
discussion on p. 229).
References
[1] A. Abdulkadiro
˘
glu, P.A. Pathak, A.E. Roth, T. Sönmez, The Boston public schools match, in: Papers and Proceed-
ings, Amer. Econ. Rev. 95 (2005) 368–371.
[2] A. Abdulkadiro
˘
glu, T. Sönmez, School choice: a mechanism design approach, Amer. Econ. Rev. 93 (2003) 729–747.
[3] J. Abrevaya, On recombinant estimation for experimental data, Exper. Econ. 11 (2008) 25–52.
[4] J. Apesteguia, M. Dufwenberg, R. Selten, Blowing the whistle, Econ. Theory 31 (2007) 143–166.
[5] C. Calsamiglia, G. Haeringer, F. Klijn, Constrained school choice: an experimental study, Amer. Econ. Rev., forth-
coming.
[6] Y. Chen, T. Sönmez, School choice: An experimental study, J. Econ. Theory 127 (2006) 202–231.
[7] Y. Chen, T. Sönmez, Corrigendum to school choice: an experimental study, mimeo, 2008.
[8] M. Dufwenberg, U. Gneezy, J.K. Goeree, R. Nagel, Price floors and competition, Econ. Theory 33 (2007) 211–224.
[9] R. Engelbrecht-Wiggans, J.A. List, D.A. Reiley, Demand in multi-unit auctions with varying number of bidders:
theory and evidence from a field experiment, Int. Econ. Rev. 47 (2006) 203–231.
[10] C.H. Mullin, D.A. Reiley, Recombinant estimation for normal-form games with applications to auctions and bar-
gaining, Games Econ. Behav. 54 (2006) 159–182.
[11] T. Sönmez, U. Ünver, Matching, allocation, and exchange of discrete resources, in: A. Bisin, J. Benhabib, M. Jack-
son (Eds.), Handbook of Social Economics, Elsevier, forthcoming.
7
Here, x>ydenotes that x has a higher per capita payoff than y at the 5% significance level or less, and x y
denotes that x has a higher per capita payoff than y but not supported at the 5% significance level.
8
See CS for details.
