1
Critiques of socio-economic school compositional effects: Are they valid?
Michael G. Sciffer (Corresponding Author)
School of Education, Murdoch University, 90 South Street, Murdoch, Western Australia 6150.
ORCID: 0000-0003-2552-8840
research@sciffer.info
Laura B. Perry
Associate Professor Education Policy and Comparative Education, School of Education, Murdoch
University.
ORCID: 0000-0003-4252-2379
l.perry@murdoch.edu.au
Andrew McConney
Associate Professor Program Evaluation, Research Methods and Classroom Assessment, School of
Education, Murdoch University.
ORCID: 0000-0001-7618-8829
This is an original manuscript/preprint of an article published by Taylor & Francis in
the British Journal of Sociology of Education on 24/4/2020, available online:
http://www.tandfonline.com/10.1080/01425692.2020.1736000
The published article in the British Journal of Sociology of Education contains
substantial modifications.
2
Abstract
Recent studies have rejected school socio-economic compositional effects based on criticisms
of the methodologies of prior studies and their own findings. We respond to the critiques of
ecological fallacies and lack of control for prior achievement in school compositional research.
We describe how prior ability control variables and fixed-effects methods have been
inappropriately applied in research critical of compositional effects. We demonstrate that
structural equation modeling can address concerns about the inflation of level-2 effects due to
level-1 measurement error whilst also finding significant socio-economic school compositional
effects. We conclude that the veracity of school socio-economic composition effects has not
been weakened by recent critical studies and remain a profound issue for researchers and
policymakers.
3
The socio-economic compositional (SEC) effect is the relationship between the socio-
economic profile of a school and individual student outcomes such as academic achievement,
attendance, completion and tertiary entrance (Rumberger & Palardy, 2005). It is a separate
construct from the relationship between individual student socio-economic status (SES) and
performance outcomes (Rumberger & Palardy, 2005). It can be thought of as the difference in
performance between two students who have the same SES due to attending schools with
different socio-economic profiles (Raudenbush and Bryk, 2002, p. 141). SEC is usually
measured through the aggregation of the SES of the students in a school or class (Willms,
2010).
A range of mechanisms have been identified that explain the relationship between
SEC and schooling outcomes. Rumberger and Palardy (2005) found that teacher
expectations, hours of homework, number of academic courses taken, and student’s sense of
safety mediate the relationship between SEC and academic achievement growth. Willms
(2010) found that quality of instruction, student engagement, curriculum coverage,
instructional time, and adequacy of school resources mediate SEC and academic
achievement. Palardy (2013) found that peer effects and school resources mediate SEC and
graduation and college enrolment rates.
School composition has been an influential construct in school effectiveness research
and broader policy reforms since the Coleman Report found that it accounted for between 3%
and 33% of the variation in performance between schools, depending on ethnic background
and grade (Coleman et al., 1966, p. 299). Lamb and Fullarton (2002) found that socio-
economic composition and tracking had larger effects at the classroom and school levels on
mathematics achievement than teacher quality, student attitudes, student beliefs and amount
of homework in Australia and the US. Chiu and Khoo (2005) found the degree of clustering
of students in schools according to parental occupational status inversely related to national-
4
level academic performance in mathematics and science. In his meta-analytical review of the
literature, Sirin (2005) found that aggregated measures of SES were stronger predictors of
student outcomes than student-level measures
1
. Successive cycles of the OECD’s Programme
for International Student Assessment (PISA) have consistently found that socio-economic
factors are stronger predictors of academic achievement between schools compared to within
schools in most participating countries (OECD, 2003, 2004, 2007, 2010, 2013, 2016). Policy
responses have led to programmes aimed at reducing socio-economic segregation between
schools (Kahlenberg, 2007) and targeting resources to lower SEC schools (Gonski et al.,
2011).
However despite this substantial body of research evidence, a series of studies by
Marks and colleagues (Armor, Marks & Malatinsky, 2018; Marks, 2010, 2015, 2017) have
challenged the substantiveness of SEC effects based on their view of research methodologies
used in school compositional research and their own research findings. They have argued that
school compositional effects may be statistical artefacts arising from inappropriate
methodologies. These works are a subset of Marks’ broader research programme critical of
the role of socio-economic status in education policy (Marks, 2014, 2016, 2017). Marks has
argued that education policymakers and researchers have had an unwarranted focus on SES
given that genetic and cognitive differences, not SES, are the dominant causes of diversity of
student outcomes (Marks, 2017).
This article will respond to Marks and colleagues’ (Armor et al., 2018; Marks, 2010,
2015, 2017) critiques of the veracity of SEC effects in three sections. Firstly, we will address
their arguments that prior school compositional research has suffered from ecological
fallacies and lack of control of prior achievement. Secondly, we will consider Marks and
colleagues’ application of residualised change and fixed effects analyses that have found null
SEC effects. Finally, we will demonstrate that structural equation modeling (SEM) can be
5
used to address Marks and colleagues’ concerns about measurement error inflating SEC
effects in multilevel research.
Critiques of Prior School Compositional Research
Marks and colleagues (Armor et al., 2018; Marks, 2010, 2015, 2017) have critiqued
school compositional research for ecological fallacies and lack of control for prior
achievement but have rarely engaged with specific studies. The ecological fallacy is when a
relationship observed at the group level is mistakenly used to describe the effect of group
membership on individuals (Robinson, 1950). Marks and colleagues have only referred to
Hauser’s (1970) critique of cross-tabulation methods and White, Reynolds, Thomas and
Gitzlaff’s (1993) critique of aggregated measures in single-level regressions when arguing
that school compositional research suffers from the ecological fallacy. This ignores the large
body of research that has utilized multi-level modeling (MLM) techniques to find SEC
effects (Ewijk & Sleegers, 2010). MLM addresses the ecological fallacy through the
aggregation of individual-level parameters to the group level (Snijders & Bosker, 2012, p.
83). Thus, MLM avoids the ecological fallacy as compositional effects are not the
relationship between a group-level predictor and the dependent variable, but the relationship
between the difference of an aggregated group-level predictor and its associated individual-
level predictor, and the dependent variable (Raudenbush & Bryk, 2002, pp. 139-141).
Marks and colleagues (Armor et al., 2018; Marks 2015, 2017) have also argued that
many school compositional studies have overestimated SEC effects by not controlling for
prior achievement. This criticism has not considered the differing aims of longitudinal and
cross-sectional compositional research. Compositional research that controls for prior
achievement can estimate compositional effects over specific time periods in a school career.
For example, Rumberger and Palardy (2005) examined the effect of SEC on achievement
growth from grades 8 to 12 in US high schools. On the other hand, cross-sectional studies
