www.ssoar.info
Why You Should Always Include a Random Slope
for the Lower-Level Variable Involved in a Cross-
Level Interaction
Heisig, Jan Paul; Schaeffer, Merlin
Veröffentlichungsversion / Published Version
Zeitschriftenartikel / journal article
Zur Verfügung gestellt in Kooperation mit / provided in cooperation with:
Wissenschaftszentrum Berlin für Sozialforschung (WZB)
Empfohlene Zitierung / Suggested Citation:
Heisig, J. P., & Schaeffer, M. (2019). Why You Should Always Include a Random Slope for the Lower-Level Variable
Involved in a Cross-Level Interaction. European Sociological Review, 35(2), 258–279. https://doi.org/10.1093/esr/
jcy053
Nutzungsbedingungen:
Dieser Text wird unter einer CC BY Lizenz (Namensnennung) zur
Verfügung gestellt. Nähere Auskünfte zu den CC-Lizenzen finden
Sie hier:
https://creativecommons.org/licenses/by/4.0/deed.de
Terms of use:
This document is made available under a CC BY Licence
(Attribution). For more Information see:
https://creativecommons.org/licenses/by/4.0
Why You Should Always Include a Random
Slope for the Lower-Level Variable Involved in
a Cross-Level Interaction
Jan Paul Heisig
1,†
and Merlin Schaeffer
2,
*
,†
1
WZB Berlin Social Science Center, Reichpietschufer 50, 10785 Berlin, Germany and
2
University of
Copenhagen, Øster Farimagsgade 5, DK-1353 København, Denmark
*Corresponding author. Email: mesc@soc.ku.dk
†
Both authors have contributed equally.
Submitted January 2018; revised December 2018; accepted December 2018
Abstract
Mixed-effects multilevel models are often used to investigate cross-level interactions, a specific type
of context effect that may be understood as an upper-level variable moderating the association be-
tween a lower-level predictor and the outcome. We argue that multilevel models involving cross-level
interactions should always include random slopes on the lower-level components of those interac-
tions. Failure to do so will usually result in severely anti-conservative statistical inference. We illus-
trate the problem with extensive Monte Carlo simulations and examine its practical relevance by
studying 30 prototypical cross-level interactions with European Social Survey data for 28 countries. In
these empirical applications, introducing a random slope term reduces the absolute t-ratio of the
cross-level interaction term by 31 per cent or more in three quarters of cases, with an average reduc-
tion of 42 per cent. Many practitioners seem to be unaware of these issues. Roughly half of the cross-
level interaction estimates published in the European Sociological Review between 2011 and 2016 are
based on models that omit the crucial random slope term. Detailed analysis of the associated test sta-
tistics suggests that many of the estimates would not reach conventional thresholds for statistical
significance in correctly specified models that include the random slope. This raises the question
how much robust evidence of cross-level interactions sociology has actually produced over the
past decades.
Introduction
One of the enduring questions of sociology is how
human attitudes and behaviour are shaped by the social
environment and how vice versa the social environment
emerges from human action. The investigation of con-
text effects, where an environmental feature (e.g., a
characteristic of a neighbourhood or country) affects
processes at a lower level (e.g., that of the individual), is
therefore central to the discipline, and one should think
that sociologists are highly proficient in modelling them
statistically.
Quantitative sociologists typically use mixed-effects
models, which are also known as ‘hierarchical models’
or simply ‘multilevel models’, to deal with the statistical
V
C
The Author(s) 2019. Published by Oxford University Press. All rights reserved.
For permissions, please e-mail: journals.permissions@oup.com
European Sociological Review, 2019, Vol. 35, No. 2, 258–279
doi: 10.1093/esr/jcy053
Advance Access Publication Date: 2 February 2019
Original Article
Downloaded from https://academic.oup.com/esr/article-abstract/35/2/258/5306121 by GESIS - Leibniz-Institut für Sozialwissenschaften user on 10 October 2019
challenges that arise in the estimation of context effects
(see the ‘Mixed Effects Models with Cross-Level
Interactions’ section and Equations 1–4 below). A cru-
cial issue in the specification of these models is the
choice of a random-effects structure (i.e., random inter-
cept and slopes), which can have important consequen-
ces both for the precision of parameter estimates
(Heisig, Schaeffer and Giesecke, 2017) and for statistical
inference (Berkhof and Kampen, 2004; Barr et al., 2013;
Bryan and Jenkins, 2016; Schmidt-Catran and
Fairbrother, 2016; Bell, Fairbrother and Jones, 2018).
The random-effects structure is also a crucial issue in
the estimation of cross-level interactions, which are a spe-
cial type of context effect where a contextual characteristic
moderates the strength of a lower-level relationship (see
Equation 4 below). To fix ideas, consider the following ex-
ample, which also serves as one of the illustrative empirical
examples presented later on: The (individual-level) rela-
tionship between fear of crime (as the outcome) and educa-
tion (as the predictor) might be weaker in less developed
countries (as indicated by the human development index;
HDI) where the generally poor living conditions instil a
fear of crime into everyone. Or to put it another way, the
better-educated tend to benefit the most from improving
societal conditions, whereas the less educated continue to
live in fear of crime even in more developed societies.
Researchers who study cross-level interactions are
interested in variation of lower-level relationships across
contexts. One might therefore expect their models to in-
clude so-called random slope terms that capture unex-
plained contextual variation in these relationships (see
Equation 3 below for a formal representation). In our
example, one would include a random slope to account
for cross-country differences in the relationship between
education and fear of crime that are not explained by
country differences in human development.
A review of published research, however, reveals that
in many analyses of cross-level interactions the corre-
sponding random slope is missing. Between 2011 and
2016, the European Sociological Review (ESR)published
28 studies that investigated cross-level interactions using
(two-level) mixed-effects multilevel models (24 of these
studies were country comparisons). More than half of
these studies (17/28 or 61 per cent) only specified random
intercept models without any random slopes (for details,
see the ‘Cross-Level Interactions in the ESR’ section).
Given that empirical practice is so inconsistent, one
may wonder whether the inclusion of random slope
terms on the lower-level components of cross-level inter-
actions is a matter of taste or whether one approach will
usually be preferable to the other. A review of promin-
ent textbooks on multilevel modelling does not provide
a clear answer. In one widely read book, Snijders and
Bosker (2012) note that ‘tested fixed effects’ should be
accompanied by ‘an appropriate error term [...] For
cross-level interactions, it is the random slope of the
level-one [i.e., lower-level] variable involved in the inter-
action’ (p. 104). Other authors take a more ambiguous
position. For example, Raudenbush and Bryk’s (2002)
book includes a section on ‘A Model with Nonrandomly
Varying Slopes’ where they suggest that a model with a
cross-level interaction may omit the corresponding ran-
dom slope if ‘little or no variance in the slopes remains to
be explained’ (p. 28). They provide no precise definition
of ‘little or no variance’, however. In their chapter on
‘Random-coefficient models’, Rabe-Hesketh and
Skrondal (2012) generally include random slope terms
alongside cross-level interactions, but they also note that
the decision whether to do so often seems to be driven by
technicalities of the software used: ‘Papers using HLM
tend to include more cross-level interactions and more
random coefficients in the models (because the level-2
[i.e., upper-level] models look odd without residuals)
than papers using, for instance, Stata’ (p. 212f.). This cer-
tainly does not sound like an emphatic recommendation
to include the random slope for statistical reasons.
In this article, we argue that such a recommendation
should be given. We explain and demonstrate that the
omission of random slopes in the analysis of cross-level
interactions constitutes a specification error that will
often have severe consequences for statistical inference
about the coefficient of the cross-level interaction term
(i.e., in our running example, the interaction between
education and HDI) and about the main effect of the
lower-level predictor involved in the interaction (i.e.,
the main effect of education). Only the main effect of
the upper-level predictor remains unaffected (provided
that the model includes a random intercept, as is gener-
ally the case in applied research).
In the next section, we briefly introduce mixed-
effects models with cross-level interactions. In the ‘Why
Always a Random Slope?’ section, we then explain that
random slopes capture cluster-driven heteroskedasticity
and cluster-correlated errors. As in standard linear re-
gression, ignoring heteroskedasticity and within-cluster
error correlation by failing to specify the appropriate
random slope term will typically lead to downward bias
in standard error estimates.
The two subsequent sections present Monte Carlo
simulations and illustrative empirical analyses that sup-
port our claims. The simulations show that (correctly
specified) mixed-effects models with a random intercept
and a random slope on the lower-level component of the
cross-level interaction generally achieve accurate
European Sociological Review, 2019, Vol. 35, No. 2 259
Downloaded from https://academic.oup.com/esr/article-abstract/35/2/258/5306121 by GESIS - Leibniz-Institut für Sozialwissenschaften user on 10 October 2019
statistical inference for all coefficients of interest. By
contrast, random intercept models that omit the random
slope term produce severely anti-conservative inference
for the cross-level interaction term and the main effect of
its lower-level component. The proportion of 95 per cent
confidence intervals that do not cover the true effect size
(i.e., the actual coverage rate) is generally smaller than the
nominal rate, and often by a substantial margin. We find
that the extent of undercoverage increases with the extent
of variation in the (unmodelled) random slope, the vari-
ance of the lower-level component, and the number of
lower-level observations per cluster. Illustrative empirical
analyses of European Social Survey (ESS) data for 28
countries indicate that the consequences of omitting the
random slope on the lower-level component are severe in
real-life settings. We examine a total of 30 cross-level
interactions and find that inclusion of the random slope
term deflates the absolute t-ratio on the cross-level inter-
action term by 31 per cent or more in three quarters of
cases, with an average reduction of 42 per cent.
We then review studies of cross-level interactions pub-
lished in the ESR between 2011 and 2016.
Unsurprisingly, we find that authors were more likely to
report statistically significant cross-level interactions
when they used a misspecified model that omitted the cor-
responding random slope. Consistent with ‘P-hacking’
(Simonsohn, Nelson and Simmons, 2014), the distribu-
tion of absolute t-ratios for models estimated without a
random slope exhibits a marked peak just above the crit-
ical value of 1.96. In combination with the results of our
Monte Carlo simulations and empirical illustrations, our
review therefore suggests that many published estimates
based on models omitting the random slope would not
have reached conventional levels of statistical significance
in a correctly specified model.
The subsequent and penultimate section presents a
further result of our analysis: the omission of a relevant
random slope also leads to anti-conservative inference
for a corresponding ‘pure’ lower-level effect. That is,
even if the model does not contain any cross-level inter-
actions involving education, accurate inference for the
average effect of education on fear of crime across the
28 ESS countries would require the inclusion of a ran-
dom slope on education—provided that such a slope is
present in the process that gave rise to the data. While
this result is troubling, there are two reasons to be less
concerned than in the cross-level interaction case. First,
most sociologists who use multilevel models are primar-
ily interested in context effects rather than pure lower-
level effects, as we confirm through a systematic analysis
of the titles, abstracts, and formal hypotheses of research
published in the ESR. Second, pure lower-level effects
can typically be estimated with much greater precision
(and correspondingly higher absolute t-statistics) than
cross-level interactions. As a consequence, estimated
lower-level effects should often stay statistically highly
significant even if the associated t-ratio declines by 50
per cent or more. In the cross-level interaction case, such
a decrease will often mean the difference between mod-
erately strong and no statistically meaningful evidence
against the null hypothesis.
The concluding section discusses the primary impli-
cations of our study. Looking backward, our findings
suggest that the empirical basis for many seemingly
well-established findings in comparative research may
be much shakier than previously thought. Looking for-
ward, a minimum requirement for future studies that
examine cross-level interactions using multilevel models
is to include a random slope on the corresponding
lower-level variable. However, our findings suggest that
fully accurate statistical inference for all coefficients,
including pure lower-level effects, requires the inclusion
of additional random slopes or alternative methods of
inference, an important issue that should be addressed in
future work.
Mixed-Effects Models with Cross-Level
Interactions
In a first step, we briefly review the general logic of
mixed-effects models with cross-level interactions (for
comprehensive introductions, see, for example,
Raudenbush and Bryk, 2002; Rabe-Hesketh and
Skrondal, 2012; Snijders and Bosker, 2012). We begin
with the following lower-level equation for the (lower-
level) outcome Y
ij
(e.g., fear of crime):
Y
ij
¼ b
ðcÞ
j
þ b
ðxÞ
j
x
ij
þ
ij
; (1)
where i indexes lower-level observations (e.g., individu-
als) and j indexes upper-level observations or clusters
(e.g., countries). b
ðcÞ
j
is the constant (i.e., intercept) and
b
ðxÞ
j
is the coefficient of lower-level predictor x
ij
(e.g.,
education). The subscript j on the two parameters, b
ðcÞ
j
and b
ðxÞ
j
, indicates that both are considered as potential-
ly varying across clusters. In terms of our example, the j
on b
ðxÞ
j
thus means that the degree to which better-
educated people are less afraid of crime might vary
across countries. The model could be extended to in-
clude additional lower-level predictors x
2ij
to x
kij
, but
for our analysis, this is not necessary.
ij
is a lower-level
error often assumed to follow
ij
Nð0; r
2
Þ, that is, to
be normally distributed with a mean of zero and con-
stant variance r
2
(homoskedasticity).
260 European Sociological Review, 2019, Vol. 35, No. 2
Downloaded from https://academic.oup.com/esr/article-abstract/35/2/258/5306121 by GESIS - Leibniz-Institut für Sozialwissenschaften user on 10 October 2019
In a cross-level interaction model, b
ðxÞ
j
is specified as de-
pendent on at least one cluster-level (i.e., contextual) vari-
able z
j
(e.g., the HDI). Typically, the model will (and
should) also allow for a relationship between the constant
b
ðcÞ
j
and z
j
. One way to formalize this is to write b
ðcÞ
j
and
b
ðxÞ
j
as the outcome variables in two cluster-level equations:
b
ðcÞ
j
¼ c
ðcÞ
þ c
ðczÞ
z
j
þ u
ðcÞ
j
(2)
and
b
ðxÞ
j
¼ c
ðxÞ
þ c
ðxzÞ
z
j
þ u
ðxÞ
j
: (3)
Here, u
ðcÞ
j
and u
ðxÞ
j
are cluster-level error terms or
‘random effects’, with the former often referred to as a
‘random intercept’ and the latter as a ‘random slope’
term. It is natural to think of these terms as capturing
the effects of unmodelled cluster-level variables on b
ðcÞ
j
and b
ðxÞ
j
. Typically, u
ðcÞ
j
and u
ðxÞ
j
are assumed to follow a
multivariate normal distribution. Equation 2 is some-
times referred to as an ‘intercept-as-outcome’ equation
and Equation 3 as a ‘slope-as-outcome’ equation.
Equations 1–3 highlight the multilevel nature of the
model. An alternative formulation can be obtained by
substituting Equations 2 and 3 into Equation 1. After
rearranging terms we end up with:
Y
ij
¼ c
ðcÞ
þ c
ðczÞ
z
j
þ c
ðxÞ
x
ij
þ c
ðxzÞ
z
j
x
ij
|fflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl{zfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl}
fixed part
þ u
ðcÞ
j
þ u
ðxÞ
j
x
ij
þ
ij
|fflfflfflfflfflfflfflfflfflfflfflfflfflffl{zfflfflfflfflfflfflfflfflfflfflfflfflfflffl}
random part ð¼v
ij
Þ
:
(4)
Equation 4 shows why c
ðxzÞ
is referred to as a ‘cross-level
interaction effect’: it is the coefficient on a multiplicative
interaction term between the lower-level predictor x
ij
and the cluster-level predictor z
j
; in our running ex-
ample, it is the interaction between the individual char-
acteristic education and the country attribute HDI. The
first part of the right-hand expression, consisting of the
linear combination of the constant and the lower- and
upper-level predictors, multiplied by their respective
coefficients (or ‘fixed effects’), is also referred to as the
fixed part of the model. Crucially, the second part shows
that the model has a complex error term v
ij
that consists
of three components: the random intercept term u
ðcÞ
j
, the
lower-level residual error
ij
, and the product of the ran-
dom slope term with the lower-level predictor u
ðxÞ
j
x
ij
.
Why Always a Random Slope?
The formal exposition of the multilevel model in the pre-
vious section provides an intuitive reason why one
should always include the random slope term u
ðxÞ
j
:
Equation 3 clarifies that omitting u
ðxÞ
j
is equivalent to
assuming that b
ðxÞ
j
is perfectly determined by z
j
, in other
words that R
2
ðb
ðxÞ
j
Þ, the R
2
of the (implicit) cluster-level
regression for b
ðxÞ
j
, equals 1. As noted above,
Raudenbush and Bryk (2002) do indeed discuss the pos-
sibility that ‘little or no variance in the slopes remains to
be explained’ (p. 28) after accounting for the cluster-level
predictor z
j
. Yet we would argue that this is an unlikely
scenario in the vast majority of social science applica-
tions. This is confirmed by the empirical examples pre-
sented in the ‘Illustrative Empirical Analyses’ section and
in the Online Supplement (see, in particular, the final col-
umns of Online Supplement Tables D1–D6). More im-
portantly, our Monte Carlo simulations will show that
omitting the random slope term can have severe conse-
quences even when there is very little unexplained vari-
ation in b
ðxÞ
j
. We find that inference can be substantially
overoptimistic even when R
2
ðb
ðxÞ
j
Þ is as high as 0.95 or
when standard model selection criteria such as likelihood
ratio tests or information criteria indicate that the
remaining variation is negligible and favour the model
that drops the random slope (the results on model selec-
tion strategies can be found in Online Supplement C).
The two-stage formulation of the model in
Equations 1–3 also suggests that omission of u
ðxÞ
j
should
primarily affect inference about c
ðxÞ
and c
ðxzÞ
because
these terms are implicitly defined in the potentially mis-
specified Equation 3. Statistical inference for estimates of
c
ðczÞ
and c
ðcÞ
should remain unaffected—as it should for
any other terms that do not appear in Equation 3, includ-
ing the coefficients of additional lower-level predictors.
We now further clarify the importance of including
random slope terms on the lower-level components of
cross-level interactions. Equation 4 shows that the pres-
ence of the random slope term u
ðxÞ
j
in the true data-
generating process (DGP) adds the component u
ðxÞ
j
x
ij
to
the complex error term. This component has important
consequences for the conditional variance of the overall
error v
ij
and for the covariance of the error terms for
lower-level observations belonging to the same cluster.
In particular, the variance of v
ij
given x
ij
will be (Snijders
and Bosker, 2012, Equation 5.5):
1
Varðv
ij
jx
ij
Þ¼Varðu
ðcÞ
j
Þþ2Covðu
ðcÞ
j
; u
ðxÞ
j
Þx
ij
þ Varðu
ðxÞ
j
Þx
2
ij
þ Varð
ij
Þ: (5)
The covariance of the error terms for two different
individuals (say, i and i
0
) belonging to the same cluster
will be (Snijders and Bosker, 2012, Equation 5.6):
Covðv
ij
; v
i
0
j
jx
ij
; x
i
0
j
Þ¼Varðu
ðcÞ
j
Þ
þ Covðu
ðcÞ
j
; u
ðxÞ
j
Þðx
ij
þ x
i
0
j
Þ
þ Varðu
ðxÞ
j
Þx
ij
x
i
0
j
: (6)
European Sociological Review, 2019, Vol. 35, No. 2 261
Downloaded from https://academic.oup.com/esr/article-abstract/35/2/258/5306121 by GESIS - Leibniz-Institut für Sozialwissenschaften user on 10 October 2019
