University of Groningen
Influential cases in multilevel modeling. A methodological comment on Ruiter and De Graaf
(ASR 2006).
Meer, T. van der; Pelzer, B.; Grotenhuis, Manfred Te
Published in:
American Sociological Review
DOI:
10.1177/0003122409359166
IMPORTANT NOTE: You are advised to consult the publisher's version (publisher's PDF) if you wish to cite from
it. Please check the document version below.
Document Version
Publisher's PDF, also known as Version of record
Publication date:
2010
Link to publication in University of Groningen/UMCG research database
Citation for published version (APA):
Meer, T. V. D., Pelzer, B., & Grotenhuis, M. T. (2010). Influential cases in multilevel modeling. A
methodological comment on Ruiter and De Graaf (ASR 2006).
American Sociological Review
,
75
(1), 179-
184. https://doi.org/10.1177/0003122409359166
Copyright
Other than for strictly personal use, it is not permitted to download or to forward/distribute the text or part of it without the consent of the
author(s) and/or copyright holder(s), unless the work is under an open content license (like Creative Commons).
The publication may also be distributed here under the terms of Article 25fa of the Dutch Copyright Act, indicated by the “Taverne” license.
More information can be found on the University of Groningen website: https://www.rug.nl/library/open-access/self-archiving-pure/taverne-
amendment.
Take-down policy
If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately
and investigate your claim.
Downloaded from the University of Groningen/UMCG research database (Pure): http://www.rug.nl/research/portal. For technical reasons the
number of authors shown on this cover page is limited to 10 maximum.
http://asr.sagepub.com/
American Sociological Review
http://asr.sagepub.com/content/75/1/173
The online version of this article can be found at:
DOI: 10.1177/0003122409359166
2010 75: 173American Sociological Review
Tom Van der Meer, Manfred Te Grotenhuis and Ben Pelzer
Influential Cases in Multilevel Modeling : A Methodological Comment
Published by:
http://www.sagepublications.com
On behalf of:
American Sociological Association
can be found at:American Sociological ReviewAdditional services and information for
http://asr.sagepub.com/cgi/alertsEmail Alerts:
http://asr.sagepub.com/subscriptionsSubscriptions:
http://www.sagepub.com/journalsReprints.navReprints:
http://www.sagepub.com/journalsPermissions.navPermissions:
What is This?
- Mar 5, 2010Version of Record >>
at University of Groningen on July 6, 2012asr.sagepub.comDownloaded from
Influential Cases in
Multilevel Modeling: A
Methodological Comment
Tom Van der Meer,
a
Manfred Te Grotenhuis,
b
and
Ben Pelzer
b
A large number of cross-national survey data-
sets have become available in recent decades.
Consequently, scholars frequently apply mul-
tilevel models to test hypotheses on both the
individual and the country level. However,
no currently available cross-national survey
project covers more than 54 countries
(GESIS 2009). Multilevel modeling therefore
runs the risk that higher-level slope estimates
(and the substantial conclusions drawn from
these estimates) are unreliable due to one or
more influential cases (i.e., countries).
This comment emphasizes the problem of
influential cases and presents ways to detect
and deal with them. To detect influential
cases, one may use both graphic tools (e.g.,
scatter plots at the aggregate level) and
numeric tools (e.g., diagnostic tests such as
Cook’s D and DFBETAS). To illustrate the
usefulness and necessity of these tools, we
apply them to a study that was recently pub-
lished in this journal (Ruiter and De Graaf
2006). Finally, we provide recommendations
and tools to detect and handle influential
cases, specifically in cross-sectional multi-
level analyses.
A CROSS-NATIONAL STUDY
ON THE EFFECT OF RELIGION
In ‘‘National Context, Religiosity, and
Volunteering: Results from 53 Countries,’ ’
Ruiter and De Graaf (2006) raise the following
question: To what extent do national religious
contexts affect volunteering? One of their cen-
tral hypotheses states that volunteer rates will
be higher in devout countries than in secular
countries. This hypothesis originates from
two previous findings. First, religious citizens
are more likely than nonreligious citizens to
volunteer (Wilson and Musick 1997). Second,
in devout societies, citizens are more likely to
have active church members in their social net-
works (Kelley and De Graaf 1997). Because
pro-civic norms and recruitment are more
widespread, due to a higher share of religious
citizens in social networks, the authors expect
to find a positive effect of countries’ degree
of devoutness on individual volunteering for
both religious and nonreligious citizens.
To test this hypothesis, Ruiter and De
Graaf (2006) applied a hierarchical 3-level
model to three waves of the European/
World Values Survey (WVS): individuals at
level 1 (N 5 117,007), surveys from three
waves at level 2 (N 5 96), and countries at
level 3 (N 5 53). A crucial step in their
test of the network explanation is the inclu-
sion of a level-2 characteristic, namely coun-
try’s average church attendance rate. This
enabled them to test whether devout societies
induce their citizens to volunteer more often
a
University of Amsterdam/Netherlands Institute
for Social Research
b
Radboud University Nijmegen
Corresponding Author:
Tom Van der Meer, Department of Political
Science, University of Amsterdam, OZ
Achterburgwal 237, 1012 DL Amsterdam, The
Netherlands
E-mail: t.w.g.vandermeer@uva.nl
American Sociological Review
75(1) 173–178
Ó American Sociological
Association 2010
DOI: 10.1177/0003122409359166
http://asr.sagepub.com
Comment on Ruiter and De Graaf, ASR, April 2006
at University of Groningen on July 6, 2012asr.sagepub.comDownloaded from
than do citizens in secular societies. Ruiter
and De Graaf found average church atten-
dance to be significantly and positively
related to volunteering.
GRAPHIC EVIDENCE FROM A
SCATTER PLOT
To test whether their findings were robust,
Ruiter and De Graaf (2006) re-estimated
their model 96 times, leaving each survey
out once, and compared the resulting esti-
mates with those from the original model
(these differences are known as DFBETA).
Based on these comparisons, they found no
influential cases. However, this method will
most likely fail to detect a cluster of two or
more influential cases that have a similar
influence on the estimates. Furthermore,
because DFBETA lacks standardization, it
is hard to tell how large a difference should
be to call a case too influential.
Because Ruiter and De Graaf (2006) were
interested in the contextual effect of average
church attendance, we will look for potential
cases at level 2 that influence this effect in an
undesirable way. To get some general clues
about potential influential cases, we first
inspect the bivariate scatter plot for volunteer
rates and average church attendance for all
96 surveys (Belsley, Kuh, and Welsch
1980:8).
Figure 1 indicates a positive association
between average church attendance and vol-
unteering rates. However, it also reveals
a cluster of three potentially influential cases:
Tanzania, Zimbabwe, and Uganda. These
countries are very devout and show high vol-
unteering rates. Notably, they are three of the
four sub-Saharan countries in the dataset,
collected during the third survey-wave of
the WVS. Exclusion of one of these three
surveys does not affect the OLS regression
slope estimate substantially. Simultaneous
exclusion of Tanzania, Zimbabwe, and
Uganda, however, causes the slope estimate
to drop from .43 to .23.
NUMERIC EVIDENCE FROM A
MULTIVARIATE
HIERARCHICAL MODEL
Although the scatter plot is a good first indi-
cator of influential cases, it is based on
0
10
20
30
40
50
60
70
80
0 1020304050607080
Volunteers (percent)
Average Church Attendance (days per year)
Zimbabwe
Uganda
Tanzania
Figure 1. Scatter Plot for Average Church Attendance and Percentage Volunteers, in 96
Surveys Conducted in 53 Countries during Three Waves
174 American Sociological Review 75(1)
at University of Groningen on July 6, 2012asr.sagepub.comDownloaded from
aggregated data that lacks individual and
contextual control factors. The proof of the
pudding is in multilevel, multivariate models.
In Table 1, Model 0, the random slope
model reported is virtually equal to Ruiter
and De Graaf’s (2006) Table 3, Model 4
(p. 201).
1
Model 0 shows the positive and
significant effect of average church atten-
dance that they found. Next, we compute
two diagnostics to detect influential cases
for all 96 surveys at level 2: Cook’s D and
DFBETAS. Cook’s D measures the influence
of one single case on all (or a subset of)
level-2 estimates in the model, whereas
DFBETAS measures a case’s influence on
each of the level-2 estimates separately.
Cook’s D is defined as:
D
j
¼
1
r
ð
ˆ
b 2
ˆ
b
ð2jÞ
Þ#
ˆ
S
21
ð2jÞ
ð
ˆ
b 2
ˆ
b
ð2jÞ
Þð1Þ
where r 5 number of fixed parameters,
ˆ
b 5
vector of estimates based on the full sample,
ˆ
b
ð2jÞ
5 vector of estimates after unit j is
excluded, and
ˆ
S
ð2jÞ
denotes the covariance
matrix after unit j is excluded (Snijders and
Berkhof 2008:158 [3.24]). Cook’s D can be
interpreted as the standardized average
squared difference between the estimates
with and without unit j.
DFBETAS is defined as:
DFBETAS
jZ
¼
ˆ
b
Z2
ˆ
b
2jZ
seð
ˆ
b
2jZ
Þ
ð2Þ
where
ˆ
b
Z2
ˆ
b
2jZ
represents the difference
between the slope estimate
ˆ
b
Z
of predictor
Z based on the full sample and the estimate
ˆ
b
2jZ
after excluding unit j, and se ð
ˆ
b
2jZ
Þ de-
notes the standard error of
ˆ
b
2jZ
. Equation 2
is analogous to Belsley and colleagues
(1980:13). One can interpret DFBETAS as
the standardized difference between the esti-
mate with and without unit j.
2
To decide which cases are too influential,
Belsley and colleagues (1980:28) propose
using 4/n
x
as the cutoff value for Cook’s D,
and 2/On
x
for the absolute value of
DFBETAS (where n
x
5 number of units at
level x).
Table 1. The Effect of Average Church Attendance after Eliminating Influential Cases
Model 0
a
Model 1 Model 2 Model 3
To neutralize their influence at
the contextual level, the
model includes dummies for:
Tanzania Tanzania
Zimbabwe
Tanzania
Zimbabwe
Uganda
Effect of average church
attendance
.018 (.005)*** .014 (.005)*** .010 (.005)** .007 (.006)
Highest DFBETAS
b
.936 .560 .644 –.583
Survey with highest
DFBETAS
c
Tanzania Zimbabwe Uganda Russia
Corresponding Cook’s D
d
.306 .185 .217 .247
Note: Standard errors are in parentheses. Estimates for all variables in models are available at http://
www.ru.nl/mt/ic/downloads/.
a
Model 0 replicates Model 4 in Ruiter and De Graaf (2006).
b
All DFBETAS values exceed 2 /On
2
(i.e., 2 /O96 5 .204 for Model 0; 2 /O95 5 .205 for Model 1; 2 /O94 5
.206 for Model 2; 2 /O93 5 .207 for Model 3), where n
2
represents the number of surveys minus the
number of survey dummies.
c
The surveys from Tanzania, Zimbabwe, and Uganda are from Wave 3. The survey from Russia is from
Wave 2.
d
Cook’s D’s in models 0 to 2 exceed 4/n
2
(i.e., 4/96 5 .0417 for Model 0; 4/95 5 .0421 for Model 1; 4/94 5
.0425 for Model 2; 4/93 5 .043 for Model 3), where n
2
represents the number of surveys minus the
number of survey dummies.
**p\ .025; ***p \ .001 (one-tailed tests).
Van der Meer et al. 175
at University of Groningen on July 6, 2012asr.sagepub.comDownloaded from
