Effect-Size Indices for Dichotomized Outcomes
in Meta-Analysis
Julio Sa´nchez-Meca and
Fulgencio Marı´n-Martı´nez
University of Murcia
Salvador Chaco´n-Moscoso
University of Seville
It is very common to find meta-analyses in which some of the studies compare 2
groups on continuous dependent variables and others compare groups on dichoto-
mized variables. Integrating all of them in a meta-analysis requires an effect-size
index in the same metric that can be applied to both types of outcomes. In this
article, the performance in terms of bias and sampling variance of 7 different
effect-size indices for estimating the population standardized mean difference from
a2×2table is examined by Monte Carlo simulation, assuming normal and
nonnormal distributions. The results show good performance for 2 indices, one
based on the probit transformation and the other based on the logistic distribution.
In the last 20 years, meta-analysis has become a
very popular and useful research methodology to in-
tegrate the results of a set of empirical studies about a
given topic. To carry out a meta-analysis an effect-
size index has to be selected to translate the results of
every study into a common metric.
When the focus of a study is to compare the per-
formance of two groups (e.g., treated vs. control, male
vs. female, trained vs. nontrained) on a continuous
dependent variable, the effect-size index most usually
applied is the standardized mean difference, d, de-
fined as the difference between the means of the two
groups divided by a within-group standard deviation
estimate. Substantial meta-analytic literature has been
devoted to showing the properties of this index, its
sampling variance, how to obtain confidence intervals
on the weighted mean effect size obtained from it, its
statistical significance, homogeneity tests, and how to
search for variables that moderate it (Cooper, 1998;
Cooper & Hedges, 1994; Hedges & Olkin, 1985; Lip-
sey & Wilson, 2001; Rosenthal, 1991).
However, in social and behavioral research it is
also very common to find studies using dichotomous
outcome measures or continuous variables that have
been dichotomized. Here we focus on dichotomized
variables, that is to say, variables that represent con-
structs continuous in nature, although being measured
as a dichotomy. This definition includes studies in
which the researchers apply a cutpoint, Y
c
, on a quan-
titative variable, as well as those in which they di-
rectly measure the dependent variable as a dichotomy
although there is an underlying continuous construct.
For many outcome measures, this is a reasonable as-
sumption. For example, studies about the effective-
ness of the treatment of tobacco addiction can mea-
sure the results as the number of cigarettes smoked in
a day or as a dichotomy (tobacco abstinence vs. non-
abstinence). In the field of delinquency treatment,
studies can record recidivism into crime as a di-
chotomy (recidivist vs. nonrecidivist) or, for example,
as the number of police contacts after prison release.
In education, the performance of students on an exam
can be measured continuously as the number of points
scored or dichotomously as passing versus failing the
exam.
Julio Sa´nchez-Meca and Fulgencio Marı´n-Martı´nez, De-
partment of Basic Psychology and Methodology, University
of Murcia, Murcia, Spain; Salvador Chaco´n-Moscoso, De-
partment of Psychology, University of Seville, Seville,
Spain.
This article was supported by a grant from the Ministerio
de Ciencia y Tecnologı´a and by funds from the Fondo Eu-
ropeo de Desarrollo Regional (FEDER; Project Number
BSO2001-0491). We gratefully acknowledge William R.
Shadish and Mark W. Lipsey for their helpful suggestions
on a first draft of this article. We also thank the referees for
comments that greatly improved the manuscript.
Correspondence concerning this article should be ad-
dressed to Julio Sa´nchez-Meca, Department of Basic Psy-
chology and Methodology, Faculty of Psychology, Campus
of Espinardo, University of Murcia, P.O. Box 4021, 30100
Murcia, Spain. E-mail: jsmeca@um.es
Psychological Methods Copyright 2003 by the American Psychological Association, Inc.
2003, Vol. 8, No. 4, 448–467 1082-989X/03/$12.00 DOI: 10.1037/1082-989X.8.4.448
448
In these cases, the study results can be summarized
asa2× 2 contingency table, where two groups are
crossed with the two outcomes, giving four possible
cell frequencies, as shown in Table 1. With n
E
and n
C
being the sample sizes, O
1E
and O
1C
being the success
frequencies, and p
E
⳱ O
1E
/n
E
and p
C
⳱ O
1C
/n
C
being
the success proportions in the experimental and con-
trol groups, respectively, different effect-size indices
have been proposed to represent the effect magnitude
(Fleiss, 1981, 1994; Laird & Mosteller, 1990; Lipsey
& Wilson, 2001; Rosenthal, 1994, 2000; Shadish &
Haddock, 1994). The first purpose of the present ar-
ticle is to explore the properties of these indices.
Three effect-size indices have been applied often:
(a) the risk difference, p
E
− p
C
, the raw difference
between the two success (or failure) proportions; (b)
the risk ratio, p
E
/p
C
, the ratio between the two pro-
portions, and (c) the odds ratio, p
E
(1 − p
C
)/p
C
(1 − p
E
),
the relative odds that one will be more successful than
the other. Of the three indices, the odds ratio is the
best one for most situations because of its good sta-
tistical properties (Fleiss, 1994; Haddock, Rindskopf,
& Shadish, 1998), although risk difference and risk
ratio can also be good alternatives under certain con-
ditions (Deeks & Altman, 2001; Hasselblad, Mo-
steller, et al., 1995; Sa´nchez-Meca & Marı´n-Martı´nez,
2000, 2001). In particular, these three indices have
been applied in meta-analyses in the health sciences,
because in this field it is very common to find re-
search issues in which the outcome is always mea-
sured as a dichotomous (or dichotomized) variable.
More commonly, some of the studies in a meta-
analysis present results comparing the performance of
two groups on continuous outcome variables, other
studies present them on dichotomized variables, and
some studies include both continuous and dichoto-
mized variables. In these cases, if some effect-size
indices are computed as d, and others as an odds ratio,
risk ratio, or risk difference, some means of convert-
ing all these diverse indices to a common effect size
is necessary to integrate all the results into a single
average effect size. Consequently, a second purpose
of the present article is to evaluate different methods
for converting different effect-size indices into the d
metric.
1
These cases are often handled in one of the follow-
ing ways. A dichotomous version of the standardized
mean difference that has been commonly applied con-
sists of calculating the difference between success (or
failure) proportions in experimental and control
groups and dividing it by an estimate of within-group
standard deviation (Fleiss, 1981); this index could be
named the standardized proportion difference, d
p
, and
in many cases it underestimates the population stan-
dardized mean difference (Fleiss, 1994; Haddock et
al., 1998).
Another strategy consists of computing the phi co-
efficient, , from each one of the 2 × 2 tables and
sometimes also transforming it into a standardized
mean difference, d
, by means of the typical r to d
translation formulas (e.g., Hedges & Olkin, 1985;
Rosenthal, 1991). In this case, phi coefficients also
underestimate the population correlation coefficient
and, therefore, d
indices would also underestimate
the meta-analytic results (Fleiss, 1994; Haddock et al.,
1998). Conversely, some meta-analysts transform ev-
1
Whitehead, Bailey, and Elbourne (1999) proposed an-
other strategy consisting of estimating the log odds ratio in
each study with continuous measures assuming normal (or
log-normal) distributions. The method requires a cutpoint,
Y
c
, for obtaining the standard normal (or log-normal) dis-
tribution function for experimental and control groups, p
E
and p
C
, in every study. Then, the log odds ratio estimates so
obtained can be quantitatively integrated with the log odds
ratios obtained from the dichotomous outcomes. This strat-
egy is especially useful when all of the studies with con-
tinuous outcomes included in the meta-analysis have used
the same scale, allowing the same cutpoint to be applied in
all of the studies. However, when the different studies have
used different measures and scales, it would be difficult
(and arbitrary) to define the cutpoints needed for estimating
the log odds ratios. Another problem in this strategy is the
loss of information produced in the process of dichotomiz-
ing variables (Dominici & Parmigiani, 2000). Therefore,
provided that meta-analyses in educational and behavioral
sciences routinely include different measures and scales of
the same construct, this strategy is not practical.
Table 1
Contingency 2 × 2 Table for Two Groups and a
Dichotomized Outcome
Outcome
Group
TotalExperimental Control
Success (Y
i
ⱖ Y
c
) O
lE
O
1C
m
l
Failure (Y
i
< Y
c
) O
2E
O
2C
m
2
Total n
E
n
C
N
Note. Y
i
⳱ the continuous outcome variable; Y
c
⳱ the cutpoint
applied for dichotomizing the dependent variable, Y; O
1E
and
O
1C
⳱ the success frequencies of the experimental and control
groups, respectively; O
2E
and O
2C
⳱ the failure frequencies of the
experimental and control groups, respectively; m
1
⳱ O
1E
+ O
1C
;
m
2
⳱ O
2E
+ O
2C
; n
E
and n
C
⳱ the sample sizes for experimental
and control groups, respectively. N ⳱ n
E
+ n
C
.
SPECIAL SECTION: EFFECT-SIZE INDICES 449
ery standardized mean difference, d, obtained from
studies with continuous outcome variables into the
Pearson correlation coefficient, r, and then integrate
them with phi coefficients obtained from studies with
2 × 2 tables. This will also underestimate the popu-
lation correlation coefficient.
Only recently have the problems of the phi coeffi-
cient and standardized-proportion-difference statistics
applied to 2 × 2 tables been discussed in the social and
behavioral sciences (Fleiss, 1994; Haddock et al.,
1998; Lipsey & Wilson, 2001). Currently, several al-
ternative strategies better than those involving d
p
and
d
indices can be applied to transform different effect-
size indices into the d metric. One of the strategies is
the correction of the attenuation in the d
p
and d
in-
dices that arises because of the dichotomization of the
underlying continuous variable (Becker & Thorndike,
1988; Hunter & Schmidt, 1990; Lipsey & Wilson,
2001). Other strategies are based on the assumption of
the logistic distribution or on the arcsine transforma-
tion. In this article we compare the performance of
seven translation formulas that put a variety of indices
on the d scale.
A third question addressed in this article is that of
the sampling variance of the effect-size indices. In
meta-analysis, such sampling variances are important
because the meta-analytic statistical models usually
weight each effect size by its inverse variance. When
the dependent variable is dichotomized, a loss of in-
formation is produced that affects the accuracy of the
effect size. The formulas derived from statistical
theory for estimating the sampling variance of each
effect-size index have to reflect the cost of dichoto-
mization in terms of accuracy. Therefore, it is ex-
pected that the sampling variances of the transformed
indices will be larger than that of the standardized
mean difference, d.
In summary, then, this article uses Monte Carlo
simulation to examine the performance of seven dif-
ferent strategies for obtaining a d index when the
dependent variable has been dichotomized. On the
basis of past research, the normal distribution is the
most usual assumption for data in the empirical stud-
ies; so we compared the bias and the sampling vari-
ance of the different effect-size indices assuming that
the two populations are normally distributed. To
check the robustness of these effect estimators, we
have also included several conditions representing
nonnormal distributions. Several factors in the simu-
lation were manipulated: the population standardized
mean difference, ␦, the value of the cutpoint, Y
c
,to
dichotomize the distributions, the sample size, the im-
balance between sample sizes, and the relationship
between sample sizes and success proportions. It was
expected that d
p
and d
would underestimate the
population standardized mean difference, that the re-
maining indices would offer a better performance, and
that indices based on normal distributions would pre-
sent the best results when normality is assumed. In
any case, the influence of varying all of these factors
on the performance of these effect-size indices is a
question that has not been yet studied.
Effect-Size Indices for Summarizing
2 × 2 Tables
We assume that the population contains two con-
tinuous distributions (those of experimental and con-
trol groups) with
E
and
C
as experimental and con-
trol population means, respectively, with being the
common population standard deviation. Thus, the
parametric effect size between experimental and con-
trol groups is defined as the standardized mean dif-
ference, ␦, and is computed as
␦ =
E
−
C
(1)
(Hedges & Olkin, 1985, Equation 2, p. 76).
We assume that the continuous variable is normally
distributed for experimental and control groups [Y
iE
∼
N(
E
,
2
); Y
iC
∼ N(
C
,
2
)]. When in a single study
the outcome variable has been measured continu-
ously, the parametric effect size, ␦, can be estimated
by means of the sample standardized mean difference,
g, computed by
g =
y
E
− y
C
S
(2)
(Hedges & Olkin, 1985, Equation 3, p. 78), with y
E
and y
C
being the sample means of the experimental
and control groups, and S being a pooled estimate of
the within-group standard deviation, given by
S =
冑
共
n
E
− 1
兲
S
E
2
+
共
n
C
− 1
兲
S
C
2
n
E
+ n
C
− 2
(3)
(Hedges & Olkin, 1985, p. 79), with S
2
E
and S
2
C
being
the sample variances of the experimental and control
groups, respectively. To correct the positive bias of
SA
´
NCHEZ-MECA, MARI
´
N-MARTI
´
NEZ, AND CHACO
´
N-MOSCOSO450
the standardized mean difference for small sample
sizes, the correction factor proposed in Hedges and
Olkin (1985, Equation 10, p. 81) would be applied to
the g index to obtain an unbiased estimate, d, of ␦:
d = c
共
m
兲
g, (4)
where c(m) is the correction factor and is obtained by
c
共
m
兲
= 1 −
3
4m − 1
, (5)
with m = n
E
+ n
C
− 2. The sampling variance of the d
index is estimated by
S
d
2
=
n
E
+ n
C
n
E
n
C
+
d
2
2
共
n
E
+ n
C
兲
(6)
(Hedges & Olkin, Equation 15, p. 86).
In the case of studies with 2 × 2 tables, it is
supposed that the continuous outcome variable has
been dichotomized by applying some cutpoint, Y
c
,to
the two original continuous populations, to classify
the subjects of the two populations into success or
failure categories. Two extensions of the standardized
mean difference d have been applied to studies with
such 2 × 2 tables: the standardized proportion differ-
ences, d
p
, obtained from success (or failure) propor-
tions, and the phi coefficient. The former is defined as
the difference between success (or failure) propor-
tions in experimental and control groups (p
E
and p
C
,
respectively), divided by an estimate of the within-
group standard deviation, S⬘. This index, d
p
, is com-
puted as
d
p
=
p
E
− p
C
S⬘
(7)
(Johnson, 1989, p. 150; Johnson & Eagly, 2000, p.
511), where S⬘ is given as
S⬘ =
冑
共
n
E
− 1
兲
p
E
共
1 − p
E
兲
+
共
n
C
− 1
兲
p
C
共
1 − p
C
兲
n
E
+ n
C
− 2
(8)
As Haddock et al. (1998) stated, and as shown in
our simulation study, d
p
underestimates the effect in
the population, ␦, whenever the marginals are not pro-
portional. The usual estimate of its sampling variance,
S
d
p
2
, is obtained by
S
d
p
2
=
n
E
+ n
C
n
E
n
C
+
d
p
2
2
共
n
E
+ n
C
兲
. (9)
The phi coefficient is the other effect-size index
usually applied to 2 × 2 tables. It is computed as
=
O
1E
O
2C
− O
2E
O
1C
公
n
E
n
C
m
1
m
2
(10)
(e.g., Fleiss, 1994, Equation 17-11, p. 249), with all of
the terms in the equation defined in Table 1. Phi can
be translated to the metric of the standardized mean
difference, d
,by
d
=
公
1 −
2
冑
df
共
n
E
+ n
C
兲
n
E
n
C
(11)
(Rosenthal, 1994, Equation 16-29, p. 239), with df ⳱
n
E
+ n
C
− 2.
The phi coefficient and, consequently, d
underes-
timate the parametric effect size (Haddock et al.,
1998). The sampling variance of d
can be ap-
proached by applying the delta method (Stuart & Ord,
1994, p. 350) to Equation 11:
S
d
2
=
n
E
+ n
C
n
E
n
C
共
1 −
2
兲
2
. (12)
Cohen (1988, p. 181) proposed the arcsine transla-
tion of the success proportion in a 2 × 2 table, p
E
and
p
C
, for obtaining an effect size in the d metric:
d
asin
= 2arcsine
公
p
E
− 2arcsine
公
p
C
. (13)
Following Lipsey and Wilson (2001, p. 56), we
expected that d
asin
would underestimate the popula-
tion effect size, unless the population distributions
were very skew. The sampling variance of d
asin
is
approximated by
S
d
asin
2
=
1
n
E
+
1
n
C
. (14)
(Rosenthal, 1994, p. 238).
A fourth index is based on the odds ratio. Assuming
logistic distributions and homogeneous variances,
Hasselblad and Hedges (1995, Equation 5, p. 170; see
also Chinn, 2000) proposed transforming the log odds
ratio into d by
d
HH
= L
OR
公
3
, (15)
SPECIAL SECTION: EFFECT-SIZE INDICES 451
where ⳱ 3.14159, L
OR
is the natural logarithm of
the odds ratio (OR); the odds ratio is easily obtained
from the 2 × 2 table by
OR =
p
E
共
1 − p
C
兲
p
C
共
1 − p
E
兲
(16)
(e.g., Shadish & Haddock, 1994, Equation 18-11, p.
269), with the precaution of adding 0.5 to all cell
frequencies when any of them is 0. Under these as-
sumptions, the log odds ratio is just the constant /
√
3
⳱ 1.81 multiplied by the standardized mean differ-
ence, d. Therefore, the d
HH
index is obtained dividing
the log odds ratio by the constant 1.81, which is also
the standard deviation of the logistic distribution. Ap-
plying the d
HH
index under the normal distribution
assumption probably will slightly underestimate the
population standardized mean difference, ␦.
If each of the two continuous populations follows a
logistic distribution with equal variances, the d
HH
in-
dex is independent of the cutpoint Y
c
and is normally
distributed, provided that O
1E
, O
1C
, O
2E
, and O
2C
are
not too small. Applying the delta method we can ap-
proximate the sampling variance of d
HH
by
S
d
HH
2
=
3
2
冋
1
O
1E
+
1
O
2E
+
1
O
1C
+
1
O
2C
册
(17)
(Hasselblad & Hedges, 1995).
An effect size similar to d
HH
was proposed by Cox
(1970) and cited in Haddock et al. (1998). It consists
of dividing the log odds ratio by the constant 1.65:
d
Cox
= L
OR
Ⲑ
1.65, (18)
and its sampling variance is estimated as
S
d
Cox
2
= 0.367
冋
1
O
1E
+
1
O
2E
+
1
O
1C
+
1
O
2C
册
. (19)
However, most primary studies assume a normal
distribution in the underlying populations, and under
these conditions, the performance of the preceding
effect-size indices based on the logit transformation
can differ. Two effect-size indices based on the nor-
mal distribution assumption are the probit transfor-
mation and the biserial-phi coefficient. Glass, Mc-
Gaw, and Smith (1981) proposed the probit
transformation to obtain an effect-size index in the d
metric. Let p
E
and p
C
be the success proportions in
experimental and control groups, respectively, ob-
tained from the 2 × 2 table in a given study. The probit
transformation, d
Probit
, is obtained by
d
Probit
=
共
z
E
− z
C
兲
(20)
(Glass et al., 1981, p. 138), z
E
and z
C
being the inverse
of the standard normal distribution function for p
E
and
p
C
, respectively [z
E
⳱ ⌽
−1
(p
E
); z
C
⳱ ⌽
−1
(p
C
)]. As-
suming normal distributions, d
Probit
will be an unbi-
ased estimator of the population standardized mean
difference. The sampling variance of d
Probit
is esti-
mated as
S
d
Probit
2
=
冋
2p
E
共
1 − p
E
兲
e
z
E
2
n
E
+
2p
C
共
1 − p
C
兲
e
z
C
2
n
C
册
(21)
(Rosenthal, 1994, p. 238).
Another option that assumes normal distributions to
summarize the results of a 2 × 2 table consists of
calculating the biserial-phi correlation coefficient,
bis
, and translating it into the d index by a typical r
to d translation formula (e.g., Hedges & Olkin, 1985;
Rosenthal, 1991). This correlation coefficient was
proposed by Thorndike (1949, 1982; see also Becker
& Thorndike, 1988) in the field of psychometrics. The
biserial-phi correlation is an unbiased estimate of the
point-biserial correlation when the continuous vari-
able has been dichotomized. It can be also defined as
a correction of the underestimation produced by the
phi coefficient, this correction being the same as that
proposed in Hunter and Schmidt (1990; see also Lip-
sey & Wilson, 2001, p. 111). The estimator is created
by multiplying the phi coefficient by the same multi-
plier used in creating the biserial. Therefore, the bi-
serial-phi coefficient can be obtained from the 2 × 2
table by
bis
=
公
p⬘q⬘
y⬘
, (22)
Becker & Thorndike, 1988, p. 525), where p⬘ is the
global success proportion in the 2 × 2 table p⬘⳱(O
1E
+ O
1C
)/N; y⬘ is the probability density function of the
standard normal distribution corresponding to p⬘; and
q⬘⳱1 − p⬘. Following the same strategy used with
the phi coefficient,
bis
is translated to a d index, d
bis
,
by means of
d
bis
=
bis
公
1 −
bis
2
冑
df
共
n
E
+ n
C
兲
n
E
n
C
(23)
SA
´
NCHEZ-MECA, MARI
´
N-MARTI
´
NEZ, AND CHACO
´
N-MOSCOSO452
