METHODOLOGICAL PAPER
A new criterion for assessing discriminant validity
in variance-based structural equation modeling
Jörg Henseler & Christian M. Ringle & Marko Sarstedt
Received: 18 March 2014 /Accepted: 29 July 2014 /Published online: 22 August 2014
#
The Author(s) 2014. This article is published with open access at Springerlink.com
Abstract Discriminant validity assessment has become a
generally accepted prerequisite for analyzing relationships
between latent variables. For variance-based structural equa-
tion modeling, such as partial least squares, the Fornell-
Larcker criterion and the examination of cross-loadings are
the dominant approaches for evaluating discriminant validity.
By means of a simulation study, we show that these ap-
proaches do not reliably detect the lack of discriminant valid-
ity in common research situations. We therefore propose an
alternative approach, based on the multitrait-multimethod ma-
trix, to assess discriminant validity: the heterotrait-monotrait
ratio of correlations. We demonstrate its superior performance
by means of a Monte Carlo simulation study, in which we
compare the new approach to the Fornell-Larcker criterion
and the assessment of (partial) cross-loadings. Finally, we
provide guidelines on how to handle discriminant validity
issues in variance-based structural equation modeling.
Keywords Structural equation modeling (SEM)
.
Partial least
squares (PLS)
.
Results evaluation
.
Measurement model
assessment
.
Discriminant validity
.
Fornell-Larcker criterion
.
Cross-loadings
.
Multitrait-multimethod (MTMM) matrix
.
Heterotrait-monotrait (HTMT) ratio of correlations
Introduction
Variance-based structural equation modeling (SEM) is
growing in popularity, which the plethora of recent devel-
opments and discussions (e.g., Henseler et al. 2014;
Hwang et al. 2010;Luetal.2011; Rigdon 2014;
Tenenhaus and Tenenhaus 2011), as well as its frequent
application across different disciplines, demonstrate (e.g.,
Hair et al. 2012a, b; Lee et al. 2011;PengandLai2012;
Ringle et al. 2012). Variance-based SEM methods—such
as partial least squares path modeling (PLS; Lohmöller
1989;Wold1982), generalized structured component
analysis (GSCA; Henseler 2012; Hwang and Takane
2004), regularized generalized canonical correlation anal-
ysis (Tene nhaus a nd Tenenhaus 2011), and best fitting
proper indices (Dijkstra and Henseler 2011)—have in
common that they employ linear composites of observed
variables as proxies for latent variables, in order to esti-
mate model relationships. The estimated strength of these
relationships, most notably between the latent variables,
can only be meaningfully interpreted if construct validity
was established (Peter and Churchill 1986). Thereby, re-
searchers ensure that the measurement models in their
studies capture what they intend to measure (Campbell
and Fiske 1959). Threats to construct validity stem from
various sou r ces . Con seq ue nt ly, r es ea rc he rs mus t emp loy
different construct validity subtypes to evaluate their re-
sults (e.g., convergent validi ty, discrimi nant validity, cri-
terion vali dity; Sarstedt and Mooi 2014).
J. Henseler
Faculty of Engineering Technology, University of Twente, Enschede,
Netherlands
e-mail: j.henseler@utwente.nl
J. Henseler
ISEGI, Universidade Nova de Lisboa, Lisbon, Portugal
C. M. Ringle
Hamburg University of Technology (TUHH), Hamburg, Germany
e-mail: c.ringle@tuhh.de
C. M. Ringle
University of Newcastle, Newcastle, Australia
M. Sarstedt
Otto-von-Guericke-University Magdeburg, Magdeburg, Germany
M. Sarstedt (*)
University of Newcastle, Newcastle, Australia
e-mail: marko.sarstedt@ovgu.de
J. of the Acad. Mark. Sci. (2015) 43:115–135
DOI 10.1007/s11747-014-0403-8
In this paper, we focus on examining discriminant validity
as one of the key building blocks of model evaluation
(e.g.,Bagozzi and Phillips 1982;Hairetal.2010).
Discriminant validity ens ures tha t a construct measure is
empirically unique and represents phenomena of interest that
other measures in a structural equation model do not capture
(Hair et al. 2010). Technically, discriminant validity requires
that “a test not correlate too highly with measures from which
it is supposed to differ” (Campbell 1960, p. 548). If discrim-
inant validity is not established, “constructs [have] an influ-
ence on the variation of more than just t he observed
variabl es to whi ch they are theo r et ica ll y related” and ,
as a consequence, “researchers cannot be certain results
confirming hypothesized structural paths are real or
whether they are a result of statistical discrepancies”
(Farrell 2010, p. 324). Against this background, discrim-
inant validity assessment has become common practice
in SEM studies (e.g., Shah and Goldstein 2006;Shook
et al. 2004).
Despite its obvious importance, researchers using variance-
based SEM usually rely on a very limited set of approaches to
establish discriminant validity. As shown in Table 1,tutorial
articles and i ntroductory books on PLS almost solely
recommend using the Fornell and Larcker (1981)criterion
and cross-loadings (Chin 1998). Reviews of PLS use suggest
that these recommendations have been widely applied in
published research in the fields of management informa-
tion systems (Ringle et al. 2012), marketing (Hair et al.
2012a), and strategic management (Hair et al. 2012b).
For example, the marketing studies in Hair et al.'s
(2012 a ) review that engage in some type of discriminant
validity assessment use the Fornell-Larcker criterion
(72.08%), c ross-loadings (7.79%), or both (26.13%).
Reviews in other di sciplines paint a similar monotonous
picture. Very few studies report other means of
assessing discriminant validity. These alternatives in-
clude testing whether the latent variable correlations
are signific antly differe nt fr om one another ( Milberg
et al. 2000) and running separate confirmatory factor
analyses prior to employing variance-based SEM
(Cording et al. 2008;Pavlouetal.2007;Ravichandran
and Rai 2000) by using, for example, Anderson and
Gerbing's (1988) test as the standard.
1
While marketing researchers routinely rely on the Fornell-
Larcker criterion and cross-loadings (Hair et al. 2012a), there
are very few empirical findings on the suitability of these
criteria for establishing discriminant validity. Recent research
suggests that the Fornell-Larcker criterion is not effective
under certain circumstances (Henseler et al. 2014;Rönkkö
and Evermann 2013), pointing to a potential weakness in the
most commonly used discriminant validity criterion.
However, these studies do not provide any systematic assess-
ment of the Fo rnell-Larcker criterion’s efficacy regarding
testing discriminant validity. Furthermore, while researchers
frequently note that cross-loadings are more liberal in terms of
indicating discriminant validity (i.e., the assessment of cross-
loadings will support discriminant validity when the Fornell-
Larcker criterion fails to do so; Hair et al. 2012a, b;Henseler
et al. 2009), prior research has not yet tested this notion.
In this research, we present three m ajor contributions to
variance-based SEM literature on marketing that are rele-
vant for the social sciences disciplines in general. First,
we show that neither the Fornell-L ar c ker criterion n or th e
assessment of the cross-loadings allows users of variance-
based SEM to determine the discriminant validity of their
measures. Second, as a solution for this critical issue, we
propose t he heterotrait-monotrait ratio of correlations
(HTMT) as a new approach to assess discriminant validity
in variance-based SEM. Third, we demonstrate the effica-
cy of HTMT by means of a Monte Carlo simulation, in
which we compa re its p erformance with that of the
Fornell-Larcker criterion and with the assessment of the
cross-loadings. B ased on our findings, we provide re-
searchers with recommendations on when an d how to
use the approach. Moreover, we offer guidelines for
treating discriminant validity problems. The findings of
this research are relevant for both researchers a nd practi-
tioners in marketi ng and other social sciences disciplines,
since we establish a new standard means of assessing
discriminant validity as part of measurement model eval-
uation in variance-based SEM.
Traditional discriminant validity assessment methods
Comparing average communality and shared variance
In their widely cited article on t ests to eva lua t e struc tu ral
equation models, Fornell and Larcker (1981)suggestthat
discriminant validity is established if a latent variable
accounts for more variance in its associated indicator
variables than it shares with other constructs in the same
model. To satisfy this requirem ent, each const ruct’sav-
erage variance extracted (AVE) must be compared with
its squared correlations with other constructs in the m od-
el. According to Gefen and Straub (2005,p.94),“[t]his
comparison harkens back to the tests of correlations in
multi-trait multi-method matrices [Campbell and Fiske,
1959], and, indeed, the logic is quite similar.”
The AVE represents the average amount of variance that a
construct explains in its indicator variables relative to the
1
It is important to note that studies may have used different ways to
assess discriminant validity assessment, but did not include these in the
main texts or appendices (e.g., due to page restrictions). We would like to
thank an anonymous reviewer for this remark.
116 J. of the Acad. Mark. Sci. (2015) 43:115–135
overall variance of its indicators. The AVE for construct ξ
j
is
defined as follows:
AVEξ
j
¼
X
K
j
k¼1
λ
2
jk
X
K
j
k¼1
λ
2
jk
þ Θ
jk
; ð1Þ
where λ
jk
is the indicator loading and Θ
jk
the error variance
of the k
th
indicator (k = 1,…,K
j
)ofconstructξ
j
. K
j
is the number
of indicators of construct ξ
j
. If all the indicators are standardized
(i.e.,haveameanof0andavarianceof1),Eq.1 simplifies to
AVEξ
j
¼
1
K
j
∑
K
j
k¼1
λ
2
jk
: ð2Þ
The AVE thus equals the average squared standardized
loading, and it is equivalent to the mean value of the indicator
reliabilities. Now, let r
ij
be the correlation coefficient between
the construct scores of constructs ξ
i
and ξ
j
The squared inter-
construct correlation r
ij
2
indicates the proportion of the vari-
ance that constructs ξ
i
and ξ
j
share. The Fornell-Larcker crite-
rion then indicates that discriminant validity is established if
the following condition holds:
AVEξ
j
> maxr
2
ij
∀i≠j: ð3Þ
Since it is common to report inter-construct correlations in
publications, a different notation can be found in most reports
on discriminant validity:
ffiffiffiffiffiffiffiffiffiffiffiffiffiffi
AVEξ
j
q
> maxjr
ij
j ∀i≠j: ð4Þ
From a c once pt ua l p er spec tive, the application of the
Fornell-Larcker criterion is not without limitations. For exam-
ple, it is well known that variance-based SEM methods tend to
overestimate indicator loadings (e.g., Hui and Wold 1982;
Lohmöller 1989). The origin of this characteristic lies in the
methods’ treatment of constructs. Variance-based SEM
methods, such as PLS or GSCA, use composites of indicator
variables as substitutes for the underlying constructs (Henseler
et al. 2014). The loading of each indicator on the composite
represents a relationship between the indicator and the com-
posite of which the indicator is part. As a result, the degree of
overlap between each indicator and composite will be high,
yielding inflated loading estimates, especially if the number of
indicators per construct (composite) is small (Aguirre-Urreta
et al. 2013).
2
Furthermore, each indicator’s error variance is
also included in the composite (e.g., Bollen and Lennox
1991), which increases the validity gap between the construct
and the composite (Rigdon 2014) and, ultimately, compounds
the inflation in the loading estimates. Similar to the loadings,
variance-based SEM methods generally underestimate struc-
tural model relationships (e.g., Reinartz et al. 2009;
Marcoulides, Chin, and Saunders 2012). While these devia-
tions are usually relatively small (i.e., less than 0.05; Reinartz
Table 1 Recommendations for
establishing discriminant validity
in prior research
Other prominent introductory
texts on PLS (e.g., Falk and Miller
1992; Haenlein and Kaplan 2004;
Lohmöller 1989; Tenenhaus et al.
2005; Wold 1982) do not offer
recommendations for assessing
discriminant validity
Reference Recommendation
Fornell-Larcker criterion Cross-loadings
Barclay, Higgins, and Thompson (1995) ✓✓
Chin (1998, 2010) ✓✓
Fornell and Cha (1994) ✓
Gefen and Straub (2005) ✓✓
Gefen, Straub, and Boudreau (2000) ✓✓
Götz, Liehr-Gobbers, and Krafft (2010) ✓
Hair et al. (2011) ✓✓
Hair et al. (2012a) ✓✓
Hair et al. (2012b) ✓✓
Hair et al. (2014) ✓✓
Henseler et al. (2009) ✓✓
Hulland (1999) ✓
Lee et al. (2011) ✓✓
Peng and Lai (2012) ✓
Ringle et al. (2012) ✓✓
Roldán and Sánchez-Franco (2012) ✓✓
Sosik et al. (2009) ✓
2
Nunnally (1978) offers an extreme example with five mutually uncor-
related indicators, implying zero loadings if all were measures of a
construct. However, each indicator’s correlation (i.e., loading) with an
unweighted composite of all five items is 0.45.
J. of the Acad. Mark. Sci. (2015) 43:115–135 117
et al. 2009), the interplay between inflated AVE values and
deflated structural model relationships in the assessment of
discriminant validity has not been systematically examined.
Furthermore, the Fornell-Larcker criterion does not rely on
inference statistics and, thus, no procedure for statistically
testing discriminant validity has been developed to date.
Assessing cross-loadings
Another popular approach for establishing discriminant validity is
the assessment of cross-loadings, which is also called “item-level
discriminant validity .” According to Gefen and Straub (2005,p.
92), “discriminant validity is shown when each measurement item
correlates weakly with all other constructs except for the one to
which it is theoretically associated.” This approach can be traced
back to exploratory factor analysis, where researchers routinely
examine indicator loading patterns to identify indicators that have
high loadings on the same factor and those that load highly on
multiple factors (i.e., double-loaders; Mulaik 2009).
In the case of PLS, Barclay et al. (1995), as well as Chin
(19 98), were the first to propose that each indicator loading
should be greater than all of its cross-loadings.
3
Otherwise, “the
measure in question is unable to discriminate as to whether it
belongs to the construct it was intended to measure or to another
(i.e., discriminant validity problem)” (Chin 2010,p.671).The
upper part a) of Fig. 1 illustrates this cross-loadings approach.
However, there has been no reflection on this approach’s
usefulness in variance-based SEM. Apart from the norm that
an item should be highly correlated with its own construct, but
have low correlations with other constructs in order to estab-
lish discriminant validity at the item level, no additional
theoretical arguments or empirical evidence of this approach’s
performance have been presented. In contrast, research on
covariance-based SEM has critically reflected on the
approach’s usefulness for discriminant validity assessment.
For example, Bollen (1989) shows that high inter-construct
correlations can cause a pronounced spurious correlation be-
tween a theoretically unrelated indicator and construct. The
paucity of research on the e fficacy of cross-loadings in
variance-based SEM is problematic, because the methods tend
to overestimate indicator loadings due to their reliance on
composites. At the same time, the introduction of composites
as substitutes for latent variables leaves cross-loadings largely
unaffected. The majority of variance-based SEM methods are
limited information approaches, estimating model equations
separately, so that the inflated loadings are only imperfectly
introduced in the cross-loadings. Therefore, the very nature of
algorithms, such as PLS, favors the support of discriminant
validity as described by Barclay et al. (1995) and Chin (1998).
Another major drawback of the aforementioned approach
is that it is a criterion, but not a statistical test. However, it is
also possible to conduct a statistical test of other constructs’
influence on an indicator using partial cross-loadings.
4
The
partial cross-loadings determine the effect of a construct on an
indicator other than the one the indicator is intended to mea-
sure after controlling for the influence of the construct that the
indicator should measure. Once the influence of the actual
construct has been partialed out, the residual error variance
should be pure random error according to the reflective mea-
surement model:
ε
jk
¼ x
jk
−λ
jk
ξ
j
; ε
jk
⊥ξ
i
∀i: ð5Þ
If ε
jk
is explained by another variable (i.e., the correlation
between the error term of an indicator and another construct is
significant), we can no longer maintain the assumption that ε
jk
is pure random error but must acknowledge that part of the
measurement error is systematic error. If this systematic error
is due to another construct ξ
i
, we must conclude that the
indicator does not indiscriminately measure its focal construct
ξ
j
, but also the other construct ξ
i
, which implies a lack of
discriminant validity. The lower part b) of Fig. 1 illustrates the
working principle of the significance test of partial cross-
loadings.
While this approach has not been applied in the context of
variance-based SEM, its use is common in covariance-based
SEM, where it is typically applied in the form of modification
indices. Substantial modification indices point analysts to the
correlations between indicator error terms and other con-
structs, which are nothing but partial correlations.
An initial assessment of traditional discriminant validity
methods
Although the Fornell-Larcker criterion was established more
than 30 years ago, there is virtually no systematic examination
of its efficacy for assessing discriminant validity. Rönkkö and
Evermann (2013) were the first to point out the Fornell-
Larcker criterion’s potential problems. Their simulation study,
which originally evaluated the performance of model valida-
tion indices in PLS, included a population model with two
identical constructs. Despite the lack of discriminant validity,
the Fornell-Larcker criterion indicated this problem in only 54
of the 500 cases (10.80%). This result implies that, in the vast
majority of situations that lack discriminant validity, empirical
3
Chin (2010) suggests examini ng the squared loadings and cross-
loadings instead of the loadings and cross-loadings. He argues that, for
instance, compared to a cross-loading of 0.70, a standardized loading of
0.80 may raise concerns, whereas the comparison of a shared variance of
0.64 with a shared variance of 0.49 puts matters into perspective.
4
We thank an anonymous reviewer for proposing this approach.
118 J. of the Acad. Mark. Sci. (2015) 43:115–135
researchers would mistakenly be led to believe that discrimi-
nant validity has been established. Unfortunately, Rönkkö and
Evermann’s(2013) study does not permit drawing definite
conclusions about extant approaches’ efficacy for assessing
discriminant validity for the following reasons: First, their
calculation of the AVE—a major ingredient of the Fornell-
Larcker criterion—was inaccurate, because they determined
one overall AVE value instead of two separate AVE values;
that is, one for each construct (Henseler et al. 2014).
5
Second,
Rönkkö and Evermann (2013) did not examine the perfor-
mance of the cross-loadings assessment.
In order to shed light on the Fornell-Larcker criterion’s
efficacy, as well as on that of the cross-loadings, we conducted
a small simulation study. We randomly created 10,000
datasets with 100 observations, each according to the one-
factor population model shown in Fig. 2, which Rönkkö and
Evermann (2013) and Henseler et al. (2014) also used. The
indicators have standardized loadings of 0.60, 0.70, and 0.80,
analogous to the loading patterns employed in previous sim-
ulation studies on variance-based SEM (e.g., Goodhue et al.
2012; Henseler and Sarstedt 2013; Reinartz et al. 2009).
To assess the performance of traditional methods regarding
detecting (a lack of) discriminant validity, we split the
construct in Fig. 2 into two separate constructs, which results
in a two-factor model as depicted in Fig. 3.Wethenusedthe
artificially generated datasets from the population model in
Fig. 2 to estimate the model shown in Fig. 3 by means of the
variance-based SEM techniques GSCA and PLS. We also
benchmarked their results against those of regressions with
summed scales, which is an alternative method for estimating
relationships between composites (Goodhue et al. 2012). No
matter which technique is used to estimate the model param-
eters, the Fornell-Larcker criterion and the assessment of the
cross-loadings should reveal that the one-factor model rather
than the two-factor model is preferable.
Table 2 shows the results of this initial study . The reported
percentage values denote the approaches’ sensitivity, indicating
their ability to identify a lack of discriminant validity (Macmillan
and Creelman 2004). For example, when using GSCA for
model estimation, the Fornell-Larcker criterion points to a lack
of discriminant validity in only 10.66% of the simulation runs.
The results of this study render the following main find-
ings: First, we can generally confirm Rönkkö and Evermann’s
(2013) report on the Fornell-Larcker criterion’s extremely
poor performance in PLS, even though our study’sconcrete
sensitivity value is somewhat higher (14.59% instead of
10.80%).
6
In additi on, we find that the sensitivity of the
5
We thank Mikko Rönkkö and Joerg Evermann for providing us with the
code of their simulation study (Rönkkö and Evermann 2013), which
helped us localize this error in their analysis.
6
The difference between these results could be due to calculation errors
by Rönkkö and Evermann (2013), as revealed by Henseler et al. (2014).
b)
a)
Fig. 1 Using the cross-loadings
to assess discriminant validity
J. of the Acad. Mark. Sci. (2015) 43:115–135 119
