1
Hierarchically
Nested
Covariance
Structure
Models
for
Multitrait-Multimethod
Data
Keith F.
Widaman
University of
California at
Riverside
A
taxonomy
of
covariance
structure
models
for
rep-
resenting
multitrait-multimethod
data
is
presented.
Us-
ing
this
taxonomy,
it
is
possible
to
formulate
alternate
series
of
hierarchically
ordered,
or
nested,
models
for
such
data.
By
specifying
hierarchically
nested
models,
significance
tests
of
differences
between
competing
models
are
available.
Within
the
proposed
framework,
specific
model
comparisons
may
be
formulated
to
test
the
significance
of
the
convergent
and
the
discriminant
validity
shown
by
a
set
of
measures
as
well
as
the
ex-
tent
of
method
variance.
Application
of
the
proposed
framework
to
three
multitrait-multimethod
matrices
al-
lowed
resolution
of
contradictory
conclusions
drawn
in
previously
published
work,
demonstrating
the
utility
of
the
present
approach.
Investigating
the
construct
validity
of
psycho-
logical
measures
is
an
involved
process,
requiring
the
collation
of
evidence
from
a
variety
of
types
of
studies.
In
the
article
that
standardized
the
use
of
the
term &dquo;construct
validity,&dquo;
Cronbach
and
Meehl
(1955)
enumerated
types
of
research
that
would
contribute
to
the
construct
validity
of
a
mea-
sure.
One
of
these
types
of
study
involves
verifying
that
a
measure
correlates
highly with
other
vari-
ables
that
purportedly
measure
the
same
construct.
To
the
extent
that
such
correlations
are
found,
mea-
sures
display
convergent
validity.
Perhaps the
most
elegant
form
of such
study
utilizes
the
multitrait-
multimethod
matrix
proposed
by
Campbell and
Fiske
(1959),
in
which
correlations
are
computed
among
measures
of
two
or
more
traits
gathered
using
two
or
more
methods
of
measurement.
Use
of
the
mul-
titrait-multimethod
matrix
leads
to
an
important
cri-
terion,
beyond
those
stated
by
Cronbach
and
R4eehl
(1955),
for
establishing
the
construct
validity
of
a
measure:
In
addition
to
correlating
highly
with
other
measures
of
the
same
construct,
a
measure
should
fail
to
correlate
as
highly
with
measures
of
differ-
ent,
distinct
constructs.
To
the
extent
the
latter
pattern
of
correlations
is
found, the
measures
dis-
play
discriminant
validity.
Campbell
and
Fiske
(1959)
suggested
several
types
of
comparison
of
correlations
in
a
multitrait-
multimethod
matrix
in
order
to
demonstrate
the
degree
of
convergent
and
discriminant
validity of
the
measured
variables.
These
comparisons
essen-
tially
involve
determining
the
proportion of
times
certain
convergent
and
discriminant
validity
cri-
teria
are
satisfied;
the
higher
the
proportion
of
times
a
given
criterion
is
satished9 the
clearer
and
stronger
the
pattern of
convergent
or
discriminant
validity.
The
comparison
procedures
proposed
by
Camp-
bell
and
Fiske
(1959),
though
rather
straightfor-
ward
to
follow, do
have
a
number
of
shortcomings,
as
noted
by several
authors
(e.g.,
Althauser &
He-
berlein,
1970; Bagozzi,
1978;
Kenny,
1976).
First,
9
the
four
Campbell
and
Fiske
criteria
involve
simply
Downloaded from the Digital Conservancy at the University of Minnesota, http://purl.umn.edu/93227.
May be reproduced with no cost by students and faculty for academic use. Non-academic reproduction
requires payment of royalties through the Copyright Clearance Center,
http://www.copyright.com/
2
comparing
the
magnitudes
of
specified
correlations
in
a
situation
in
which
testing
the
statistical
sig-
nificance
of
the
overall
pattern
is
of
questionable
value
and
appropriateness,
due
to
the
lack
of
in-
dependence
of
correlations
used
in
the
various
comparisons.
Although there
have been
recent
at-
tempts to
develop
significance
tests
for
hypotheses
regarding
differences
in
level
of
correlations
that
reflect
the
Campbell
and
Fiske
criteria
(e.g.,
Hu-
bert &
Baker,
1978,
1979),
developments
to
date
have
been
rather
limited.
For
example,
the
statis-
tical
tests
derived
by
Hubert and
Baker
(1978)
are
not
sensitive
to
the
particular
pattern
of
correlations
among
traits,
though
Hubert
and
Baker
mentioned
that
unspecified
extensions
to
their
procedures
could
be
made
that
might
be
sensitive
to
differential
pat-
teg°rbs
of
trait
relationships.
A
second
problem
with
the
Campbell
and
Fiske
(1959)
criteria
is
that
precise
estimates
of
the
amounts
of
trait-related
and
method-related
variance
for
each
measure
are
not
obtainable,
though
such
estimates
would
be
very
useful
for
indicating
which
measures
should
be
refined
for
use
in
further
research.
A
third
problem
is
that
the
Campbell
and
Fiske
cri-
teria
are
evaluated
on
the
observed
correlations
among
measures,
and
differences
among
variables
in
their
level
of
reliability
will
distort
both
the
cor-
relations
among
measures
and
any
summary
mea-
sures
derived
from
the
correlations.
A
number
of
researchers
have
noted
the
short-
comings
of
and
ambiguities
associated
with
the
Campbell
and
Fiske
(1959)
criteria,
and
have
pro-
posed
alternative
approaches
to
evaluating
multi-
trait-multimethod
data.
Some
developed
one-step
(Jackson,
1969,
1971;
but
see
Conger,
1971)
or
two-step
(Golding &
Seidman,
1974;
Jackson,
1975;
but
see
Golding,
1977,
Jackson,
1977)
component
analytic
approaches
to
identifying
trait
and
method
factors.
Others
have
applied
structural
equation
modeling
procedures
to
multitrait-multimethod
data.
Of
the
latter,
some
proposed
procedures
have
been
path
analytic
in
nature
or
conception
(Althauser,
1974;
Althauser
&
Heberlein,
1970;
Althauser,
He-
berlein,
&
Scott,
1971;
Alevin, 1974;
Werts &
Linn,
1970),
others
have been
based
on
confirmatory
fac-
tor
analysis
(Bagozzi,
1978,
1980;
Boruch &
Wol-
ins,
1970;
Joreskog,
1971,
1974;
Kalleberg &
Kluegel,
1975;
Kenny,
1976;
Lee,
1980;
Marsh
&
Hocevar,
1983;
Schmitt,
1978;
Schmitt,
Coyle, &
Saari,
1977;
Schmitt
&
Saari,
1978;
Schwarzer,
1982),
while
still
others
employed
direct
product
models
(Bentler &
Lee,
1979;
Browne,
1984).
Although
many
contributions
have
been
made
to
the
modeling
of
multitrait-multimethod
data, a
unified
and
comprehensive
strategy
for
testing
structural
models
for
such
data
has
never
been
de-
veloped. The
current
state
of
the
literature
is
per-
haps best
exemplified
by Schmitt
(1978)
and
Ba-
gozzi
(1978),
the
most
extensive
presentations
in
the
literature
of
structural
equation
modeling
of
multitrait-multimethod
data.
Schmitt
and
Bagozzi
each
analyzed
the
same
two
multitrait-multimethod
matrices,
which
came
from
studies
published
by
Ostrom
(1969)
and
Kothandapani
(1971).
Al-
though they
analyzed
the
same
data
and
used
very
similar
confirmatory
factor
analytic
approaches,
Schmitt
and
Bagozzi
arrived
at
contradictory
con-
clusions
regarding
the
two
multitrait-multimethod
matrices:
Schmitt
stated
that
the
Kothandapani
ma-
trix
evidenced a
greater
degree
of
convergent
and
discriminant
validity
than
did
the
Ostrom
matrix,
whereas
Bagozzi
claimed
that
the
Ostrom
matrix
was
the
one
showing
the
clearer,
stronger
pattern
of
convergent
and
discriminant
validity.
These
con-
tradictory
conclusions,
coupled
with
the
presence
of a
number
of
calculational
errors
in
Schmitt
(1978)
and
several
invalid
interpretations
of
model
com-
parisons
by
Bagozzi
(1978),
prompted
the
present
attempt
to
standardize
the
fitting
of
structural
models
to
multitrait-multimethod
data.
The
aim
of
the
present
paper
is
to
describe a
general
procedure
for
specifying
and
testing a
hi-
erarchically
nested
set
of
models
for
multitrait-mul-
timethod
data.
Within
the
proposed
approach,
it
is
a
simple
matter
to
formulate
and
interpret
tests
of
the
degree
of
convergent
and
of
discriminant
va-
lidity shown
by a
set
of
measures,
as
well
as
to
estimate
the
amount
of
method
variance
in
the
mea-
sures.
Applying
the
proposed
approach
to
the
Os-
trom
(1969)
and
Kothandapani
(1971)
multitrait-
multimethod
matrices
will
demonstrate
both
how
the
present
approach
is
implemented
and
the
utility
Downloaded from the Digital Conservancy at the University of Minnesota, http://purl.umn.edu/93227.
May be reproduced with no cost by students and faculty for academic use. Non-academic reproduction
requires payment of royalties through the Copyright Clearance Center,
http://www.copyright.com/
3
of
the
proposed
approach
over
those
previously
presented.
In
so
doing, a
variety
of
misinterpre-
tations
by
Bagozzi
(1978)
are
resolved.
Method
Confirmatory
Factor
Model
For
34Ultitrait-fi4Ultimeth0d
Data’
Assume
that
each
person
in a
sample
of
IV
sub-
jects
has a
score
on
each
of
mt
measures
that
rep-
resent
t
trait
constructs
assessed
under
each
of
m
methods.
The
correlation
matrix Y,
of
the
met
ob-
served
variables
may
be
expressed,
in
the
standard
factor
analytic
decomposition
into
r
common
fac-
tors,
as
where A
is
an
met --
r
matrix
of
factor
loadings,
~
is
an
~°
><
r
matrix
of
correlations
among
factors,
and
~ is
an
mt
X mt
diagonal
matrix
of
unique
factor
variances.
If
the
t
trait
measures
are
arranged
within
meth-
ods;
the
factor
analytic
decomposition
of I
rep-
resented
by
Equation
l may
be
more
easily
adapted
to
fitting
multitrait-multimethod
data
if
A
is
par-
titioned
in
the
following
way:
where
Ay-
is
an
mt x
t submatrix
of
A
that
con-
tains
loadings
of
observed
variables
on
the
t
trait
factors,
is
an
mt
x
m
submatrix
of A
that
con-
tains
method
factor
loadings,
9
the
T,
are
t X
t
diagonal
submatrices
of
AT
containing
trait
factor
loadings
for
the
t
measures
gathered
using
method
a,
and
the
M;
are
column
vectors
within
Am
contain-
ing
t
loadings
on
method
factor
i.
It
is
also
useful
to
partition
(~
as
- .
-
where
4),
is a
t x
symmetric
submatrix
of
0
that
contains
trait
factor
in-
tercorrelations,
y
~M~. is
an
m
x m
symmetric
submatrix
of 4)
that
contains
method
fac-
tor
intercorrelations
and
4),,,
(= 4o’,)
is
an
m
x
t rectangular
submatrix
of #
that
contains
correlations
of
the
m
method
factors
with
the
train
factors.
Substituting
Equations
2
and
3
into
Equation
1
results
in
the
following
representation:
Assuming
that
the
factor
analytic
model
of Y,
rep-
resented
by
Equation
1
is
valid
and
that
scores
on
the
aaat
observed
measures
follow
a
multivariate
normal
distribution,
Joreskog
(1969,
1971)
devel-
oped
procedures
for
obtaining
maximum
likelihood
estimates
of
all
model
parameters
in
A,
(~,
and
~.
The
significance
of
each
of
the
parameters
in
the
model
may
be
tested
by
forming a
z-ratio
of
the
parameter
estimate
divided
by
its
asymptotic
stan-
dard
error;
parameter
estimates
with
z-ratios
greater
than
12.001
are
typically
considered
significant
be-
yond
the
.05
level.
In
addition
to
the
test
of
each
individual
parameter
estimate,
maximum
likeli-
hood
estimation
yields
an
overall
X2
goodness-of-
fit
test.
The X2
test
is a
test
of
the
difference
in
fit
1
In
the
present
paper,
for
ease
of
presentation
and
with
no
loss
of
generality,
models
are
proposed
under
the
assumption
that
observed
variables
have
been
standard-
ized
to
zero
mean
and
unit
variance,
and
thus
that I
is
a
correlation
matrix.
It
should
be
noted,
however,
that
the
generalization
of
the
procedures
outlined
to
analyses
of
covariance
matrices
is
straightforward.
The
decision
to
employ
the
metric
of
correlations
was
made
in
order
to
conform
to
the
standard
way
of
presenting
multitrait-
multimethod
data,
which
is
in
the
form
of
a
correlation,
as
opposed
to
a
covariance,
matrix.
Downloaded from the Digital Conservancy at the University of Minnesota, http://purl.umn.edu/93227.
May be reproduced with no cost by students and faculty for academic use. Non-academic reproduction
requires payment of royalties through the Copyright Clearance Center,
http://www.copyright.com/
4
between
a
given
model
and
a
completely
saturated
model
that
perfectly
represents
the
data.
Thus,
if
the
~2
value
associated
with
a
model
is
statistically
significant,
there
is
a
statistical
basis
for
rejecting
the
given
model
in
favor
of
one
with
fewer
restric-
tions 9 so
that
the
patterns
in
the
data
may
be
mod-
eled
more
accurately.
In
addition
to
the
statistical
fit
of
a
model, the
level
of
practical fit
of
a
covariance
structure
model
must
also
be
considered.
Because
the
x2
test
is
directly
related
to
sample
size,
a
model
that
rep-
resents
well
a
set
of
data, that
is,
that leaves
small
residual
covariances,
may
still
have
a
significant
~2
if
the
sample
is
rather
large.
On
the
other
hand,
a
model
that
fails
to
represent
a
data
set
well
may
have
a
nonsignificant
~2, which
would
support
acceptance
of
the
model,
if
the
sample
size
was
rather
small.
To
remedy
this
situation,
Bentler
and
Bonett
(1980)
proposed
two
measures
of
practical,
as
opposed to
statistical,
fit
of
covariance
structure
models.
These
two
measures
are
termed
rho
and
delta
and
are
calculated
in
the
following
manner:
rho - ~Xn~d.~~) -
(X21df
,
(5)
rho=
(X,2,ldf,) - s1
’
(5)
~ l~.n l LC~n )
I
and
(X,2
2
delta
=
~~~
s
(6)
delta =
(X?’ -2 X;) ,
1
(6)
d~’JLta
-
~/~n ~n
2
~s ~
9
’LD
where X2
is
the
chi-square
associated
with
the
null
model,
df~
is
the
degrees
of
freedom
for
the
null
model,
~s
is
the
chi-square
associated
with
a
sub-
stantive
model
under
consideration,
and
df,
is
the
degrees
of
freedom
for
the
sub-
stantive
model.
The
first
measure,
rho,
is
a
generalization
to
restricted
covariance
structure
models
of
the
Tucker
and
Lewis
(1973)
reliability
coefficient
for
unre-
stricted
factor
analysis
models.
Rho
is
a
relative
measure
of
off-diagonal
covariation
among
ob-
served
variables
explained
by a
rr~odel, a
relative
measure
because
the
fit
of
each
model
is
evaluated
with
regard to
the
degrees of
freedom
for
the
model.
The
second
measure,
delta,
is
an
absolute
measure
of
fit,
because
delta
generally
represents
the
pro-
portion of
off-diagonal
covariation
explained
by a
model,
regardless
of
degrees
of
freedom.
Bentler
and
Bonett
(1980)
stated
that
rho
and/or
delta
should
attain
values
of
.90
or
above
for a
model
to
be
accepted,
because
models
with
fit
indices
below
.90
can
usually be
simply
and
substantially
im-
proved.
Although
standards
for
evaluating
differ-
ences
in
measures
of
practical
fit
have
not
been
developed,
in
the
present
paper
differences
be-
tween
models
in
either
rho
or
delta
of
less
than
.01
1
were
considered
unimportant
on
practical
grounds.
A
third
general
type
of
consideration,
which
may
be
used
in
conjunction
with
statistical
significance
and
practical
importance
when
evaluating
covari-
ance
structure
models,
is
the
stability
of
parameter
estimates
after
respecification
of a
model.
If
Model
B
is
a
respecification
of
Model
A
obtained
by add-
ing
one
or
more
theoretically
meaningful
parameter
estimates
to
Model
A,
the
stability
of
estimates
of
parameters
that
are
common
to
Models
A
and B
is
of
interest.
If
the
common
parameter
estimates
show
a
high level
of
stability,
Model B
introduced
ad-
ditional
parameters
whose
effects
were
relatively
independent
of
those
of
parameters
in
Model
A.
If, on
the
other
hand,
the
estimates
of
common
parameters
are
rather
different
in
Model B
than
in
Model
A,
then
it
may
be
concluded
that,
regardless
of
differences
between
Models
A
and B
in
levels
of
statistical
and
practical
significance,
Model B
should
be
accepted.
All
other
considerations
(e.g.,
magnitude
of
standard
errors)
being
equal,
Model
B
may
have
allowed
less
biased
estimation
of
pa-
rameters
common
to
the
two
models.
This
stability
consideration
is
related
to
the
well-known
&dquo;third
variable,&dquo;
or
omitted
variable,
problem
in
path
analysis
(Duncan,
1975;
Kenny,
1979).
Although
highly
restricted
models
are
preferred
on
theoretical
and
practical
grounds,
specification
of
overly re-
stricted
models
that
omit
variables
important
to
the
system
will result
in
bias
in
estimation
of
param-
eters.
By
examining
the
stability
of
parameter
es-
timates
across
competing
covariance
structure
models
estimated
from
the
same
data,
it
is
possible
to
ob-
serve
whether
bias
is
apparent
in
particular
models.
Downloaded from the Digital Conservancy at the University of Minnesota, http://purl.umn.edu/93227.
May be reproduced with no cost by students and faculty for academic use. Non-academic reproduction
requires payment of royalties through the Copyright Clearance Center,
http://www.copyright.com/
5
Hierarchically
Models
For
h4ultitrait-hiultinaethod
Data
A
taxonomy
of
structural
models
for
multitrait-
multimethod
data.
Given
the
general
model
for
multitrait-multimethod
data
represented
by
Equa-
tions
2
through
4
above,
it
is
possible
to
specify
a
variety
of
theoretically
interesting
models
that
are
special
cases
of
the
general
model.
One
way
to
generate
the
array
of
structural
models
in
a
system-
atic
fashion
is
to
consider
separately
the
structures
that
may
be
specified
for
the
trait
and
method
factor
spaces.
Consider
first
the
trait
factor
space.
There
are
three
types
of
models
for
the
trait
space,
models
with
the
following
specifications:
(1)
no
trait
fac-
tors,
(2)
t trait
factors
with
fixed
intercorrelations,
and
(3)
t trait
factors
with
freely estimated
inter-
correlations.
These
three
types
of
models
were
translated
into
the
following
trait
structures
con-
sidered
in
the
present
article:
1
=
No
trait
factors
2
=
t
trait
factors,
fixed
unit
intercorrelations
2’
=
t trait
factors,
fixed
zero
intercorrelations
3
=
t trait
factors, freely
estimated
intercorrela-
tions
The
two
structures
with
fixed
trait
factor
inter-
correlations
are
given
identical
Arabic
numeral
la-
bels,
2
and
2’
(with
the
superscript
to
distinguish
the
structures),
because
the
two
structures
entail
the
same
number
of
parameter
estimates.
An
iden-
tical
number
of
estimates
is
made
under
Structures
2
and
2’,
since
in
neither
case
are
correlations
among
trait
factors
estimated:
under
Structure
2,
the
cor-
relations
among
trait
factors
are
fixed
at
unity,
re-
sulting
in
a
model
with
a
single,
general
trait
factor,
whereas
under
Structure
2’
the
trait
factors
are
forced
to
be
orthogonal.
Because
Structures
2
and
2’
have
the
same
number
of
estimates,
it
is
not
possible
to
obtain
a
statistical
test
of
the
difference
in
fit
of
the
two
structures,
since
neither
structure
is
nested
within
the other.
However,
given the
identical
number
of
parameter
estimates,
the
structure
as-
sociated
with
the
smaller
~2
value
would
be
pre-
ferred.
Finally,
note
that
the
magnitude
of
the
Arabic
numerals
assigned
to
structures
is
a
key
to
the
nest-
ing
of
the
structures.
When
comparing
two
struc-
tures,
the
structure
labeled
with
the
larger
number
is
the
more
inclusive
structure
and
has
the
larger
number
of
estimates;
the
structure
with
the
smaller
number
is
therefore
nested
within
the
more
inclu-
sive
structure.
That
is,
Structure
3
is
the
most
in-
clusive
structure,
Structures
2
and
2’
are
nested
within
Structure
3,
and
Structure
1
is
nested
within
Structures
2
and
2’
and
therefore
within
Structure
3
as
well.
A
set
of
structures
for
the
method
factor
space,
parallel
to
those
for
the
trait
factor
space,
may
also
be
specified.
This
results
in
the
following
short
list
of
method
factor
structures:
A
=
No
method
factors
B =
~a
method
factors,
fixed
unit
intercorrelations
B’
= ara
method
factors,
fixed
zero
intercorrelations
C
= m
method
factors,
freely
estimated
intercor-
relations
Distinctions
like
those
made
for
the
trait
space
structures
may
also
be
drawn
for
the
method
space
structures.
For
example,
Structures B
and
B’
are
alternative
structures
with
the
same
number
of
es-
timates
and
thus
cannot
be
compared
statistically.
In
addition,
the
letters
assigned
are
a
key
to
the
nesting
of
the
method
space
structures.
Of
two
method
structures,
the
structure
labeled
with
the
higher
letter
(i.e.,
the
letter
occurring
later
in
the
alphabet)
is
the
more
inclusive
structure
having
the
greater
number
of
estimates,
and
the
structure
with
the
smaller
(i.e.,
earlier) letter
is
nested
within
the
former
structure.
A
taxonomy
of
models
for
multitrait-multi-
method
data
may
then
be
generated
by
cross-clas-
sifying
the
four
trait
structures
and
the
four
method
structures,
as
shown
in
Table
1.
Each
of
the
models
in
Table
1
may
then
be
designated
by
one
number
and
one
letter,
which
would
denote
the
trait
and
method
structures,
respectively,
embodied
in
the
model.
For
example,
Model
3B’
has
t freely
cor-
related
trait
factors
and
m
orthogonal
method
fac-
tors.
Given
the
number
and
letter
labels
of
the
models
in
Table
1,
it
is
a
simple
matter
to
determine
whether
one
model
is
nested
within
another
model.
A
given
Downloaded from the Digital Conservancy at the University of Minnesota, http://purl.umn.edu/93227.
May be reproduced with no cost by students and faculty for academic use. Non-academic reproduction
requires payment of royalties through the Copyright Clearance Center,
http://www.copyright.com/