159
Factor
Indeterminacy
in
Generalizability
Theory
David
G.
Ward
Fordham
University
Generalizability
theory
and
common
factor
analysis
are
based
upon
the
random
effects
model
of
the
analy-
sis
of
variance,
and
both
are
subject
to
the
factor
inde-
terminacy
problem:
The
unobserved
random
variables
(common
factor
scores
or
universe
scores)
are
indeter-
minate.
In
the
one-facet
(repeated
measures)
design,
the
extent
to
which
true
or
universe
scores
and
com-
mon
factor
scores
are
not
uniquely
defined
is
shown
to
be
a
function
of
the
dependability
(reliability)
of
the
data.
The
minimum
possible
correlation
between
equivalent
common
factor
scores
is
a
lower
bound
es-
timate
of
reliability.
The
equivalence
of
the
random
effects
model
of
the
analysis
of
variance
and
the
model
of
factor
analysis
is
well-established
(Bock,
1960;
Burt,
1947;
Creasy,
1957),
as
is
the
equivalence
of
the
classical
test
theory
model
(the
true
score
model)
and
the
common
factor
model
(e.g.,
Steiger &
Sch6ne-
mann,
1978).
Generalizability
theory
(Cronbach,
Gleser,
Nanda, &
Rajaratnam,
1972)
extends
clas-
sical
test
theory
by
ascribing
variations
in
obser-
vations
to
specific
sources
(facets
or
conditions
of
measurement).
Because
generalizability
theory
is
based
upon
the
random
effects
model,
problems
associated
with
the
factor
model
may
also
be
as-
sociated
with
generalizability
studies.
One
such
problem
in
the
factor
model
is
the
lack
of
determinacy
of
the
factor
scores
which
presum-
ably
underlie
the
observed
scores
(e.g.,
Cattell,
1973,
p.
303;
Guttman,
1955;
Schbnemann,
1971).
In
short,
there
are
infinitely
many
solutions
for
the
unobserved
factor
scores,
and
thus
no
unique
so-
lution for
the
factor
model
as
defined
in
the
pop-
ulation.
Universe
scores
(person
variables)
are
unobserved
common
factor
scores,
and
they
too
are
subject
to
the
indeterminacy
problem.
There
is
considerable
controversy
surrounding
the
factor
indeterminacy
issue
(e.g.,
Jensen,
1983;
McDonald,
1977;
Sch6nemann,
1983;
Steiger
&
Schonemann,
1978).
For
example,
McDonald
(1974,
1977)
presented
a
case
for factor
variables
as
being
unique
but
unknown.
The
present
approach
takes
a
strictly
mathematical
view
of
the
linear
model
used
in
generalizability
theory,
namely
that
a
sys-
tem
of
linear
equations
is
indeterminate
if
there
are
more
unknown
variables
(common
and
unique
fac-
tors)
than
known
(observed)
variables.
This
study
examined
the
degree
of
factor
indeterminacy
in
the
one-facet
(repeated
measures)
design
used
in
gen-
eralizability
theory.
Generalizability
studies
focus
on
estimating
the
variance
components
associated
with
sources
of
variation
rather
than
on
estimating
individual
uni-
verse
scores
or
the
effects
associated
with
the
mea-
surement
conditions,
as
in
decision
studies
(e.g.,
Cronbach
et
al.,
1972,
pp.
16-17).
Variability
in
scores
is
examined
rather
than
estimation
of
uni-
verse
scores,
because
the
levels
(e.g.,
observers,
situations,
items)
are
assumed
to
be
randomly
se-
lected
from
some
relevant
universe,
as
in
random
Downloaded from the Digital Conservancy at the University of Minnesota, http://purl.umn.edu/93227.
May be reproduced with no cost by students and faculty for academic use. Non-academic reproduction
requires payment of royalties through the Copyright Clearance Center,
http://www.copyright.com/
160
effects
analysis of
variance.
There
is
considerable
literature
on
estimation
of
true
scores
(e.g.,
Lord
&
Novick,
1968,
pp.
152-153)
and
universe
scores
(e.g.,
Cronbach
et
al.,
1972,
pp.
73ff.),
but
the
factor
indeterminacy
problem
is
not
simply
one
of
estimating
universe
scores
or
person
effects.
Rather,
there
is
no
way
to
determine
a
person’s
universe
score
uniquely,
even
if
a
person’s
universe
score
is
defined
a
priori
as
a
parameter-that
is,
the
ex-
pectation
of
observed
scores
over
all
conditions
of
measurement
(randomly
parallel
tests), as
in
true
score
theory.
This
is
equivalent
to
taking
the
ex-
pectation
over
all
possible
levels
of
the
facet,
which
is
not
feasible,
especially with
large
universes
such
as
observers,
occasions,
or
situations.
It
will
be
shown
that
both
the
generalizability
coefficient
and
an
index
of
factor
indeterminacy
are
estimates
of
reliability.
The
lowest
possible
reliability
is
the
minimum
correlation
between
equivalent
universe
(or
common
factor)
scores.
While
the
goal
of
this
study
was
determining
the
number
of
observers,
occasions,
or
situations
needed
to
obtain
acceptable
reliability
in direct
behavioral
observation
or
rating
studies,
there
are
also
impli-
cations
for
factor
analytic
test
construction
(with
respect
to
choosing
the
number
of
items
in
a
test).
The
pragmatic
question,
then, is:
To
what
extent,
if
any,
does
this
indeterminacy
have
practical
sig-
nificance ?
What
is
the
number
of
items
or
obser-
vations
needed
to
achieve
a
given
level
of
reliable
measurement?
Overview
of
Generalizability
Theory
Attention
here
is
limited
to
fully
crossed
(bal-
anced)
one-facet
(repeated-measures)
designs,
though
the
results
may
extend
directly
to
other
models.
Working
with
deviation
scores
allows
con-
siderable
simplification
without
any
loss in
gen-
erality,
because
the
variance
component
estimates
are
unaffected
by
subtracting
the
population
mean
from
each
observation.
Thus,
an
observation
in
deviation
score
form
may
be
decomposed
into
in-
dependent
random
variables
corresponding
to
the
sources
of
variation
in
the
experimental
design,
plus
a
random
error
variable.
A
typical
application
of
a one-facet
design,
such
as
P
x
S
(person x
situation),
is
collecting
behav-
ioral
observations
or
ratings
on
~c
people
in
n,
dif-
ferent
situations.
The
~c
people
are
assumed
to
be
randomly
sampled
from
the
population
of
all
such
people, and
the n,
levels
of
facet S
are
assumed
to
be
randomly
selected
from
the
universe
of
all
such
conditions
of
measurement. A
sample
deviation
score
(Cronbach
et
al.,
1972,
p.
26)
is
represented
as
where
y,~
is
the
observed
deviation
score
for
person
i
under
condition
j;
pi
is
the
effect
of
person
(th~
universe
score
for
person
i minus
the
population
mean);
9
sj
is
the
effect
of
condition
j;
p.s,~
is
the
P
x
S
interaction
effect
of
person
i
under
condition
j;
and
e~~
is
random
error.
Because
differences
in
means
for
levels
of
facet S
do
not
affect
the
person-related
variance
compo-
nent
estimates,
the
different
effects
due
to
facet S
can
be
ignored
for
the
present
purposes.
The
model
assumptions
are:
and
p,
ps,
and e
are
independent.
~~er~~e9
In
the
usual
sample
estimation
problem,
psi~
and
eii
cannot
be
measured
separately
and
are
con-
founded
(the
residual
~sel,).
Thus, the
model
ac-
tually
used
is
where
pi
and
pseii
are
independent,
so
that
-2 - -2 I -2
~C11
The
variance
components
may
be
estimated
using
the
C®rnfield-Tuk~y
algorithm,
and
Downloaded from the Digital Conservancy at the University of Minnesota, http://purl.umn.edu/93227.
May be reproduced with no cost by students and faculty for academic use. Non-academic reproduction
requires payment of royalties through the Copyright Clearance Center,
http://www.copyright.com/
161
where
MSp
and
MSP-
are
the
mean
squares
for
persons
and
residual,
respectively.
Considering
the
population
case,
the
so-called
unit-sample
generalizability
coefficient
(~-~~ef~-
cient) p2
is
an
mtraclass
correlation coefficient
which
measures
the
reliability
of
a
single
judge,
or a
single
item
in
a
test,
as
in
traditional
reliability
(e.g.,
McKeon,
1969;
Winer,
1971,
pp.
283-286):
The
generalizability
of
the
mean
of
n,
measures,
the
full-sample
generalizability
coefficient
(or
coefficient
alpha;
Mulaik,
1972,
p.
209),
is
given
by
The
full-sample
g-coefficient
corresponds
to
the
reliability
of a
whole
test
comprised
of
n,
items,
which
may
also
be
computed
using
the
Spearman-
Brown
prophecy
formula.
Sch6nemann
(1971)
has
shown
that
for
a given
set
of
observations
which
satisfy
the
factor
model,
a
minimally
correlated
equivalent
set
of
uncorre-
lated
factor
scores
may
be
generated
which
cor-
respond
to
the
same
observations.
This
minimally
correlated
equivalent
set
of
orthogonal
factors
rep-
resents
the
worst
possible
case
of
factor
indeter-
minacy,
and
such
results
may
be used
to
evaluate
the
random
effects
model
as
used
in
generaliza-
bility
theory.
Factor
Indeterminacy
To
phrase the
random
effects
model
in
factor
analysis
terms,
consider
the
model
in
Equation
8.
This
discussion
holds
for
any
number
of
repeated
measures,
but
for
illustration,
consider
the
case
rcs
=
2.
This
discussion
is
based
on
Sch6nemann
(1971,
1983)
and
Steiger
and
Sch6nemann
(1978).
The
model
in
Equation 8
may
be
written
x
and
z.
are
independent
(j
=
1,2).
The
two
observed
random
variables
yj
in
y’
=
(y,,
Y2)
may
be
written
in
vector
notation
as
where
x is
an
unobserved
random
vari-
able,
the
standardized
&dquo;com-
mon
factor&dquo;;
the
zj
in
z’ =
Z2)
are
two
unobserved
random
variables,
standardized
&dquo;unique
factors&dquo;;
a
is
a
2
x
1
vector
of
factor
load-
ings,
the
common
factor
pat-
tern,
where
a~
=
(o~,
cr,);
U
is
a
positive
definite
diagonal
matrix,
the
unique
factor
pat-
tern,
where
U
= U&dquo;pseI2;
and
the
matrix
(3, U) =
(~P, o°pSe~)
is
the
total
pattern
(cf.
Sch6nemann,
1971,
p.
22).
The
&dquo;person
variable&dquo;
is
p
=
orpx,
and
the
person
x
situation
residual
variables
are
psel -
(3’pseZI
and
pse2
=
~pSez2.
Thus,
the
one-facet
(P
x
S)
design
corresponds
to
the
two-factor
model,
where
cYP2
is
the
person
variance
component
(com-
mon
variance
or
communality),
crp2s,
is
the
residual
variance
component
(unique
variance,
consisting
of
&dquo;true&dquo;
specific
and
error
variance),
and
the
co-
variance
matrix
of
the
observations
is
The
universe
(or
true)
scores
correspond
to
the
first
centroid
of
the
observed
data.
(The
possibility
of
the
person
variable
being
composed
of
more
than
one
common
factor
is
not
pursued
here,
nor
are
the
various
other
generalizability
models.)
In
the
usual
sample
estimation
problem,
Y
=
6rx’
+
ô-psJZ ,
y
(17)
where
Y(2
x
r~)
are
the
observed
deviation
scores;
x’(1
x
n)
is a
vector
of
standardized
&dquo;com-
mon
factor
scores&dquo;;
;
~(2 x
n)
is
a
matrix
of
standardized
and
un-
correlated’ ’unique
factor
scores&dquo;;
and
Downloaded from the Digital Conservancy at the University of Minnesota, http://purl.umn.edu/93227.
May be reproduced with no cost by students and faculty for academic use. Non-academic reproduction
requires payment of royalties through the Copyright Clearance Center,
http://www.copyright.com/
162
1b~
and
&,_ I
are
sample
estimates
of
up
and
o=ps,l.
Then,
9
The
essence
of
factor
indeterminacy
is
that
an
infinite
number
of
solutions
for
x
and z
exist
which
satisfy
Equation
15.
More
formally,
Sch6nemann
(1971)
has
shown
that
for
any
set
of
factors
(~, z~)
which
satisfy
Equation
15,
another
set
of
equiva-
lent
factors
may
be
found
which
also
satisfy
and
Equation
15,
where
the
transformation
matrix
T
is
an
orthogonal
right
unit
of
(a,U)
(Sch6ne-
mann,
1971,
p.
23).
An
orthogonal
right
unit
of
the
matrix
(a,U)
has
the
properties:
(a,U)T =
(a,
U)
and
TT’ =
T’T =
1.
T
is
also
the
coffe-
lation
matrix
between
equivalent
sets
of
factors:
The
T
which
minimizes
the
correlation
between
factor
pairs
is
given
by
(Sch6nemann,
1971,
Equation
5.2,
based
on
Gutt-
man,
1955,
and
I~eerrnann9
1966).
Although
T .. i,,
minimizes
the
sum
of
the
correlations
between
equivalent
factor
pairs
[i.c.,
tr(Tmin)],
which
in-
cludes
both
common
and
unique
factors,
the
min-
imum
possible
correlation
between
the
equivalent
common
factor
scores
may
even
be
lower
(cf.
Sch6nemann &
Wang,
1972).
In
general,
the
min-
imum
average
correlation
between
equivalent
sets
of
uncorrelated
factors
is
given
by
where
p
is
the
number
of
observed
random
vari-
ables,
m
is
the
number
of
common
factors,
and
tkk
are
the
diagonal
elements
of
T min
(i.e.
the
minimum
correlations
between
equivalent
sets
of
common
and
unique
factors;
Sch6-
nemann,
9 1971 ) .
In a
one-facet
generalizability
model, then, T =
(M, -
1)1(n,
+
1),
and
for
n,
=
2,
r -
1/3.
Perhaps
more
relevant
is
the correlation
between
equivalent
common
factors.
Schonemann
(1971,
p.
27)
discussed
the
minimum
average
correlation
between
equivalent
sets
of
~aa
uncorrelated
common
factors,
given
by
where
l’
is
the
mean
of
the
first
m
diagonal
elements
of
T min’
For
the
one-common
factor
model
used
in
generalizability
theory,
this
minimum
correlation
between
equivalent
sets
of universe
scores
based
on
minimizing
tr(T~)
is
t, r
element
(1,I)
of
T.i,,.
For
convenience,
we
can
define
this
corre-
lation
as
r min’
which
is
given
by
(from
Schonemann,
1971,
p.
27,
and
from
Equa-
tion
22).
Substituting
Op
and
or,,,
into
this
definition
of
r~n yields
the
following
in
terms
of
the
variance
components:
The
minimum
correlation
between
equivalent
com-
mon
factors
is
thus
the
full-sample
generalizability
minus
the
ratio
of
the
residual
variance
to
the
ob-
served
score
variance
used
in
the
denominator
of
the
full-sample
g-coefficient.
Downloaded from the Digital Conservancy at the University of Minnesota, http://purl.umn.edu/93227.
May be reproduced with no cost by students and faculty for academic use. Non-academic reproduction
requires payment of royalties through the Copyright Clearance Center,
http://www.copyright.com/
163
r min
reflects
the
degree
of
variability
in
possible
values
for
the
universe
scores
of
a
given
person
in
the
population,
as
a lower
bound
estimate
of
reli-
ability.
r min
is
the
cosine
of
the
angle
formed
by
possible
equivalent
universe
score
vectors:
The
smaller
the
angle,
the
less
the
indeterminacy.
The
ratio
of
r min
tons
is
-~-P
p
For
two
standardized
observed
variables
(ns
=
2), the
following
relationships
hold
(where
~°,2
is
the
Pearson
product-moment
correlation
between
the
two
measures):
Equation
29
is
derived
by
substituting
Equation
28
in
Equation
12,
solving
for
and
substituting
this
result in
Equation
26.
Thus,
if
r12 =
1,
then
p2
=
1
and
rmin
=
1;
and
if
r12
=
0,
then
p2
=
0
and
r~~
= -1.
If
the
observed
PPM
correlation
is
as
low
as
r12
=
1/3,
then
the
correlation
between
equivalent
common
factors
(universe
scores)
may
be
as
low
as
run
= 0.
For
studies
involving
more
than n,
= 2
(i.e.,
more
than
2
observers
or
items),
the
relationship
between
reliability
(p2 ns
and
r
and ns
is
shown
in
the
accompanying
figure
for
a
representative
value
of
the
unit-sample
g-coefficient
(p2
=
.25,
based
on
the
relationship
between
o-p
and
o-,2,,).
The
full-
sample
generalizability
coefficient,
which
is
the
reliability
of
an
ns-item
extended
test
and
represents
the
Spearman-Brown
prophecy
graph,
and
the
min-
imum
correlation
between
equivalent
universe
scores,
are
both
monotonically
increasing
functions
of
the
number
of
items
or
levels
of
the
facet.
The
difference
between
the
g-coefficient (top
curve)
and
rmin
(lower
curve)
is
as
in
Equation
26.
Other
initial
values
of
the
unit-
Figure
1
Generalizability
and
Minimum
Correlation
Between
Equivalent
Common
Factors
as
a
Function
of
Number
of
Levels
of
the
Facet
Downloaded from the Digital Conservancy at the University of Minnesota, http://purl.umn.edu/93227.
May be reproduced with no cost by students and faculty for academic use. Non-academic reproduction
requires payment of royalties through the Copyright Clearance Center,
http://www.copyright.com/