scispace - formally typeset
Search or ask a question
Journal ArticleDOI

A multitrait-multimethod validation of the Implicit Association Test: implicit and explicit attitudes are related but distinct constructs.

01 Jan 2007-Experimental Psychology (Hogrefe & Huber Publishers)-Vol. 54, Iss: 1, pp 14-29
TL;DR: The authors applied Campbell and Fiske's (1959) classic multitrait-multimethod design precepts to test the construct validity of implicit attitudes as measured by the Implicit Association Test (IAT).
Abstract: Recent theoretical and methodological innovations suggest a distinction between implicit and explicit evaluations. We applied Campbell and Fiske's (1959) classic multitrait-multimethod design precepts to test the construct validity of implicit attitudes as measured by the Implicit Association Test (IAT). Participants (N = 287) were measured on both self-report and IAT for up to seven attitude domains. Through a sequence of latent-variable structural models, systematic method variance was distinguished from attitude variance, and a correlated two-factors-per-attitude model (implicit and explicit factors) was superior to a single-factor-per-attitude specification. That is, despite sometimes strong relations between implicit and explicit attitude factors, collapsing their indicators into a single attitude factor resulted in relatively inferior model fit. We conclude that these implicit and explicit measures assess related but distinct attitude constructs. This provides a basis for, but does not distinguish between, dual-process and dual-representation theories that account for the distinctions between constructs.

Summary (5 min read)

Introduction

  • Also, this multitrait-multimethod (MTMM) approach does not identify the cognitive processes that distinguish the constructs.
  • The context of this analysis follows from Cronbach and Meehl’s (1955) classic discussion of construct validation where a construct is an indeterminant function of representation and process.
  • The authors research provides another avenue of evidence for this growing nomological net by examining the relationship between implicit and explicit attitude measures to determine whether they can be fairly interpreted as measuring a single construct, or whether they assess related, but distinct constructs.

Preliminary Evidence

  • Greenwald and Farnham (2000) observed that a model describing implicit and explicit self-esteem as distinct latent factors provided a better fit than a single self-esteem conceptualization.
  • Likewise, Cunningham, Preacher, and Banaji (2001) found implicit and explicit measures of racial attitudes to reveal related, but distinct factors, as did Cunningham, Netlek, and Banaji (2004) for implicit and explicit ethnocentrism.
  • Following this approach of comparing single versus dual factors in structural equation modeling, the authors reanalyzed a large dataset reported by Nosek (2005).
  • To transcend this inferential limitation, here, guided by principles articulated by Campbell and Fiske (1959), the authors use a MTMM design and comparative structural modeling analyses.
  • The authors findings demonstrate (1) convergent and discriminant validity of the IAT, (2) that a model of distinct, but related implicit and explicit attitudes best fits the data, and (3) that this characterization is not attributable to attitude-irrelevant method variance of the IAT or of self-report.

MTMM and Confirmatory Factor Analysis

  • In their classic article on construct validation, Campbell and Fiske (1959) articulated a strategy for using MTMM matrices to evaluate convergent and discriminant validity.
  • Campbell and Fiske argued that “the clear-cut demonstration of the presence of method variance requires both several traits and several methods” (p. 85).
  • Confirmatory factor analysis (CFA) has emerged as a tool well-suited to the partitioning of MTMM data envisioned by Campbell and Fiske (Jöreskog, 1974; Widaman, 1985).
  • 2 Cunningham, Nezlek et al. (2004) footnoted this limitation, but argued that, since a control IAT, birds vs. trees, did not load with five ingroup-outgroup IATs on an implicit ethnocentrism factor, IAT method variance was not a strong driver of the two-factor solution.
  • By comparing the fits of nested structural models, the relative merits of alternative hypotheses concerning the structure of trait and method variance can be systematically tested (Jöreskog & Sörbom, 1979; Loehlin, 2004; McDonald, 1985).

Study Overview

  • Data are from four laboratory studies in which attitudes toward seven different attitude-object pairs were measured: flowers–insects, Democrats–Republicans, humanities–science, straight–gay, Whites–Blacks, creationism–evolution, and thin people–fat people.
  • These domains were selected because on their face they cover a broad range of attitudes.
  • They reported support for this idea and also found that implicit and explicit ethnocentrism factors were related, but distinct.
  • The authors sought to demonstrate discriminant validity between attitude domains – hypothesizing that the attitude domains would form distinct factors; and convergent validity across measurement types (IAT and selfreport) – hypothesizing that the implicit and explicit attitude constructs would be related, but retain distinctiveness not accounted for by method factors.
  • This simultaneous examination of discriminant and convergent validity is the core value of the MTMM approach.

Implicit Association Test (IAT)

  • One of the four samples received IATs for all seven object pairs, while the others received subsets of at least four pairs, including the flowers–insects and Democrats–Republicans pairs .
  • For the remaining IATs, response blocks for all tasks were randomized and single-discrimination practice blocks were eliminated.
  • The authors allow substantial opportunity for method factors to influence IAT performance and challenge the hypothesis that distinguishable attitude factors can be identified despite intermixed performance blocks.
  • Using four indicators was preferable, however, in terms of attaining stable estimates in the more complex models.

Self-Report Measures

  • Participants reported attitudes toward each of the target objects per pair independently using two 9-point semantic differentials.
  • Anchors for these differentials varied across data collections, including, for a given study, two of the following four pairs: cold–warm, unpleasant–pleasant, bad–good, or unfavorable–favorable.
  • Positive values indicate greater liking for the object that was implicitly preferred on average (–8 to +8).

Procedure

  • Similar procedures were used across all four data collections.
  • After informed consent, participants completed a selection of IATs and self-report measures.
  • The order of implicit and explicit measures was counterbalanced between subjects.
  • The correlation matrices are substantively similar when each data collection is considered independently.

Analyses

  • Following guidelines suggested by Campbell and Fiske (1959), the authors first describe and offer interpretations of the reliability and validity relations in the MTMM matrix.
  • Thus, if the 95% CI for the εaΔ includes .05, the models being compared are considered close to one another in fit, and one would be preferred only to the extent that it involves fewer parameter estimates (i.e., is more parsimonious).
  • In Model 1 the authors specify two method factors to account for the covariances among the 42 observed indicators (14 explicit and 28 implicit) across all seven general attitude domains.
  • This is accomplished by specifying models that are identical to Model 3 except that, in turn, a common implicit method factor is not specified (Model 4) and a common explicit method factor is not specified (Model 5).
  • By comparing the fits of each with that of Model 3, the authors may discern to what extent accounting for common method variance is important to understanding the structure of relations among these measures.

Description of the MTMM Correlations

  • Table 2 is a MTMM matrix for the two methods and seven attitude object pairs.
  • The first three rows of the table provide descriptive statistics for each of the fourteen measurements (i.e., full IAT D scores and averages of the self-reports).
  • Reliability estimates (Cronbach’s α) are shown in parentheses along the main diagonal in the top-left (IAT) and bottom-right (self-report) panels, and intramethod (Campbell and Fiske’s monomethod-heterotrait) correlations are listed in the other cells of these respective panels.
  • For the IAT, reliabilities are based on D scores for split-halves formed from alternating couplets of trials, since a couplet consisting of an object stimulus (e.g., a science word) and an evaluative stimulus (e.g., a “good” word) occurred every two trials.
  • That is, little evidence of common method variance is apparent across attitude domains.

IAT

  • Correlations in the gray diagonal of the bottom-left panel can be used to assess convergent validity (monotrait-heteromethod), and discriminant validity (heterotrait-heteromethod) can be assessed by those off the diagonal in this panel.
  • These data are consistent with their hypotheses for the convergent and discriminant validity of the IAT and self-report across attitude domains.
  • The power of the MTMM design is not fully harnessed by scrutinizing correlation matrices.
  • The relative merits of competing hypotheses about the structure of the data can be tested formally by comparing the fits of nested, but differentially specified, structural equation models.

MTMM Structural Equation Models

  • Summary statistics from the confirmatory structural models are listed in Table 3 (details of each model’s specifications and parameter estimates – all fit with Mplus statistical software (Muthén & Muthén, 1998–2004) – are available in the supplement to this paper at http://briannosek.com/).
  • In Model 1, the first of substantive interest , two method factors are specified, one loading on the explicit indicators and one on the implicit.
  • Comparing this model’s fit with that of Model 2 tests whether specifying distinct implicit and explicit attitude factors is superior to a single-attitude factor per domain model for these data.
  • To summarize, there was relatively little common method variance to account for in these data; statistically significant, but small amounts were found for both the explicit and implicit measurements.

Follow-Up Analyses

  • The authors conducted additional analyses to evaluate the possibility that method variance is underestimated in these models because both the IAT and the explicit measures have rational zero points; both indicate a preference for one category compared to another .
  • Some factors may primarily influence the extremity of the score away from neutrality (0).
  • People who are more skilled at task-switching will achieve less extreme scores regardless of whether, for example, they are proDemocrat or pro-Republican.
  • This provides a liberal test for the method factor influences because it reduces the constructvalid variance by treating positive and negative score values as the same, and enhances the opportunity to see influences of extremity (distance from 0) as indicating common influence on the implicit or explicit measures.
  • Refitting the sequence of models summarized in Table 3 with the absolute values for each indicator yielded the same pattern of results (a table in which these results are listed is part of the online supplement, http://briannosek.com/).

Highly Correlated Domains

  • To more rigorously test the generality of their two-attitude specification, the authors fit the sequence of models only to the data for the three domains with relatively strong implicit–explicit correlations: creationism–evolution, Democrats–Republicans, and science–humanities.
  • Thus, by using the most highly correlated domains, and again partitioning common method variance, the authors increased the likelihood that a single attitude specification would suffice to account for variable interrelations.

Discussion

  • The results of this study add to construct validation evidence for the IAT as a measure of attitudes (Greenwald & Nosek, 2001).
  • The convergent validity of the IAT was evidenced by significant factor correlations between the implicit and explicit attitude constructs in five of the seven attitude domains, while its discriminant validity was simultaneously evidenced by the statistical superiority of the two-attitude model to the single-attitude model.
  • There was a modest amount of common method variance to account for, with statistically significant but small portions isolated from both the explicit and implicit measures, and this was also observed when absolute values of the indicators were used.
  • As Campbell and Fiske (1959, p. 84) observed, “In any given psychological measuring device, there are certain features or stimuli introduced specifically to represent the trait that it is intended to measure.
  • The MTMM design, coupled with comparative structural modeling, allowed common implicit method variance to be distinguished from implicit attitude variance.

Other Components of the Nomological Net for the Implicit Attitude Construct

  • This research provides a basis for some key components of validation of the IAT and implicit attitudes.
  • If the IAT reflected evaluations of the stimulus items and not the categories, then the implicit–explicit distinction might be explained by this difference.
  • The effort required for the chemist to specify the snowboarder’s constructs in terms of a single H20 construct would produce ungainly process theories that would likely “miss the point” of the snow and ice constructs.
  • In short, the present evidence for distinct implicit and explicit attitude constructs does not rule out the possibility that the two constructs derive from common evaluative content.

Conclusion

  • Convergent evidence across a variety of research programs suggests that the IAT is a valid measure of attitudes (see Nosek et al., 2006, for a review).
  • Like other methods such as semantic differentials, Likert scales, sequential priming, and the Stroop task, the IAT can be adapted to measure evaluations of many types of social categories.
  • The cumulative evidence identifies design factors that will influence the method’s validity, and provides a nomological net of knowledge to accelerate validation of novel applications of the IAT.
  • The emergence of the implicit attitude construct has spurred investigations to test the strength and limitations of this concept and its measurement tools, like the IAT.
  • The authors found simultaneous evidence of convergent and discriminant validity of the IAT and self-report as measures of related but distinct attitude constructs, and as distinct from methodological variation.

Did you find this useful? Give us your feedback

Content maybe subject to copyright    Report

B.A. Nosek & F.L. Smyth: A Multitrait-MultimethodValidation of the Implicit Association TestExperimentalP sychology 2007; Vol. 54 (1):14–29© 2007 Hogrefe & Huber Publishers
A Multitrait-Multimethod Validation
of the Implicit Association Test
Implicit and Explicit Attitudes Are Related
butDistinctConstructs
Brian A. Nosek and Frederick L. Smyth
University of Virginia, Charlottesville, VA, USA
Abstract. Recent theoretical and methodological innovations suggest a distinction between implicit and explicit evaluations. We applied
Campbell and Fiske’s (1959) classic multitrait-multimethod design precepts to test the construct validity of implicit attitudes as measured
by the Implicit Association Test (IAT). Participants (N = 287) were measured on both self-report and IAT for up to seven attitude domains.
Through a sequence of latent-variable structural models, systematic method variance was distinguished from attitude variance, and a
correlated two-factors-per-attitude model (implicit and explicit factors) was superior to a single-factor-per-attitude specification. That is,
despite sometimes strong relations between implicit and explicit attitude factors, collapsing their indicators into a single attitude factor
resulted in relatively inferior model fit. We conclude that these implicit and explicit measures assess related but distinct attitude constructs.
This provides a basis for, but does not distinguish between, dual-process and dual-representation theories that account for the distinctions
between constructs.
Keywords: implicit social cognition, attitudes, individual differences, construct validity, structural equation modeling
Realizing that the human mind is more than the sum of its
conscious processes, a number of theorists have proposed
a conceptual distinction between evaluations that are the
products of introspection, called explicit attitudes, and
those that occur automatically and may exist outside of
conscious awareness, called implicit attitudes (Greenwald
& Banaji, 1995; Wilson, Lindsey, & Schooler, 2000).
Greenwald and Banaji (1995, p. 8), for example, defined
implicit attitudes as “introspectively unidentified (or inac-
curately identified) traces of past experience that mediate
favorable or unfavorable feelings toward an attitude ob-
ject.” This theory has developed in conjunction with the
invention of measurement tools that assess automatic eval-
uative associations without introspection (e.g., Fazio, San-
bonmatsu, Powell, & Kardes, 1986; Greenwald, McGhee,
& Schwartz, 1998; Nosek & Banaji, 2001; Wittenbrink,
Judd, & Park, 1997).
Some experiences with these new measurement tools
have spawned doubts about whether they measure attitudes
at all (Karpinski & Hilton, 2001), and whether a conceptual
distinction between implicit and explicit attitudes is worth-
while (Fazio & Olson, 2003). Fazio and Olson contend “it
is more appropriate to view the measures as implicit or
explicit, not the attitude (or whatever other construct)”
(2003, p. 303). The purpose of the research we report was
to test whether the structure of attitude variance derived
from an implicit measure (the Implicit Association Test
[IAT]; Greenwald et al., 1998) and from an explicit one
(semantic differentials) is best represented by one latent
factor or by two correlated latent factors, when stripped of
confounding method variance. If our hypothesis that the
latter specification will fit the data better is sustained, it will
support a view that substantively different attitude con-
structs, distinguishable from the techniques used to mea-
sure them, underlie data collected by explicit and implicit
methods.
The finding would not, however, lend weight to one side
or the other in the debate about origins of the distinction
between implicit and explicit attitudes i.e., are they de-
rived from a single representation at different stages of pro-
cessing (Fazio & Olson, 2003) or do they reflect distinct
evaluative sources (Strack & Deutsch, 2004; Wilson et al.,
2000). Also, this multitrait-multimethod (MTMM) ap-
proach does not identify the cognitive processes that dis-
tinguish the constructs.
The context of this analysis follows from Cronbach and
Meehl’s (1955) classic discussion of construct validation
where a construct is an indeterminant function of represen-
tation and process. So, our reference to distinct implicit and
explicit attitude constructs should be interpreted as refer-
ring to distinguishable attitudinal components, without im-
plying a specific commitment to distinguishable formative
processes, single versus dual mental representations, or sin-
gle versus dual operative processes.
Any of these theoretical perspectives can explain dual
constructs by postulating combinations of representations
DOI 10.1027/1618-3169.54.1.14
Experimental Psychology 2007; Vol. 54(1):14–29 © 2007 Hogrefe & Huber Publishers

and processes to account for the observed differences be-
tween constructs. Dual-construct validation justifies the
need to have theoretical models account for the distinction
without providing evidence for or against any particular
explanation. Further illustration of the difference between
construct validation versus commitments to dual-process
or dual-representation theories appears in the discussion
(see also Greenwald, Nosek, & Banaji, in press).
Construct Validation
The conceptual and empirical justification for a psycholog-
ical construct requires development of a nomological net
of facts, relationships, and validity evidence that clarifies
the identity and utility of the construct (Cronbach & Meehl,
1955; McArdle & Prescott, 1992). The nomological net
supporting the validity of implicit attitudes has been gain-
ing strength (Greenwald & Banaji, 1995; Nosek, Green-
wald, & Banaji, in press; Wilson et al., 2000). For example,
Poehlman, Uhlmann, Greenwald, and Banaji (2004) con-
ducted a meta-analysis of studies examining the predictive
validity of the IAT, a measure thought to be influenced by
automatic associations, and found that the IAT had robust
predictive validity across domains, and outperformed self-
report measures in some domains (stereotyping and preju-
dice), while self-report outperformed the IAT in other do-
mains (e.g., political preferences). Also, recent social-neu-
roscience research finds evidence for a neurological
distinction between automatic and controlled evaluative
processes (Cunningham, Johnson, Gatenby, Gore, & Bana-
ji, 2003; Cunningham, Johnson et al., 2004). Our research
provides another avenue of evidence for this growing no-
mological net by examining the relationship between im-
plicit and explicit attitude measures to determine whether
they can be fairly interpreted as measuring a single con-
struct, or whether they assess related, but distinct con-
structs.
Preliminary Evidence
Greenwald and Farnham (2000) observed that a model de-
scribing implicit and explicit self-esteem as distinct latent
factors provided a better fit than a single self-esteem con-
ceptualization. Likewise, Cunningham, Preacher, and Ba-
naji (2001) found implicit and explicit measures of racial
attitudes to reveal related, but distinct factors, as did Cun-
ningham, Netlek, and Banaji (2004) for implicit and explic-
it ethnocentrism. Following this approach of comparing
single versus dual factors in structural equation modeling,
we reanalyzed a large dataset reported by Nosek (2005).
We found support for the generalizability of a model of
distinct-but-related latent implicit and explicit attitude con-
structs across 56 of 57
1
widely varying attitude domains
showing that this observation is quite general (see Table 1
and the supplement to this paper available at http://brian-
nosek.com/ for a full report). Even so, in this and the other
previous structural modeling studies, specification of im-
plicit and explicit attitude constructs is confounded with
measurement method. As a consequence, a two-factor so-
lution is an indeterminant function of both attitude and
method variance.
2
To transcend this inferential limitation, here, guided by
principles articulated by Campbell and Fiske (1959), we use
a MTMM design and comparative structural modeling anal-
yses. This approach allowed us to distinguish attitude and
method variance from IAT and thermometer ratings, the op-
erationalizations of implicit and explicit attitudes, respective-
ly. Our findings demonstrate (1) convergent and discriminant
validity of the IAT, (2) that a model of distinct, but related
implicit and explicit attitudes best fits the data, and (3) that
this characterization is not attributable to attitude-irrelevant
method variance of the IAT or of self-report.
MTMM and Confirmatory Factor
Analysis
In their classic article on construct validation, Campbell
and Fiske (1959) articulated a strategy for using MTMM
matrices to evaluate convergent and discriminant validity.
This strategy requires measurement of two or more osten-
sibly distinct trait constructs by two or more measurement
methods. Convergent validity is demonstrated when indi-
cators of a given trait (or, in our study, attitude) correlate
highly across measurement method, while discriminant va-
lidity obtains when correlations between ostensibly differ-
ent traits are low. Campbell and Fiske argued that “the
clear-cut demonstration of the presence of method variance
requires both several traits and several methods” (p. 85).
They described ways to statistically evaluate the respective
contributions of trait and method factors, but looked for-
ward to continued progress in developing more rigorous
validation methods.
Confirmatory factor analysis (CFA) has emerged as a
tool well-suited to the partitioning of MTMM data envi-
sioned by Campbell and Fiske (Jöreskog, 1974; Widaman,
1985). According to Marsh and Grayson (1995, p. 181),
B.A. Nosek & F.L. Smyth: A Multitrait-Multimethod Validation of the Implicit Association Test 15
© 2007 Hogrefe & Huber Publishers Experimental Psychology 2007; Vol. 54(1):14–29
1 The two-factor model for the males-females attitude domain failed to converge, leaving the hypothesis untested for this domain.
2 Cunningham, Nezlek et al. (2004) footnoted this limitation, but argued that, since a control IAT, birds vs. trees, did not load with five
ingroup-outgroup IATs on an implicit ethnocentrism factor, IAT method variance was not a strong driver of the two-factor solution. They
further suggested that if “systematic measurement error alone” was responsible for the implicit ethnocentrism factor, then the substantial
correlation between implicit and explicit ethnocentrism (r = .47) would be unlikely. We do not disagree, and we test this supposition
systematically.

Table 1. Results of one- and two-factor (oblique) attitude models fit to measures of 57 topics, a structural equation model
reanalysis of Nosek (2005)
Attitude topic comparisons One-factor model Two-factor model
AB N χ² ε
a
90%CIε
a
χ² ε
a
90%CIε
a
Factors r (t)
Whites Asians 279 44 .28 .21–.35 1.4 .04 .00–.17 .01 (0.1)
Cold Hot 211 262 .79 .71–.87 0.5 .00 .00–.16 .13 (2.1)
Skirts Pants 255 73 .37 .30–.45 0.1 .00 .00–.10 .15 (3.1)
Future Past 235 76 .40 .32–.48 3 .10 .00–.23 .26 (1.9)
Thin people Fat people 275 50 .30 .23–.37 0.4 .00 .00–.14 .27 (2.0)
Approaching Avoiding 180 28 .27 .19–.36 0.3 .00 .00–.16 .29 (1.3)
Simple Difficult 210 56 .36 .28–.44 0.0 .00 .00–..01 .29 (3.4)
Public Private 196 40 .31 .23–.40 0.0 .00 .00–..05 .30 (3.7)
Freedom Security 220 67 .38 .31–.47 1 .00 .00–.16 .31 (2.5)
Short people Tall people 226 53 .34 .26–.42 3 .09 .00–.22 .31 (2.9)
Family Career 238 42 .29 .22–.37 0.1 .00 .00–.12 .33 (2.7)
Married Single 261 169 .57 .50–.64 0.5 .00 .00–.14 .33 (3.4)
Rich people Poor people 222 89 .44 .37–.52 1 .00 .00–.17 .34 (3.2)
Education Defense 214 50 .33 .26–.42 2 .07 .00–.21 .36 (3.7)
Letters Numbers 228 75 .40 .33–.48 0.1 .00 .00–.12 .37 (3.4)
Nerds Jocks 239 80 .40 .33–.48 0.0 .00 .00–..00 .38 (4.0)
Young people Old people 250 42 .28 .21–.36 0.0 .00 .00–.00 .40 (3.1)
Imprisonment Capital punishment 260 66 .35 .28–.43 0.2 .00 .00–.13 .41 (3.7)
Yankees Diamondbacks 200 42 .32 .24–.40 1 .04 .00–.20 .41 (4.5)
Flexible Stable 201 35 .29 .21–.37 0.0 .00 .00–.10 .42 (4.4)
Meg Ryan Julia Roberts 255 41 .28 .21–.35 1 .03 .00–.17 .43 (4.7)
Emotion Reason 175 84 .49 .40–.58 0.0 .00 .00–.09 .44 (3.9)
Conforming Rebellious 208 132 .56 .48–.64 0.4 .00 .00–.16 .44 (4.2)
Summer Winter 260 133 .50 .43–.58 0.4 .00 .00–.14 .44 (5.5)
Leaders Helpers 265 60 .33 .26–.41 2 .07 .00–.19 .45 (5.4)
Tom Cruise Denzel Washington 242 52 .32 .25–.40 0.3 .00 .00–.14 .46 (4.8)
Management Labor 194 47 .34 .26–.43 0.1 .00 .00–.14 .47 (3.9)
Exercising Relaxing 258 185 .60 .53–.67 0.3 .00 .00–.14 .48 (5.8)
Jay Leno David Letterman 196 49 .35 .27–.43 0.0 .00 .00–.04 .48 (4.5)
American places Foreign places 205 44 .32 .24–.40 0.0 .00 .00–.11 .49 (5.3)
Microsoft Apple 205 77 .43 .35–.51 3 .09 .00–.23 .49 (4.1)
California New York 253 66 .36 .29–.43 2 .05 .00–.18 .49 (4.3)
Tea Coffee 250 90 .42 .35–.50 0.0 .00 .00–.00 .50 (5.6)
Abstaining Drinking 249 100 .44 .37–.52 0.2 .00 .00–.13 .50 (5.9)
Christian Jewish 253 77 .39 .31–.46 0.0 .00 .00–.00 .50
(5.2)
Classical Hip Hop 243 106 .47 .39–.54 1 .00 .00–.16 .51 (6.0)
Northerners Southerners 207 62 .39 .30–.47 1 .00 .00–.17 .51 (5.8)
Jews Muslims 243 52 .32 .25–.40 2 .07 .00–.20 .52 (5.5)
Books Television 233 77 .40 .33–.48 0.1 .00 .00–.11 .54 (5.2)
Cats Dogs 258 77 .38 .31–.46 1 .00 .00–.15 .54 (5.6)
16 B.A. Nosek & F.L. Smyth: A Multitrait-Multimethod Validation of the Implicit Association Test
Experimental Psychology 2007; Vol. 54(1):14–29 © 2007 Hogrefe & Huber Publishers

“the operationalizations of convergent validity, discrimi-
nant validity, and method effects in the CFA approach ap-
parently better reflect Campbell and Fiske’s (1959) original
intentions than do their own guidelines.” By comparing the
fits of nested structural models, the relative merits of alter-
native hypotheses concerning the structure of trait (atti-
tude) and method variance can be systematically tested
(Jöreskog & Sörbom, 1979; Loehlin, 2004; McDonald,
1985). We used this approach to distinguish method vari-
ance from trait variance for seven attitude object pairs.
And, within this framework, we tested whether a single
attitude construct or distinct implicit and explicit attitude
constructs provides a better fit for the data.
This MTMM design, because it enables direct modeling
of method factors, increases confidence that a finding of
distinct implicit and explicit attitude factors indicates a sub-
stantive distinction and not one that is driven by confound-
ing influences in the measurement requirements. Our im-
plicit measurement instrument, the IAT, measures associa-
tions between concepts (e.g., thin–fat) and attributes (e.g.,
good–bad) by comparing the average response times for
sorting exemplars of those concepts and attributes in two
distinct response conditions one in which sorting thin and
good exemplars requires a single response and sorting fat
and bad exemplars requires an alternate response, and a
second in which sorting fat and good exemplars requires a
single response and sorting thin and bad exemplars requires
an alternate response.
This method is distinct from attitude self-report in which
a participant self-assesses attitudes by reporting the mag-
nitude of good or bad feelings on a response scale. Because
of their radically different measurement properties, it is
possible that the unique components of the better two-fac-
tor models observed in earlier research are the result of
sources of method variance such as cognitive fluency or
task-switching ability, two known influences on IAT effects
(Mierke & Klauer, 2003; see Nosek et al., 2006, for a re-
view).
Study Overview
Data are from four laboratory studies in which attitudes
toward seven different attitude-object pairs were measured:
flowers–insects, Democrats–Republicans, humanities–sci-
ence, straight–gay, Whites–Blacks, creationism–evolution,
and thin people–fat people. These domains were selected
because on their face they cover a broad range of attitudes.
This was important so that our goal of accounting for
common method variance would not be confounded with
common substantive variance. For example, Cunningham,
Nezlek, and Banaji’s (2004) examination of implicit and
explicit ethnocentrism specifically hypothesized that atti-
tudes for a variety of ingroup-outgroup domains (e.g.,
poor–rich, Blacks–Whites, Jews–Christians) would share a
Attitude topic comparisons One-factor model Two-factor model
AB N χ² ε
a
90%CIε
a
χ² ε
a
90%CIε
a
Factors r (t)
European Americans African Americans 254 39 .27 .20–.35 0.0 .00 .00–.09 .55 (4.7)
American Canadian 290 44 .27 .20–.34 0.5 .00 .00–.14 .55 (4.7)
Teen pop Jazz 239 81 .41 .33–.48 1 .00 .00–.15 .56 (6.1)
Vegetables Meat 234 76 .40 .32–.48 0.3 .00 .00–.14 .56 (5.4)
Social programs Tax reductions 188 77 .45 .36–.53 5 .15 .04–.28 .57 (5.6)
USA Japan 246 49 .31 .24–.39 0.0 .00 .00–.00 .57 (5.2)
Gun rights Gun control 216 85 .44 .36–.52 1 .04 .00–.19 .59 (6.8)
Straight people Gay people 175 35 .31 .22–.40 0.1 .00 .00–.14 .60 (5.3)
Religion Atheism 211 160 .61 .53–.69 2 .05 .00–.20 .61 (7.1)
Coke Pepsi 250 71 .37 .30–.45 1 .00 .00–.16 .66 (7.1)
Liberals Conservatives 215 124 .53 .46–.61 1 .00 .00–.17 .67 (6.9)
Creationism Evolution 231 99 .46 .39–.54 3 .08 .00–.21 .68 (7.2)
Feminism Traditional values 226 75 .40 .33–.48 0.4 .00 .00–.15 .71 (7.4)
Gore Bush 211 61 .37 .30–.46 0.0 .00 .00–.09 .74 (7.2)
Democrats Republicans 195 42 .32 .24–.41 2 .08 .00–.23 .75 (6.7)
Prochoice Prolife 242 52 .32 .25–.40 0.4 .00 .00–.10 .79 (8.6)
Females Males 289 112 .44 .37–.51 * * * *
Note. Attitude object A was implicitly preferred on average. All one-factor models have df = 2, and two-factor models have df =1.ε
a
=
root-mean-square error of approximation. CI = confidence interval. t = r/se. Factor correlations in boldface have t 2.0. * model did not
converge.
Table 1 continued.
B.A. Nosek & F.L. Smyth: A Multitrait-Multimethod Validation of the Implicit Association Test 17
© 2007 Hogrefe & Huber Publishers Experimental Psychology 2007; Vol. 54(1):14–29

common ethnocentrism factor. They reported support for
this idea and also found that implicit and explicit ethnocen-
trism factors were related, but distinct.
Our goal, in one sense, was the opposite of Cunningham
et al.’s: they sought to demonstrate convergent validity be-
tween conceptually related attitude domains revealing an
ethnocentrism factor. We sought to demonstrate discrimi-
nant validity between attitude domains hypothesizing that
the attitude domains would form distinct factors; and con-
vergent validity across measurement types (IAT and self-
report) hypothesizing that the implicit and explicit atti-
tude constructs would be related, but retain distinctiveness
not accounted for by method factors. This simultaneous ex-
amination of discriminant and convergent validity is the
core value of the MTMM approach.
Campbell and Fiske (1959) stressed that “One cannot
define [a construct] without implying distinctions, and the
verification of these distinctions is an important part of the
validational processes” (p. 84). Self-report and IAT mea-
sures were obtained for each attitude object pair (e.g.,
Whites–Blacks) and participants were measured on multi-
ple pairs. We fitted a sequence of nested covariance struc-
ture models, beginning with one in which method variance
was partitioned into latent factors, and we predicted that a
model specifying distinct implicit and explicit attitude fac-
tors would provide the best data fit, whether or not the par-
titioning of method variance proved important.
Method
Participants
A total of 287 Yale University undergraduates from four
data collections in 2000 and 2001 (n = 81, 86, 60, 60) com-
prise the study sample.
Materials
Implicit Association Test (IAT)
One of the four samples received IATs for all seven object
pairs, while the others received subsets of at least four
pairs, including the flowers–insects and Democrats–Re-
publicans pairs (patterns of measured variables are listed
in the Appendix). All IAT category headings and exem-
plars are listed in the Appendix. IAT D scores were com-
puted based on the scoring algorithm suggested by Green-
wald, Nosek, and Banaji (2003), that is, by taking the dif-
ference in mean response latency between the two critical
block conditions and scaling it by the participant’s aver-
age latency standard deviation for both blocks. Most of
the IATs administered across the four samples consisted
of 56 trials in each of the critical double-discrimination
blocks.
3
IAT scores were removed from analysis if more
than 10% of trials were unreasonably fast (< 300 ms) or
if the error rate for any block of trials was greater than
39%. This cleaning resulted in elimination of less than 1%
of IAT scores (14 of 1475).
In all data collections, participants first completed a
flowers–insects IAT with the order of blocks conforming
to that suggested by Greenwald et al. (2003), i.e., (1) a
single-discrimination block of trials for bad–good exem-
plars, followed by (2) a single-discrimination block for
flower–insect exemplars, then (3) a double-discrimination
block of (counterbalanced) either flowers+good/in-
sects+bad or flowers+bad/insects+good, followed by (4)
another single-discrimination block to practice the
switched flower–insect key assignments for the (5) final,
reversed, double-discrimination block. For the remaining
IATs, response blocks for all tasks were randomized and
single-discrimination practice blocks were eliminated.
For example, after the flowers–insects IAT, a participant
could receive the race attitude compatible block of trials
(i.e., compatible with the dominant prejudice) in which
Black faces and bad words are to be categorized with one
response key and White faces and good words are catego-
rized with another key, without any opportunity to prac-
tice the simple, single-discrimination task of sorting Black
from White faces; then the participant could receive the
incompatible block for the gay–straight IAT (gay+good
with one key, straight+bad with the other); then the Dem-
ocrat+bad/Republican+good block; then the incompatible
block for race attitude, etc. This atypical approach pro-
vides a tough test for identifying attitude factors because
the component performance tasks are intermixed with per-
formance tasks for the other attitude domains. In this way,
we allow substantial opportunity for method factors to in-
fluence IAT performance and challenge the hypothesis
that distinguishable attitude factors can be identified de-
spite intermixed performance blocks.
To facilitate latent variable analyses, four IAT D scores
were calculated for each attitude domain based on the dif-
ference between the means of each fourth of the trials, in
turn, across the critical blocks. That is, for IATs with 56
critical trials in each double-discrimination block, the
mean latency for trials 1–14 in one double-discrimination
condition was compared with that of trials 1–14 in the
other, and so on for sets of trials 15–28, 29–42, and
43–56.
4
18 B.A. Nosek & F.L. Smyth: A Multitrait-Multimethod Validation of the Implicit Association Test
Experimental Psychology 2007; Vol. 54(1):14–29 © 2007 Hogrefe & Huber Publishers
3 Exceptions to the 56-trial critical block format were the flowers–insects IAT for two of the samples (40 trials in critical blocks) and the
creation–evolution and thin–fat IATs for 13 participants in one of the samples, also 40 critical trials.
4 We also conducted the analysis with just two IAT indicators, first half vs. second half of trials from each block, and found comparable results
in our comparative structural modeling. Using four indicators was preferable, however, in terms of attaining stable estimates in the more
complex models.

Citations
More filters
Journal ArticleDOI
TL;DR: It is suggested that including measures of narcissism in psychological assessment batteries administered to sexual offenders against children could be beneficial and the wisdom of targeting low self-esteem in correctional treatment programs aimed at reducing recidivism is questioned.
Abstract: The relationship between implicit and incongruent explicit and implicit (i.e., discrepant) self-esteem, narcissism, and sexual offending against children requires further research. We examined the relationships between self-esteem, narcissism, and risk of violent (including sexual) recidivism and compared sexual offenders against children (n = 28) and non-sexual offenders (n = 44) on these measures. All participants were adult men. In both groups, greater narcissism was associated with greater risk. Among sexual offenders against children, contrary to theoretical accounts and previous research findings, higher explicit self-esteem was associated with greater risk. However, further analysis indicated that explicit self-esteem may only be relevant to recidivism risk insofar as it reflects narcissism. Neither implicit nor discrepant self-esteem were related to recidivism risk in either group. Group comparisons indicated that sexual offenders against children had lower explicit self-esteem and narcissism than non-sexual offenders. In summary, our findings suggest that although sexual offenders against children have lower explicit self-esteem and narcissism than other offenders, within this group, higher explicit self-esteem and greater narcissism may be associated with greater risk. Taken together, previous and current findings suggest that including measures of narcissism in psychological assessment batteries administered to sexual offenders against children could be beneficial. Our findings also further question the wisdom of targeting low self-esteem in correctional treatment programs aimed at reducing recidivism.

3 citations

Journal ArticleDOI
05 Jan 2022
TL;DR: This article found that the audiovisual benefit for non-native speech is smaller than the benefit for native speech, an effect that may be partially accounted for by listeners' implicit racial biases.
Abstract: Speech intelligibility is improved when the listener can see the talker in addition to hearing their voice. Notably, though, previous work has suggested that this “audiovisual benefit” for nonnative (i.e., foreign-accented) speech is smaller than the benefit for native speech, an effect that may be partially accounted for by listeners’ implicit racial biases (Yi et al., 2013, The Journal of the Acoustical Society of America, 134[5], EL387–EL393.). In the present study, we sought to replicate these findings in a significantly larger sample of online participants. In a direct replication of Yi et al. (Experiment 1), we found that audiovisual benefit was indeed smaller for nonnative-accented relative to native-accented speech. However, our results did not support the conclusion that implicit racial biases, as measured with two types of implicit association tasks, were related to these differences in audiovisual benefit for native and nonnative speech. In a second experiment, we addressed a potential confound in the experimental design; to ensure that the difference in audiovisual benefit was caused by a difference in accent rather than a difference in overall intelligibility, we reversed the overall difficulty of each accent condition by presenting them at different signal-to-noise ratios. Even when native speech was presented at a much more difficult intelligibility level than nonnative speech, audiovisual benefit for nonnative speech remained poorer. In light of these findings, we discuss alternative explanations of reduced audiovisual benefit for nonnative speech, as well as methodological considerations for future work examining the intersection of social, cognitive, and linguistic processes.

3 citations

Journal ArticleDOI
TL;DR: Like the volumes of previous years, Volume 56/2009 of Experimental Psychology is characterized by both continuity and change, and one of the changes in 2009 refers to the team of editors.
Abstract: Like the volumes of previous years, Volume 56/2009 of Experimental Psychology is characterized by both continuity and change. One of the changes in 2009 refers to the team of editors. After serving as editors for several years, Ute J. Bayen, Anthony G. Greenwald, Glynn W. Humphreys, Dirk Kerzel, and Herbert Schriefers decided to step down upon completion of Volume 55. I would like to thank all of the outgoing editors for their excellent editorial work and their continuous support in the past years. Starting with Issue 1 of Volume 56, the outgoing editors will be replaced by seven new editors, namely, Arndt Broder (Rheinische Friedrich-Wilhelms University, Bonn, Germany), Roberto Dell’Acqua (University of Padova, Italy), Adele Diederich (Jacobs University, Bremen, Germany), Nachshon Meiran (Ben-Gurion University of the Negev, Beer-Sheva, Israel), Neil W. Mulligan (University of North Carolina, Chapel Hill, NC, USA), Manuel Perea (University of Valencia, Spain), and Melanie C. Steffens (FriedrichSchiller-University, Jena, Germany). I would like to thank these colleagues for their willingness to contribute to our common editorial work in the next years. In addition, I would also like to thank Iring Koch, Hartmut Leuthold, Thorsten Meiser, Jochen Musch, Klaus Rothermund, and Eva Walther for agreeing to stay in the team of editors. We are thus well prepared to manage the steadily increasing number of manuscript submissions to Experimental Psychology until the next major change in the team of editors in 2011. A second change concerns the editorial board. To cope with the wealth and the heterogeneity of topics addressed in research papers submitted to our journal, we decided to extend the editorial board of Experimental Psychology. I am grateful to Ute J. Bayen (Heinrich-Heine University, Dusseldorf, Germany), Hartmut Blank (University of Portsmouth, UK), Anthony G. Greenwald (University of Washington, Seattle, WA, USA), Ralph Hertwig (University of Basel, Switzerland), Dirk Kerzel (University of Geneva, Switzerland), Elizabeth F. Loftus (University of California, Irvine, CA, USA), Michel Regenwetter (University of Illinois at Urbana-Champaign, IL, USA), Jeffrey N. Rouder (University of Missouri-Columbia, Columbia, MO, USA), and Herbert Schriefers (Catholic University of Nijmegen, The Netherlands) for agreeing to supplement our editorial board in the next years. A third change concerns the book review section that will not be continued in 2009. Although Experimental Psychology switched from a quarterly to a bimonthly publication cycle in 2008, journal space continues to be scarce. To be able to publish accepted research articles as quickly as possible, we were forced to drop the book review section. This means that, beginning with the year 2009, new book review submissions will not be considered for publication anymore. Book reviews that have already been accepted will, of course, be published. The decision to drop the book review section was not easy. It has a long tradition both in Experimental Psychology and its predecessor journal, the Zeitschrift fur Experimentelle Psychologie. In recent years, Andreas Voss, our book review editor, did a great job in acquiring excellent book reviews addressing topics of high relevance for experimental psychologists. Nevertheless, given the current situation, we all agreed that the goal of reducing publication lag is more important than the continuation of the book review section. As with the other outgoing editors, I would also like to thank Andreas Voss for his excellent work as book review editor in the past years. Despite these major changes in the editorial team, the content and the scope of Experimental Psychology will not change. As Klauer (2002, p. 1) already stated in his first editorial of Experimental Psychology, ‘‘the scope of the journal is defined by the experimental method, and thus, papers based on experiments from all areas of psychology are welcome’’. We will stick to this strict definition in the future. By implication, papers comparing nonexperimental groups of participants (e.g., men and women, old and young people, amnesics and controls, participants with and without prosopagnosia, etc.) will not be considered for publication unless they report manipulations of at least one experimental independent variable (IV) in addition to the nonexperimental IV of interest. In other words, papers comparing experimental treatment effects between nonexperimental groups of participants would be appropriate in principle (e.g., experimental age group comparisons; cf. Carretti, Borella, & De Beni, 2007; Jansen-Osmann & Heil, 2007; Taconnat, Froger, Sacher, & Isingrini, 2008; Tsujimoto, Kuwajima, & Sawaguchi, 2007). We are also willing to consider correlational studies to some extent (see, e.g., Friese, Bluemke, & Wanke, 2007; Hilbig & Pohl, 2008; Nosek &

3 citations

Journal ArticleDOI
TL;DR: The authors investigated how implicit association test and self-report personality scores relate to each other and found that although extraversion and neuroticism selfreport and implicit measures are related, they do measure distinct constructs.

3 citations

Journal ArticleDOI
TL;DR: It is shown that both explicit and implicit attitudes towards free will are more positive than attitudes towards determinism, and that people experience cultural pressure to value free will, and to devalue determinism.
Abstract: Whether you believe free will exists has profound effects on your behaviour, across different levels of processing, from simple motor action to social cognition. It is therefore important to understand which specific lay theories are held in the general public and why. Past research largely focused on investigating free will beliefs (FWB, 'Do you think free will exists?'), but largely ignored a second key aspect: free will attitudes (FWA, 'Do you like/value will?'). Attitudes are often independently predictive of behaviour, relative to beliefs, yet we currently know very little about FWAs in the general public. One key issue is whether such attitudes are subject to biased, socially desirable responding. The vast majority of the general public strongly believes in the existence of free will, which might create cultural pressure to value free will positively as well. In this registered report, we used a very large (N = 1100), open available dataset measuring implicit and explicit attitudes towards free will and determinism to address this issue. Our results indicate that both explicit and implicit attitudes towards free will are more positive than attitudes towards determinism. We also show that people experience cultural pressure to value free will, and to devalue determinism. Yet, we found no strong evidence that this cultural pressure affected either implicit or explicit attitudes in this dataset.

3 citations

References
More filters
Journal ArticleDOI
TL;DR: In this paper, two types of error involved in fitting a model are considered, error of approximation and error of fit, where the first involves the fit of the model, and the second involves the model's shape.
Abstract: This article is concerned with measures of fit of a model. Two types of error involved in fitting a model are considered. The first is error of approximation which involves the fit of the model, wi...

25,611 citations

Journal ArticleDOI
TL;DR: This transmutability of the validation matrix argues for the comparisons within the heteromethod block as the most generally relevant validation data, and illustrates the potential interchangeability of trait and method components.
Abstract: Content Memory (Learning Ability) As Comprehension 82 Vocabulary Cs .30 ( ) .23 .31 ( ) .31 .31 .35 ( ) .29 .48 .35 .38 ( ) .30 .40 .47 .58 .48 ( ) As judged against these latter values, comprehension (.48) and vocabulary (.47), but not memory (.31), show some specific validity. This transmutability of the validation matrix argues for the comparisons within the heteromethod block as the most generally relevant validation data, and illustrates the potential interchangeability of trait and method components. Some of the correlations in Chi's (1937) prodigious study of halo effect in ratings are appropriate to a multitrait-multimethod matrix in which each rater might be regarded as representing a different method. While the published report does not make these available in detail because it employs averaged values, it is apparent from a comparison of his Tables IV and VIII that the ratings generally failed to meet the requirement that ratings of the same trait by different raters should correlate higher than ratings of different traits by the same rater. Validity is shown to the extent that of the correlations in the heteromethod block, those in the validity diagonal are higher than the average heteromethod-heterotrait values. A conspicuously unsuccessful multitrait-multimethod matrix is provided by Campbell (1953, 1956) for rating of the leadership behavior of officers by themselves and by their subordinates. Only one of 11 variables (Recognition Behavior) met the requirement of providing a validity diagonal value higher than any of the heterotrait-heteromethod values, that validity being .29. For none of the variables were the validities higher than heterotrait-monomethod values. A study of attitudes toward authority and nonauthority figures by Burwen and Campbell (1957) contains a complex multitrait-multimethod matrix, one symmetrical excerpt from which is shown in Table 6. Method variance was strong for most of the procedures in this study. Where validity was found, it was primarily at the level of validity diagonal values higher than heterotrait-heteromethod values. As illustrated in Table 6, attitude toward father showed this kind of validity, as did attitude toward peers to a lesser degree. Attitude toward boss showed no validity. There was no evidence of a generalized attitude toward authority which would include father and boss, although such values as the VALIDATION BY THE MULTITRAIT-MULTIMETHOD MATRIX

15,795 citations

Journal ArticleDOI
TL;DR: The present interpretation of construct validity is not "official" and deals with some areas where the Committee would probably not be unanimous, but the present writers are solely responsible for this attempt to explain the concept and elaborate its implications.
Abstract: Validation of psychological tests has not yet been adequately conceptualized, as the APA Committee on Psychological Tests learned when it undertook (1950-54) to specify what qualities should be investigated before a test is published. In order to make coherent recommendations the Committee found it necessary to distinguish four types of validity, established by different types of research and requiring different interpretation. The chief innovation in the Committee's report was the term construct validity.[2] This idea was first formulated by a subcommittee (Meehl and R. C. Challman) studying how proposed recommendations would apply to projective techniques, and later modified and clarified by the entire Committee (Bordin, Challman, Conrad, Humphreys, Super, and the present writers). The statements agreed upon by the Committee (and by committees of two other associations) were published in the Technical Recommendations (59). The present interpretation of construct validity is not "official" and deals with some areas where the Committee would probably not be unanimous. The present writers are solely responsible for this attempt to explain the concept and elaborate its implications.

9,935 citations

Journal ArticleDOI
TL;DR: An implicit association test (IAT) measures differential association of 2 target concepts with an attribute when instructions oblige highly associated categories to share a response key, and performance is faster than when less associated categories share a key.
Abstract: An implicit association test (IAT) measures differential association of 2 target concepts with an attribute. The 2 concepts appear in a 2-choice task (e.g., flower vs. insect names), and the attribute in a 2nd task (e.g., pleasant vs. unpleasant words for an evaluation attribute). When instructions oblige highly associated categories (e.g., flower + pleasant) to share a response key, performance is faster than when less associated categories (e.g., insect + pleasant) share a key. This performance difference implicitly measures differential association of the 2 concepts with the attribute. In 3 experiments, the IAT was sensitive to (a) near-universal evaluative differences (e.g., flower vs. insect), (b) expected individual differences in evaluative associations (Japanese + pleasant vs. Korean + pleasant for Japanese vs. Korean subjects), and (c) consciously disavowed evaluative differences (Black + pleasant vs. White + pleasant for self-described unprejudiced White subjects).

9,731 citations

Journal ArticleDOI
TL;DR: In this article, a framework for hypothesis testing and power analysis in the assessment of fit of covariance structure models is presented, where the value of confidence intervals for fit indices is emphasized.
Abstract: A framework for hypothesis testing and power analysis in the assessment of fit of covariance structure models is presented. We emphasize the value of confidence intervals for fit indices, and we stress the relationship of confidence intervals to a framework for hypothesis testing. The approach allows for testing null hypotheses of not-good fit, reversing the role of the null hypothesis in conventional tests of model fit, so that a significant result provides strong support for good fit. The approach also allows for direct estimation of power, where effect size is defined in terms of a null and alternative value of the root-mean-square error of approximation fit index proposed by J. H. Steiger and J. M. Lind (1980). It is also feasible to determine minimum sample size required to achieve a given level of power for any test of fit in this framework. Computer programs and examples are provided for power analyses and calculation of minimum sample sizes.

8,401 citations

Frequently Asked Questions (1)
Q1. What are the contributions in "A multitrait-multimethod validation of the implicit association test implicit and explicit attitudes are related but distinct constructs" ?

This provides a basis for, but does not distinguish between, dual-process and dual-representation theories that account for the distinctions between constructs.