A multitrait-multimethod validation of the Implicit Association Test: implicit and explicit attitudes are related but distinct constructs.
Abstract: Recent theoretical and methodological innovations suggest a distinction between implicit and explicit evaluations. We applied Campbell and Fiske's (1959) classic multitrait-multimethod design precepts to test the construct validity of implicit attitudes as measured by the Implicit Association Test (IAT). Participants (N = 287) were measured on both self-report and IAT for up to seven attitude domains. Through a sequence of latent-variable structural models, systematic method variance was distinguished from attitude variance, and a correlated two-factors-per-attitude model (implicit and explicit factors) was superior to a single-factor-per-attitude specification. That is, despite sometimes strong relations between implicit and explicit attitude factors, collapsing their indicators into a single attitude factor resulted in relatively inferior model fit. We conclude that these implicit and explicit measures assess related but distinct attitude constructs. This provides a basis for, but does not distinguish between, dual-process and dual-representation theories that account for the distinctions between constructs.
Summary (5 min read)
- Also, this multitrait-multimethod (MTMM) approach does not identify the cognitive processes that distinguish the constructs.
- The context of this analysis follows from Cronbach and Meehl’s (1955) classic discussion of construct validation where a construct is an indeterminant function of representation and process.
- The authors research provides another avenue of evidence for this growing nomological net by examining the relationship between implicit and explicit attitude measures to determine whether they can be fairly interpreted as measuring a single construct, or whether they assess related, but distinct constructs.
- Greenwald and Farnham (2000) observed that a model describing implicit and explicit self-esteem as distinct latent factors provided a better fit than a single self-esteem conceptualization.
- Likewise, Cunningham, Preacher, and Banaji (2001) found implicit and explicit measures of racial attitudes to reveal related, but distinct factors, as did Cunningham, Netlek, and Banaji (2004) for implicit and explicit ethnocentrism.
- Following this approach of comparing single versus dual factors in structural equation modeling, the authors reanalyzed a large dataset reported by Nosek (2005).
- To transcend this inferential limitation, here, guided by principles articulated by Campbell and Fiske (1959), the authors use a MTMM design and comparative structural modeling analyses.
- The authors findings demonstrate (1) convergent and discriminant validity of the IAT, (2) that a model of distinct, but related implicit and explicit attitudes best fits the data, and (3) that this characterization is not attributable to attitude-irrelevant method variance of the IAT or of self-report.
MTMM and Confirmatory Factor Analysis
- In their classic article on construct validation, Campbell and Fiske (1959) articulated a strategy for using MTMM matrices to evaluate convergent and discriminant validity.
- Campbell and Fiske argued that “the clear-cut demonstration of the presence of method variance requires both several traits and several methods” (p. 85).
- Confirmatory factor analysis (CFA) has emerged as a tool well-suited to the partitioning of MTMM data envisioned by Campbell and Fiske (Jöreskog, 1974; Widaman, 1985).
- 2 Cunningham, Nezlek et al. (2004) footnoted this limitation, but argued that, since a control IAT, birds vs. trees, did not load with five ingroup-outgroup IATs on an implicit ethnocentrism factor, IAT method variance was not a strong driver of the two-factor solution.
- By comparing the fits of nested structural models, the relative merits of alternative hypotheses concerning the structure of trait and method variance can be systematically tested (Jöreskog & Sörbom, 1979; Loehlin, 2004; McDonald, 1985).
- Data are from four laboratory studies in which attitudes toward seven different attitude-object pairs were measured: flowers–insects, Democrats–Republicans, humanities–science, straight–gay, Whites–Blacks, creationism–evolution, and thin people–fat people.
- These domains were selected because on their face they cover a broad range of attitudes.
- They reported support for this idea and also found that implicit and explicit ethnocentrism factors were related, but distinct.
- The authors sought to demonstrate discriminant validity between attitude domains – hypothesizing that the attitude domains would form distinct factors; and convergent validity across measurement types (IAT and selfreport) – hypothesizing that the implicit and explicit attitude constructs would be related, but retain distinctiveness not accounted for by method factors.
- This simultaneous examination of discriminant and convergent validity is the core value of the MTMM approach.
Implicit Association Test (IAT)
- One of the four samples received IATs for all seven object pairs, while the others received subsets of at least four pairs, including the flowers–insects and Democrats–Republicans pairs .
- For the remaining IATs, response blocks for all tasks were randomized and single-discrimination practice blocks were eliminated.
- The authors allow substantial opportunity for method factors to influence IAT performance and challenge the hypothesis that distinguishable attitude factors can be identified despite intermixed performance blocks.
- Using four indicators was preferable, however, in terms of attaining stable estimates in the more complex models.
- Participants reported attitudes toward each of the target objects per pair independently using two 9-point semantic differentials.
- Anchors for these differentials varied across data collections, including, for a given study, two of the following four pairs: cold–warm, unpleasant–pleasant, bad–good, or unfavorable–favorable.
- Positive values indicate greater liking for the object that was implicitly preferred on average (–8 to +8).
- Similar procedures were used across all four data collections.
- After informed consent, participants completed a selection of IATs and self-report measures.
- The order of implicit and explicit measures was counterbalanced between subjects.
- The correlation matrices are substantively similar when each data collection is considered independently.
- Following guidelines suggested by Campbell and Fiske (1959), the authors first describe and offer interpretations of the reliability and validity relations in the MTMM matrix.
- Thus, if the 95% CI for the εaΔ includes .05, the models being compared are considered close to one another in fit, and one would be preferred only to the extent that it involves fewer parameter estimates (i.e., is more parsimonious).
- In Model 1 the authors specify two method factors to account for the covariances among the 42 observed indicators (14 explicit and 28 implicit) across all seven general attitude domains.
- This is accomplished by specifying models that are identical to Model 3 except that, in turn, a common implicit method factor is not specified (Model 4) and a common explicit method factor is not specified (Model 5).
- By comparing the fits of each with that of Model 3, the authors may discern to what extent accounting for common method variance is important to understanding the structure of relations among these measures.
Description of the MTMM Correlations
- Table 2 is a MTMM matrix for the two methods and seven attitude object pairs.
- The first three rows of the table provide descriptive statistics for each of the fourteen measurements (i.e., full IAT D scores and averages of the self-reports).
- Reliability estimates (Cronbach’s α) are shown in parentheses along the main diagonal in the top-left (IAT) and bottom-right (self-report) panels, and intramethod (Campbell and Fiske’s monomethod-heterotrait) correlations are listed in the other cells of these respective panels.
- For the IAT, reliabilities are based on D scores for split-halves formed from alternating couplets of trials, since a couplet consisting of an object stimulus (e.g., a science word) and an evaluative stimulus (e.g., a “good” word) occurred every two trials.
- That is, little evidence of common method variance is apparent across attitude domains.
- Correlations in the gray diagonal of the bottom-left panel can be used to assess convergent validity (monotrait-heteromethod), and discriminant validity (heterotrait-heteromethod) can be assessed by those off the diagonal in this panel.
- These data are consistent with their hypotheses for the convergent and discriminant validity of the IAT and self-report across attitude domains.
- The power of the MTMM design is not fully harnessed by scrutinizing correlation matrices.
- The relative merits of competing hypotheses about the structure of the data can be tested formally by comparing the fits of nested, but differentially specified, structural equation models.
MTMM Structural Equation Models
- Summary statistics from the confirmatory structural models are listed in Table 3 (details of each model’s specifications and parameter estimates – all fit with Mplus statistical software (Muthén & Muthén, 1998–2004) – are available in the supplement to this paper at http://briannosek.com/).
- In Model 1, the first of substantive interest , two method factors are specified, one loading on the explicit indicators and one on the implicit.
- Comparing this model’s fit with that of Model 2 tests whether specifying distinct implicit and explicit attitude factors is superior to a single-attitude factor per domain model for these data.
- To summarize, there was relatively little common method variance to account for in these data; statistically significant, but small amounts were found for both the explicit and implicit measurements.
- The authors conducted additional analyses to evaluate the possibility that method variance is underestimated in these models because both the IAT and the explicit measures have rational zero points; both indicate a preference for one category compared to another .
- Some factors may primarily influence the extremity of the score away from neutrality (0).
- People who are more skilled at task-switching will achieve less extreme scores regardless of whether, for example, they are proDemocrat or pro-Republican.
- This provides a liberal test for the method factor influences because it reduces the constructvalid variance by treating positive and negative score values as the same, and enhances the opportunity to see influences of extremity (distance from 0) as indicating common influence on the implicit or explicit measures.
- Refitting the sequence of models summarized in Table 3 with the absolute values for each indicator yielded the same pattern of results (a table in which these results are listed is part of the online supplement, http://briannosek.com/).
- The results of this study add to construct validation evidence for the IAT as a measure of attitudes (Greenwald & Nosek, 2001).
- The convergent validity of the IAT was evidenced by significant factor correlations between the implicit and explicit attitude constructs in five of the seven attitude domains, while its discriminant validity was simultaneously evidenced by the statistical superiority of the two-attitude model to the single-attitude model.
- There was a modest amount of common method variance to account for, with statistically significant but small portions isolated from both the explicit and implicit measures, and this was also observed when absolute values of the indicators were used.
- As Campbell and Fiske (1959, p. 84) observed, “In any given psychological measuring device, there are certain features or stimuli introduced specifically to represent the trait that it is intended to measure.
- The MTMM design, coupled with comparative structural modeling, allowed common implicit method variance to be distinguished from implicit attitude variance.
Other Components of the Nomological Net for the Implicit Attitude Construct
- This research provides a basis for some key components of validation of the IAT and implicit attitudes.
- If the IAT reflected evaluations of the stimulus items and not the categories, then the implicit–explicit distinction might be explained by this difference.
- The effort required for the chemist to specify the snowboarder’s constructs in terms of a single H20 construct would produce ungainly process theories that would likely “miss the point” of the snow and ice constructs.
- In short, the present evidence for distinct implicit and explicit attitude constructs does not rule out the possibility that the two constructs derive from common evaluative content.
- Convergent evidence across a variety of research programs suggests that the IAT is a valid measure of attitudes (see Nosek et al., 2006, for a review).
- Like other methods such as semantic differentials, Likert scales, sequential priming, and the Stroop task, the IAT can be adapted to measure evaluations of many types of social categories.
- The cumulative evidence identifies design factors that will influence the method’s validity, and provides a nomological net of knowledge to accelerate validation of novel applications of the IAT.
- The emergence of the implicit attitude construct has spurred investigations to test the strength and limitations of this concept and its measurement tools, like the IAT.
- The authors found simultaneous evidence of convergent and discriminant validity of the IAT and self-report as measures of related but distinct attitude constructs, and as distinct from methodological variation.
Did you find this useful? Give us your feedback
Related Papers (5)
Frequently Asked Questions (1)
Q1. What are the contributions in "A multitrait-multimethod validation of the implicit association test implicit and explicit attitudes are related but distinct constructs" ?
This provides a basis for, but does not distinguish between, dual-process and dual-representation theories that account for the distinctions between constructs.