Confidence Limits for the Indirect Effect: Distribution of the Product and Resampling Methods.
Summary (3 min read)
Estimation of the Indirect Effect and Standard Error
- The indirect effect model is shown in Figure 1 and is summarized in the three equations described below (see also Allison, 1995a and MacKinnon & Dwyer, 1993) .
- The residuals have expected values of zero.
- Two extensive simulation studies (MacKinnon et al., 1995; Stone & Sobel, 1990) showed an imbalance in the number of times a true value fell to the left or right of the confidence limits.
The Distribution of the Product
- The assumption that the indirect effect divided by its standard error has a normal sampling distribution is incorrect in some situations.
- In these situations, the confidence limits calculated using Equation 5 will be incorrect.
- Because the indirect effect is the product of regression estimates which are normally distributed asymptotically (Hanushek & Jackson, 1977) , an alternative method for testing indirect effects can be developed based on the distribution of the product of two normally distributed random variables (Aroian, 1947; Craig, 1936; Springer, 1979) .
Simulation Description
- The SAS ® (1989) programming language was used to conduct the statistical simulations.
- Third, one thousand replications were conducted for each of the 40 combinations of sample size and parameters.
- Fourth, for each of the 40,000 (4 combinations of sample size times 10 parameter value combinations times 1000 replications) different data sets, six resampling methods were applied.
Confidence Limits
- Sample values were inserted in Equation 5to obtain upper and lower confidence limits for the z test.
- For the bootstrap methods, the confidence limits were obtained from the bootstrap distribution.
- Confidence limits for the indirect effect were calculated for 80%, 90%, and 95% intervals.
- Table 4 summarizes the performance of the confidence limits by showing the number of times that the observed percentage was outside the robustness interval for each of the three confidence intervals and nine methods 3 .
- All of these methods were considerably better than the jackknife (59 times) which only had slightly better performance than the traditional z test (61 times).
Results
- Confidence limits were calculated for the z and M tests.
- The proportions of times that true values of the indirect effect fell to the left and right of the confidence limits are shown in Tables 1 and 2 .
- The distribution of the product, M, has more balanced confidence limits because it incorporates the skewness and kurtosis of the product distribution.
- Most importantly, note that the confidence limits based on the distribution of the product are nearly always as close or closer to the expected Type I error rate of .025 than the traditional test.
Discussion
- The confidence limits for the indirect effect based on the distribution of the product were more accurate than the confidence limits based on the normal distribution assumption.
- The Type I error rates based on the confidence limits did not exceed nominal rates using this method for any combination of parameter values both in Study 1 1 and in MacKinnon et al. (2002) .
- The proportions outside the confidence limits for the product distribution were often less than the expected values for small effect sizes and small sample sizes (i.e., small values of ␦ ␣ and ␦  ).
- One possible explanation for this discrepancy is that the appropriate comparison distribution is the product of two t distributions rather than two normal distributions.
- The fourth group consists of the bias-corrected bootstrap which had slightly more power than methods in the second category and had the most accurate confidence intervals.
Study 2
- In Study 1, although confidence limits for the indirect effect were more accurate when the distribution of the product was taken into account, there were still cases where the number of times that the true value was outside the range of the confidence limits was smaller than expected.
- Several researchers have suggested that resampling methods such as the jackknife and the bootstrap may provide more accurate tests of the indirect effect (Bollen & Stine, 1990; Lockwood & MacKinnon, 1998; Shrout & Bolger, 2002) .
- Bollen and Stine (1990) found that bootstrap confidence limits for the indirect effect were asymmetric.
- Most recently, Shrout and Bolger (2002) recommended bootstrap methods to assess mediation for small to moderate sample sizes.
Type I Error Rates and Statistical Power
- The observed Type I error rates and statistical power were also computed for each method.
- An effect was considered statistically significant if zero was not included in the confidence interval.
- For Type I error rates, the liberal Bradley (1978) robustness interval was also computed and Type I error rates outside the interval are indicated by an asterisk in the Tables.
Single Sample Methods
- The calculation of the M test confidence limits was also the same as reported in Study 1 with one minor exception.
- The critical values for the M test confidence limits in Study 2 come from an augmented table for the 95% confidence limits.
- These additional values were obtained with a FORTRAN algorithm written by Alan Miller which is a minor modification of the method in Meeker and Escobar (1994) and is available at http://users.bigpond.net.au/amiller (file name: fnprod.f90).
- This method is called the empirical-M method in this article.
- The values were standardized so that they could be used for any sample size.
Resampling Methods
- Six resampling methods were evaluated in this study: jackknife, percentile bootstrap, bias-corrected bootstrap, bootstrap-t, bootstrap-Q, and Monte Carlo.
- All of the methods adjust for nonnormal distributions although the bootstrap-Q and the bias-corrected bootstrap may be especially appropriate for severely nonnormal data (Chernick, 1999; Manly, 1997) .
- The jackknife estimate is the average estimate across the N jackknife samples.
- The basic bootstrap confidence limits were obtained with the percentile method as described by Efron and Tibshirani (1993) .
- It requires the standard error of the parameter estimate for each bootstrap sample which is the sampling standard deviation of the bootstrap sample.
Example
- The following example illustrates the methods used in this article with data from the Adolescents Training and Learning to Avoid Steroids program.
- The data for this simplified example were from 861 cases (from 15 treatment schools and 16 control schools) with complete data on three variables, X-exposure to the program or not, X M -perceived severity of anabolic steroid use, and Y-nutrition behaviors.
- Confidence limits for the indirect effect were computed for three single sample tests: the traditional z, the M test, and the empirical-M test, as well as six resampling methods: the jackknife, percentile bootstrap, bootstrap-t, bootstrap-Q, bias-corrected bootstrap, and the Monte Carlo method.
- These values were used in Equations 6 and 7 to find the upper and lower M test confidence limits.
General Discussion
- The purpose of this article was to evaluate two alternatives to improve confidence limit coverage for the indirect effect.
- In Study 2, resampling methods had better performance than the method based on the normal distribution, with the exception of the jackknife.
- There are limitations to the use of resampling methods, however.
- The practical implication of the results of this article is that the traditional z test confidence limits can be substantially improved by using a method such as the M test that incorporates the distribution of the product of two normal random variables.
Did you find this useful? Give us your feedback
Citations
25,799 citations
Cites background or methods from "Confidence Limits for the Indirect ..."
...…bases inference on a mathematical derivation of the distribution of the product of two normally distributed variables (Aroian, 1947; Craig, 1936; MacKinnon et al., 2004; Springer, 1979) and thus acknowledges the skew of the distribution of products rather than imposing the assumption of…...
[...]
...In extensive sets of simulations, MacKinnon et al. (2002; MacKinnon et al., 2004) examined the performance of these methods (among others) to assess their Type I error rates and power....
[...]
...For details of its application to simple mediation models, see Bollen and Stine (1990), Lockwood and MacKinnon (1998), MacKinnon et al. (2004), Shrout and Bolger (2002), and Preacher and Hayes (2004, 2008)....
[...]
7,973 citations
Cites background or methods from "Confidence Limits for the Indirect ..."
...The distribution of the product strategy is probably the most accurate analytic method available for determining the significance of, and confidence intervals (CIs) for, a1b1 in simple mediation models (MacKinnon et al., 2004)....
[...]
...Some research has been undertaken with respect to the power of tests of simple mediation (MacKinnon et al., 2004), and our own simulation addressed power under a set of limited conditions....
[...]
...A growing literature now advocates the use of bootstrapping for assessing indirect effects (Bollen & Stine, 1990; Lockwood & MacKinnon, 1998; MacKinnon et al., 2004; Preacher & Hayes, 2004; Shrout & Bolger, 2002)....
[...]
...MacKinnon et al. (2004) showed that such corrections can improve CIs and inferences when used in the context of simple mediation models....
[...]
...Bootstrapping has been advocated as an alternative to normal-theory tests of mediation (Lockwood & MacKinnon, 1998; MacKinnon et al., 2004; Preacher & Hayes, 2004; Shrout & Bolger, 2002)....
[...]
7,914 citations
Cites methods from "Confidence Limits for the Indirect ..."
...Simulation research shows that bootstrapping is one of the more valid and powerful methods for testing intervening variable effects (MacKinnon et al., 2004; Williams & MacKinnon, 2008) and, for this reason alone, it should be the method of choice....
[...]
3,624 citations
Cites methods from "Confidence Limits for the Indirect ..."
...The bootstrap has been used to test indirect effects in mediated models ( MacKinnon, Lockwood, & Williams, 2004; Shrout & Bolger, 2002) and can be extended to models that combine mediation and moderation, as we later illustrate....
[...]
3,165 citations
Cites background or methods from "Confidence Limits for the Indirect ..."
...A word of caution is needed for the bias-corrected bootstrap test, however, as it has been found to have larger-than-normal Type I error rates in certain conditions (see MacKinnon et al., 2004, for more information)....
[...]
...Aword of caution is needed for the bias-corrected bootstrap test, however, as it has been found to have larger-than-normal Type I error rates in certain conditions (see MacKinnon et al., 2004, for more information)....
[...]
...Initial sample sizes were estimated using results from MacKinnon et al. (2002, 2004) ....
[...]
...MacKinnon et al. (2004) compared the confidence limits for the indirect effect, ab, from MacKinnon and Lockwood’s (2001) asymmetric confidence-interval test with more traditional symmetric confidence intervals and with confidence intervals from six resampling methods....
[...]
... MacKinnon et al. (2004) compared the confidence limits for the indirect effect, ab, from MacKinnon and Lockwood’s (2001) asymmetric confidence-interval test with more traditional symmetric confidence intervals and with confidence intervals from six resampling methods....
[...]
References
115,069 citations
80,095 citations
"Confidence Limits for the Indirect ..." refers background or methods in this paper
...These include the steps mentioned in Baron and Kenny (1986) and Judd and Kenny (1981) and the joint significance test of and described in MacKinnon et al. (2002), which do not include explicit methods to compute confidence limits....
[...]
...Hypotheses regarding indirect or mediated effects are implicit in social science theories (Alwin & Hauser, 1975; Baron & Kenny, 1986; Hyman, 1955; Sobel, 1982)....
[...]
37,183 citations
26,683 citations
"Confidence Limits for the Indirect ..." refers background in this paper
...Examples of indirect effect hypotheses are that attitudes affect intentions which then affect behavior (Ajzen & Fishbein, 1980), that poverty reduces local social ties which increases assault and burglary rates (Warner & Rountree, 1997), that social status has an indirect effect on depression…...
[...]
20,904 citations