scispace - formally typeset
Search or ask a question
Book

Methods of Meta-Analysis: Correcting Error and Bias in Research Findings

TL;DR: In this article, the authors present a meta-analysis of Artifact Distributions and their impact on study outcomes. But they focus mainly on the second-order sampling error and related issues.
Abstract: PART ONE: INTRODUCTION TO META-ANALYSIS Integrating Research Findings Across Studies Study Artifacts and Their Impact on Study Outcomes PART TWO: META-ANALYSIS OF CORRELATIONS Meta-Analysis of Correlations Corrected Individually for Artifacts Meta-Analysis of Correlations Using Artifact Distributions Technical Questions in Meta-Analysis of Correlations PART THREE: META-ANALYSIS OF EXPERIMENTAL EFFECTS AND OTHER DICHOTOMOUS COMPARISONS Treatment Effects Experimental Artifacts and Their Impact Meta-Analysis Methods for d Values Technical Questions in Meta-Analysis of d Values PART FOUR: GENERAL ISSUES IN META-ANALYSIS Second Order Sampling Error and Related Issues Cumulation of Findings within Studies Methods of Integrating Findings Across Studies Locating, Selecting, and Evaluating Studies General Criticisms of Meta-Analysis Summary of Psychometric Meta-Analysis
Citations
More filters
Journal ArticleDOI
TL;DR: The four articles in this special section onMeta-analysis illustrate some of the complexities entailed in meta-analysis methods and contributes both to advancing this methodology and to the increasing complexities that can befuddle researchers.
Abstract: During the past 30 years, meta-analysis has been an indispensable tool for revealing the hidden meaning of our research literatures. The four articles in this special section on meta-analysis illus...

20,272 citations

Journal ArticleDOI
TL;DR: The metafor package provides functions for conducting meta-analyses in R and includes functions for fitting the meta-analytic fixed- and random-effects models and allows for the inclusion of moderators variables (study-level covariates) in these models.
Abstract: The metafor package provides functions for conducting meta-analyses in R. The package includes functions for fitting the meta-analytic fixed- and random-effects models and allows for the inclusion of moderators variables (study-level covariates) in these models. Meta-regression analyses with continuous and categorical moderators can be conducted in this way. Functions for the Mantel-Haenszel and Peto's one-step method for meta-analyses of 2 x 2 table data are also available. Finally, the package provides various plot functions (for example, for forest, funnel, and radial plots) and functions for assessing the model fit, for obtaining case diagnostics, and for tests of publication bias.

11,237 citations

Journal ArticleDOI
TL;DR: In this article, the authors investigated the relation of the Big Five personality dimensions (extraversion, emotional stability, Agreeableness, Conscientiousness, and Openness to Experience) to three job performance criteria (job proficiency, training proficiency, and personnel data) for five occupational groups (professionals, police, managers, sales, and skilled/semi-skilled).
Abstract: This study investigated the relation of the “Big Five” personality dimensions (Extraversion, Emotional Stability, Agreeableness, Conscientiousness, and Openness to Experience) to three job performance criteria (job proficiency, training proficiency, and personnel data) for five occupational groups (professionals, police, managers, sales, and skilled/semi-skilled). Results indicated that one dimension of personality, Conscientiousness, showed consistent relations with all job performance criteria for all occupational groups. For the remaining personality dimensions, the estimated true score correlations varied by occupational group and criterion type. Extraversion was a valid predictor for two occupations involving social interaction, managers and sales (across criterion types). Also, both Openness to Experience and Extraversion were valid predictors of the training proficiency criterion (across occupations). Other personality dimensions were also found to be valid predictors for some occupations and some criterion types, but the magnitude of the estimated true score correlations was small (ρ < .10). Overall, the results illustrate the benefits of using the 5-factor model of personality to accumulate and communicate empirical findings. The findings have numerous implications for research and practice in personnel psychology, especially in the subfields of personnel selection, training and development, and performance appraisal.

8,018 citations

Journal ArticleDOI
TL;DR: In this article, KlUGER and Denisi analyzed all the major reasons to reject a paper from the meta-analysis, even though the decision to exclude a paper came at the first identification of a missing inclusion criterion.
Abstract: the total number of papers may exceed 10,000. Nevertheless, cost consideration forced us to consider mostly published papers and technical reports in English. 4 Formula 4 in Seifert (1991) is in error—a multiplier of n, of cell size, is missing in the numerator. 5 Unfortunately, the technique of meta-analysis cannot be applied, at present time, to such effects because the distribution of dis based on a sampling of people, whereas the statistics of techniques such as ARIMA are based on the distribution of a sampling of observations in the time domain regardless of the size of the people sample involved (i.e., there is no way to compare a sample of 100 points in time with a sample of 100 people). That is, a sample of 100 points in time has the same degrees of freedom if it were based on an observation of 1 person or of 1,000 people. 258 KLUGER AND DENISI From the papers we reviewed, only 131 (5%) met the criteria for inclusion. We were concerned that, given the small percentage of usable papers, our conclusions might not fairly represent the larger body of relevant literature. Therefore, we analyzed all the major reasons to reject a paper from the meta-analysis, even though the decision to exclude a paper came at the first identification of a missing inclusion criterion. This analysis showed the presence of review articles, interventions of natural feedback removal, and papers that merely discuss feedback, which in turn suggests that the included studies represent 1015% of the empirical FI literature. However, this analysis also showed that approximately 37% of the papers we considered manipulated feedback without a control group and that 16% reported confounded treatments, that is, roughly two thirds of the empirical FI literature cannot shed light on the question of FI effects on performance—a fact that requires attention from future FI researchers. Of the usable 131 papers (see references with asterisks), 607 effect sizes were extracted. These effects were based on 12,652 participants and 23,663 observations (reflecting multiple observations per participant). The average sample size per effect was 39 participants. The distribution of the effect sizes is presented in Figure 1. The weighted mean (weighted by sample size) of this distribution is 0.41, suggesting that, on average, FI has a moderate positive effect on performance. However, over 38% of the effects were negative (see Figure 1). The weighted variance of this distribution is 0.97, whereas the estimate of the sampling error variance is only 0.09. A potential problem in meta-analyses is a violation of the assumption of independence. Such a violation occurs either when multiple observations are taken from the same study (Rosenthal, 1984) or when several papers are authored by the same person (Wolf, 1986). In the present investigation, there were 91 effects derived from the laboratory experiments reported by Mikulincer (e.g., 1988a, 1988b). This raises the possibility that the average effect size is biased, because his studies manipulated extreme negative FIs and used similar tasks. In fact, the weighted average d in Mikulincer's studies was —0.39; whereas in the remainder of the

5,126 citations

Journal ArticleDOI
TL;DR: Motivated performance tasks elicited cortisol responses if they were uncontrollable or characterized by social-evaluative threat (task performance could be negatively judged by others), when methodological factors and other stressor characteristics were controlled for.
Abstract: This meta-analysis reviews 208 laboratory studies of acute psychological stressors and tests a theoretical model delineating conditions capable of eliciting cortisol responses. Psychological stressors increased cortisol levels; however, effects varied widely across tasks. Consistent with the theoretical model, motivated performance tasks elicited cortisol responses if they were uncontrollable or characterized by social-evaluative threat (task performance could be negatively judged by others), when methodological factors and other stressor characteristics were controlled for. Tasks containing both uncontrollable and social-evaluative elements were associated with the largest cortisol and adrenocorticotropin hormone changes and the longest times to recovery. These findings are consistent with the animal literature on the physiological effects of uncontrollable social threat and contradict the belief that cortisol is responsive to all types of stressors.

5,028 citations