This study used meta-analytic methods to compare the interrater and intrarater reliabilities of ratings of 10 dimensions of job performance used in the literature; ratings of overall job performance were also examined. There was mixed support for the notion that some dimensions are rated more reliably than others. Supervisory ratings appear to have higher interrater reliability than peer ratings. Consistent with H. R. Rothstein (1990), mean interrater reliability of supervisory ratings of overall job performance was found to be .52. In all cases, interrater reliability is lower than intrarater reliability, indicating that the inappropriate use of intrarater reliability estimates to correct for biases from measurement error leads to biased research results. These findings have important implications for both research and practice.

Comparative analysis of the reliability of job performance ratings

A meta-analysis of studies containing both objective and subjective ratings of employee performance resulted in a corrected mean correlation of .389. This value, although significantly greater than zero, indicates that objective and subjective performance measures should not be used interchangeably. Moreover, in no moderator subgroup examined did the correlation suggest convergent validity. After discussing issues related to resolving the previous anomalies of primary and meta-analytic results, a secondary analysis suggested that objective and subjective measures of the same construct at the same level may be used inter-changeably. The secondary analysis, however, was based on a very limited sample. Future research should address the appropriate dimensionality of employee performance.

On the interchangeability of objective and subjective measures of employee performance: a meta-analysis

A database integrating 90 years of empirical studies reporting intercorrelations among rated job performance dimensions was used to test the hypothesis of a general factor in job performance. After controlling for halo error and 3 other sources of measurement error, there remained a general factor in job performance ratings at the construct level accounting for 60% of total variance. Construct-level correlations among rated dimensions of job performance were substantially inflated by halo for both supervisory (33%) and peer (63%) intrarater correlations. These findings have important implications for the measurement of job performance and for theories of job performance.

Is there a general factor in ratings of job performance? A meta-analytic framework for disentangling substantive and error influences.

This paper presents a theory of learned helplessness to account for passive maladaptive behavior in organizations. The essential hypothesis is that the properties of formal organizations often inadvertently condition employee failure and that this behavior frequently continues even after environmental changes make success possible. A model describing the intrapersonal process of becoming helpless is developed. Particular attention is devoted to the description of strategies for minimizing organizationally induced helplessness.

Learned Helplessness: An Alternative Explanation for Performance Deficits

In several social perception studies investigators have concluded that raters' semantic conceptual similarity schemata serve to guide and constrain dimensional covariance in the rating judgment process. This effect has been hypothesized to be most likely when ratings are memory based and raters lack relevant job or ratee information. Recent research that has explored the effects of conceptual similarity schemata on performance ratings and halo error has provided some limited support for this systematic distortion hypothesis (SDH). However, these studies are limited because researchers have examined this phenomena using group-level analyses, whereas the theory references individual-level judgment processes. The present study investigated the phenomena at the individual level. The effects of varying levels of rater job knowledge (high, medium, and low) and familiarity with ratees (high and low) were examined for conceptual similarity-rating and rating-true-score covariation relations, for measures of halo, and for rating accuracy components. Results provided support for the SDH, but indicated a boundary condition for its operation and revealed some surprising findings for individual-level rater halo.

Jack E. McClary

Papers

Relationships among ratings, production, efficiency, and the General Aptitude Test Battery scales in an industrial setting.