the total number of papers may exceed 10,000. Nevertheless, cost consideration forced us to consider mostly published papers and technical reports in English. 4 Formula 4 in Seifert (1991) is in error—a multiplier of n, of cell size, is missing in the numerator. 5 Unfortunately, the technique of meta-analysis cannot be applied, at present time, to such effects because the distribution of dis based on a sampling of people, whereas the statistics of techniques such as ARIMA are based on the distribution of a sampling of observations in the time domain regardless of the size of the people sample involved (i.e., there is no way to compare a sample of 100 points in time with a sample of 100 people). That is, a sample of 100 points in time has the same degrees of freedom if it were based on an observation of 1 person or of 1,000 people. 258 KLUGER AND DENISI From the papers we reviewed, only 131 (5%) met the criteria for inclusion. We were concerned that, given the small percentage of usable papers, our conclusions might not fairly represent the larger body of relevant literature. Therefore, we analyzed all the major reasons to reject a paper from the meta-analysis, even though the decision to exclude a paper came at the first identification of a missing inclusion criterion. This analysis showed the presence of review articles, interventions of natural feedback removal, and papers that merely discuss feedback, which in turn suggests that the included studies represent 1015% of the empirical FI literature. However, this analysis also showed that approximately 37% of the papers we considered manipulated feedback without a control group and that 16% reported confounded treatments, that is, roughly two thirds of the empirical FI literature cannot shed light on the question of FI effects on performance—a fact that requires attention from future FI researchers. Of the usable 131 papers (see references with asterisks), 607 effect sizes were extracted. These effects were based on 12,652 participants and 23,663 observations (reflecting multiple observations per participant). The average sample size per effect was 39 participants. The distribution of the effect sizes is presented in Figure 1. The weighted mean (weighted by sample size) of this distribution is 0.41, suggesting that, on average, FI has a moderate positive effect on performance. However, over 38% of the effects were negative (see Figure 1). The weighted variance of this distribution is 0.97, whereas the estimate of the sampling error variance is only 0.09. A potential problem in meta-analyses is a violation of the assumption of independence. Such a violation occurs either when multiple observations are taken from the same study (Rosenthal, 1984) or when several papers are authored by the same person (Wolf, 1986). In the present investigation, there were 91 effects derived from the laboratory experiments reported by Mikulincer (e.g., 1988a, 1988b). This raises the possibility that the average effect size is biased, because his studies manipulated extreme negative FIs and used similar tasks. In fact, the weighted average d in Mikulincer's studies was —0.39; whereas in the remainder of the

The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory.

How do people monitor their knowledge during acquisition? A cue-utilization approach to judgments of learning (JOLs) is outlined, distinguishing 3 types of cues for JOLs: intrinsic, extrinsic, and mnemonic. In 4 experiments using paired-associates learning, item difficulty (intrinsic) exerted similar effects of JOLs and recall. In contrast, the extrinsic factors of list repetition, item repetition within a list, and stimulus duration affected JOLs less strongly than recall, supporting the proposition that extrinsic factors are discounted in making JOLs. Although practice impaired calibration, increasing underconfidence, it did improve resolution (i.e., the recall-JOL correlation). This improvement was seen to reflect a shift in the basis of JOLs with practice, from reliance on intrinsic factors, towards greater reliance on mnemonic-based heuristics.

Monitoring one's own knowledge during study : A cue-utilization approach to judgments of learning

When people are allowed freedom to volunteer or withhold information, they can enhance the accuracy of their memory reports substantially relative to forced-report performance. A theoretical framework addressing the strategic regulation of memory reporting is put forward that delineates the mediating role of metamemorial monitoring and control processes. Although the enhancement of memory accuracy is generally accompanied by a reduction in memory quantity, experimental and simulation results indicate that both of these effects depend critically on (a) accuracy incentive and (b) monitoring effectiveness. The results are discussed with regard to the contribution of meta-memory processes to memory performance, and a general methodology is proposed that incorporates these processes into the assessment of memory-accuracy and memory-quantity performance.

Monitoring and control processes in the strategic regulation of memory accuracy

Even when Ss fail to recall a solicited target, they can provide feeling-of-knowing (FOK) judgments about its availability in memory. Most previous studies addressed the question of FOK accuracy, only a few examined how FOK itself is determined, and none asked how the processes assumed to underlie FOK also account for its accuracy. The present work examined all 3 questions within a unified model, with the aim of demystifying the FOK phenomenon. The model postulates that the computation of FOK is parasitic on the processes involved in attempting to retrieve the target, relying on the accessibility of pertinent information. It specifies the links between memory strength, accessibility of correct and incorrect information about the target, FOK judgments, and recognition memory. Evidence from 3 experiments is presented. The results challenge the view that FOK is based on a direct, privileged access to an internal monitor.

How do we know that we know? The accessibility model of the feeling of knowing.

Much research in the last 2 decades has demonstrated that humans deviate from normative models of decision making and rational judgment. In 4 studies involving 954 participants, the authors explored the extent to which measures of cognitive ability and thinking dispositions can predict discrepancies from normative responding on a variety of tasks from the heuristics and biases literature including the selection task, belief bias in syllogistic reasoning, argument evaluation, base-rate use, covariation detection, hypothesis testing, outcome bias, if-only thinking, knowledge calibration, hindsight bias, and the false consensus paradigm. Significant relationships involving cognitive ability were interpreted as indicating algorithmic-level limitations on the computation of the normative response. Relationships with thinking dispositions were interpreted as indicating that styles of epistemic regulation can predict

http://www.keithstanovich.com/Site/Research_on_Reasoning_files/Stanovich-Ind-Dif-JEPG98.pdf

Individual differences in rational thought.

People's ability to assess probabilities of various events has been the topic of much interest in the areas of judgment, prediction, decision making, and memory. The evaluation of probabilistic judgments, however, raises interesting logical questions as to what it means to be a"good" judge. In this article, we focus on a normative concept of probabilistic accuracy that we call discrimination and present a measure of a judge's discrimination skill. This measure builds on an earlier index (Murphy, 1973) and has the advantages that (a) it can be interpreted as the percentage of variance accounted for by the judge and (b) it is unbiased. By way of deriving this new discrimination measure, we also show that it is related to Pearson's chi-square statistic, a result which may be useful in the future development of hypothesis testing and estimation procedures.

Measures of Discrimination Skill in Probabilistic Judgment

People frequently must make judgments of quantities. This article concerns how increasing the number of cues supporting such judgments affects accuracy as well as the judgment process itself. Research on these issues is sparse. This seems partly due to ambiguity about what it means to say that quantity judgments are accurate. Thus, a major focus of the article is on integrating 2 disparate approaches to analyzing quantity judgment, lens model analysis from psychology and mean squared error analysis from economic forecasting

J. Frank Yates

Papers

Measures of Discrimination Skill in Probabilistic Judgment

How quantity judgment changes as the number of cues increases: An analytical framework and review.

Cross-National Variation in Probability Judgment: (665412011-508)