Measures of Discrimination Skill in Probabilistic Judgment

TLDR: In this paper, the authors focus on a normative concept of probabilistic accuracy that they call discrimination and present a measure of a judge's discrimination skill, which can be interpreted as the percentage of variance accounted for by the judge and it is unbiased.

Abstract: People's ability to assess probabilities of various events has been the topic of much interest in the areas of judgment, prediction, decision making, and memory. The evaluation of probabilistic judgments, however, raises interesting logical questions as to what it means to be a"good" judge. In this article, we focus on a normative concept of probabilistic accuracy that we call discrimination and present a measure of a judge's discrimination skill. This measure builds on an earlier index (Murphy, 1973) and has the advantages that (a) it can be interpreted as the percentage of variance accounted for by the judge and (b) it is unbiased. By way of deriving this new discrimination measure, we also show that it is related to Pearson's chi-square statistic, a result which may be useful in the future development of hypothesis testing and estimation procedures.