Calibration of probabilities: the state of the art to 1980

TLDR: In this paper, a review of the literature concerning calibration of probabilistic assessments is presented, where the authors identify two kinds of "goodness" in probability assessments: normative goodness, which reflects the degree to which assessments express the assessor's true beliefs and conform to the axioms of probability theory, and substantive goodness, reflecting the amount of knowledge of the topic area contained in the assessments.

Abstract: From the subjectivist point of view (de Finetti, 1937/1964), a probability is a degree of belief in a proposition. It expresses a purely internal state; there is no “right,” “correct,” or “objective” probability residing somewhere “in reality” against which one's degree of belief can be compared. In many circumstances, however, it may become possible to verify the truth or falsity of the proposition to which a probability was attached. Today, one assesses the probability of the proposition “it will rain tomorrow.” Tomorrow, one looks at the rain gauge to see whether or not it has rained. When possible, such verification can be used to determine the adequacy of probability assessments. Winkler and Murphy (1968b) have identified two kinds of “goodness” in probability assessments: normative goodness, which reflects the degree to which assessments express the assessor's true beliefs and conform to the axioms of probability theory, and substantive goodness, which reflects the amount of knowledge of the topic area contained in the assessments. This chapter reviews the literature concerning yet another aspect of goodness, called calibration. If a person assesses the probability of a proposition being true as .7 and later finds that the proposition is false, that in itself does not invalidate the assessment. However, if a judge assigns .7 to 10,000 independent propositions, only 25 of which subsequently are found to be true, there is something wrong with these assessments.