Determinants of linear judgment
1
Determinants of linear judgment: A meta-analysis of lens model studies
∗
Natalia Karelaia
1
& Robin M. Hogarth
2
HEC Université de Lausanne
1
, Lausanne
ICREA
& Universitat Pompeu Fabra
2
, Barcelona,
February 1, 2007
∗ This research was financed partially by grants from the Swiss National Science Foundation (Karelaia)
and the Spanish Ministerio de Educación y Ciencia (Hogarth). We are particularly indebted to Thomas
Stewart, Michael Doherty, and the library at Universitat Pompeu Fabra for helping us locate many lens
model studies as well as to Marcus O’Connor for providing data. We thank Chris White and Mandeep
Dhami for helpful comments.
Email: natalia.karelaia@unil.ch
and robin.hogarth@upf.edu
Determinants of linear judgment
2
Abstract
The mathematical representation of Brunswik’s lens model has been used
extensively to study human judgment and provides a unique opportunity to conduct a
meta-analysis of studies that covers roughly five decades. Specifically, we analyze
statistics of the “lens model equation” (Tucker, 1964) associated with 259 different task
environments obtained from 78 papers. In short, we find – on average – fairly high levels
of judgmental achievement and note that people can achieve similar levels of cognitive
performance in both noisy and predictable environments. Although overall performance
varies little between laboratory and field studies, both differ in terms of components of
performance and types of environments (numbers of cues and redundancy). An analysis
of learning studies reveals that the most effective form of feedback is information about
the task. We also analyze empirically when bootstrapping is more likely to occur. We
conclude by indicating shortcomings of the kinds of studies conducted to date, limitations
in the lens model methodology, and possibilities for future research.
Keywords: judgment, lens model, linear models, learning, bootstrapping
Determinants of linear judgment
3
Since the 1960s, many psychologists have used the framework of Brunswik’s
(1952) lens model to study processes where humans make predictions of specific criteria
(see, e.g., Brehmer & Joyce, 1988; Cooksey, 1996; Hastie & Kameda, 2005). For
example, a person might make a judgment (i.e., prediction) about another person’s
intelligence, about the likelihood of rain, whether a job candidate will be successful, and
so on. In all these cases, the simple beauty of Brunswik’s model lies in recognizing that
both the person’s judgment and the actual criterion predicted can be thought of as two
separate functions of cues that are available in the environment. Thus, the accuracy of
human judgment depends on the extent to which the function that describes it matches its
environmental counterpart.
But how good or accurate are people at making judgments and on what does this
depend? These are important questions that have generated considerable controversy in
the psychological literature (Cohen, 1981; Gigerenzer, 1996; Kahneman & Tversky,
1996). Whereas it is unlikely that these questions can be answered satisfactorily by any
particular approach, an advantage of research conducted within the Brunswikian tradition
is the use of a common methodology for formalizing the lens model. Thus, not only can
researchers within this tradition communicate results within a common framework, it is
possible to aggregate results quantitatively across many studies and make statements that
reflect the accumulation of results. This is the purpose of the current paper in which we
present a meta-analysis of studies conducted using the lens model over a period of five
decades.
The paper is organized as follows. We first describe the mathematical formulation
of the lens model. Second, we specify how we identified and included particular studies
Determinants of linear judgment
4
in our analysis. Third, by summarizing the results of these studies we illuminate the issue
of how accurate human judgment is and the factors that affect it. Since psychologists
typically study judgment within laboratories but people use judgment outside
laboratories, we pay particular attention to differences between laboratory and field
studies. Fourth, since the topic of learning has been central to studies within the lens
model tradition, we make a separate analysis of learning studies. Key topics center on
how much learning occurs, what affects this and the impact of different types of
feedback. Fifth, we contribute to the discussion of the relative advantages of clinical
judgments and their paramorphic representations (Hoffman, 1960) or bootstrapping
models (e.g., Goldberg, 1970; Dawes, 1971; Camerer, 1981) by analyzing the conditions
under which people are more likely to be outperformed by models of their judgments.
Finally we conclude by summarizing the main substantive conclusions of the analysis,
indicating shortcomings of the kinds of studies conducted to date, and suggesting avenues
for future research.
The mathematical formulation of Brunswik’s lens model
The use of Brunswik’s lens model received an important impetus in 1964 when a
series of papers showed how statistical methods could be used to capture judgmental
processes (Hammond, Hursch, & Todd, 1964; Hursch, Hammond, & Hursch, 1964;
Tucker, 1964. See also Castellan, 1973). In this, human judgment, denoted Y
s
, is
modeled as a linear function of a set of k cues, X
j
, j = 1,…k. Thus,
s
k
j
jjss
XY
εβ
+=
∑
=1
,
(1)
Determinants of linear judgment
5
where the
β
s,j
’s represent the weights that the person (or judge) gives to the different cues
and
ε
s
is the error term of the regression of Y
s
on the X
j
’s.
Similarly, the environmental criterion, Y
e
, can be modeled as a function of the
same cues, X
j
, j = 1,…k. That is,
e
k
j
jjee
XY
εβ
+=
∑
=1
,
(2)
where the
β
e,j
’s represent the weights that the environment gives to the different cues and
ε
e
is the error term of the regression of Y
e
on the X
j
’s – see Figure 1.
-------------------------------------------------------
Insert Figure 1 about here
-------------------------------------------------------
The logic of the lens model is that the person’s decisions will match the
environmental criterion to the extent that the weights the judge gives to the cues match
those used by the model of the environment, i.e., the matches between
β
s,j
and
β
e,j
for all j
= 1,…k. Moreover, the correlation between criterion and judgment,
se
YY
ρ
– the so-called
“achievement” index or r
a
– can be expressed by the “lens model equation”
()
(
)
22
11
sesea
RRCRGRr −−+=
(3)
where G =
se
YY
ˆˆ
ρ
(the “matching” index) is the correlation between the predictions of both
models, i.e., between
∑
=
k
j
jje
X
1
,
β
and
∑
=
k
j
jjs
X
1
,
β
; R
e
and R
s
are, respectively, the multiple
correlations of the models of the environment and the judge, and capture, on the one
hand, environmental predictability (R
e
), and on the other hand, the consistency with
which the judge executes the decision rule (R
s
); and C =
se
εε
ρ
is the correlation between
