When to trust the data: Further investigations of system error in a scientific reasoning task

TLDR: Two studies that focus on subjects’ responses to erroneous feedback in a hypothesis testing situation—a variant of Wason’s (1960) 2–4–6 rule discovery task in which some feedback was subject to system error: “hits” were reported as “misses” and vice versa.

Abstract: When evaluating experimental evidence, how do people deal with the possibility that some of the feedback is erroneous? The potential for error means that evidence evaluation must include decisions about when to "trust the data." In this paper we present two studies that focus on subjects' responses to erroneous feedback in a hypothesis testing situation-a variant of Wason's (1960) 2-4-6 rule discovery task in which some feedback was subject to system error: "hits" were reported as "misses" and vice versa. Our results show that, in contrast to previous research, people are equally adept at identifying false negatives and false positives; further, successful subjects were less likely to use a positive test strategy (Klayman & Ha, 1987) than were unsuccessful subjects. Finally, although others have found that generating possible hypotheses prior to experimentation increases success and task efficiency, such a manipulation did little to mitigate the effects of system error.