ROC analyses in eyewitness identification research
TL;DR: In this article, a recent surge of interest in analyzing the results of eyewitness identification experiments using Receiver Operating Characteristic (ROC) curves has been discussed, and it has been argued that ROC analyses can provide an index of underlying memory discriminability.
Abstract: There has been a recent surge of interest in analyzing the results of eyewitness identification experiments using Receiver Operating Characteristic (ROC) curves. Proponents of this approach have argued not only that ROC analyses are useful but that more traditional approaches are deficient and should not be used. Three arguments might be made for why researchers would prefer ROC approaches over other techniques. The first is that ROC analyses can provide an index of underlying memory discriminability. The second is that ROC analyses provide useful information about the practical utility of identification procedures. The third is that ROC analyses are useful for testing theory. In this article, I critically examine each of these arguments and conclude that recent claims that ROC methods provide the only justifiable method of comparing identification procedures are overstated.
Citations
More filters
TL;DR: In this article, the authors show that ROC analysis confuses filler siphoning with an improvement in underlying discriminability, thereby fostering misleading theoretical conclusions about how lineups work, and illustrate how this approach misfires as a measure of underlying discrimINability.
Abstract: Some researchers have been arguing that eyewitness identification data from lineups should be analyzed using Receiver Operating Characteristic (ROC) analysis because it purportedly measures underlying discriminability. But ROC analysis, which was designed for 2 × 2 tasks, does not fit the 3 × 2 structure of lineups. Accordingly, ROC proponents force lineup data into a 2 × 2 structure by treating false-positive identifications of lineup fillers as though they were rejections. Using data from lineups versus showups, we illustrate how this approach misfires as a measure of underlying discriminability. Moreover, treating false-positive identifications of fillers as if they were rejections hides one of the most important phenomena in eyewitness lineups, namely filler siphoning. Filler siphoning reduces the risk of mistaken identification by drawing false-positive identifications away from the innocent suspect and onto lineup fillers. We show that ROC analysis confuses filler siphoning with an improvement in underlying discriminability, thereby fostering misleading theoretical conclusions about how lineups work.
55 citations
TL;DR: This work compared three fair-lineup techniques used by the police with unfair lineups in which the authors did nothing to prevent distinctive suspects from standing out, and found doing nothing not only increased subjects' willingness to identify the suspect but also markedly impaired subjects’ ability to distinguish between innocent and guilty suspects.
Abstract: Eyewitness-identification studies have focused on the idea that unfair lineups (i.e., ones in which the police suspect stands out) make witnesses more willing to identify the police suspect. We examined whether unfair lineups also influence subjects’ ability to distinguish between innocent and guilty suspects and their ability to judge the accuracy of their identification. In a single experiment (N = 8,925), we compared three fair-lineup techniques used by the police with unfair lineups in which we did nothing to prevent distinctive suspects from standing out. Compared with the fair lineups, doing nothing not only increased subjects’ willingness to identify the suspect but also markedly impaired subjects’ ability to distinguish between innocent and guilty suspects. Accuracy was also reduced at every level of confidence. These results advance theory on witnesses’ identification performance and have important practical implications for how police should construct lineups when suspects have distinctive features.
53 citations
Cites result from "ROC analyses in eyewitness identifi..."
...We also fit a signal detection process model of lineup performance to our data to further confirm these findings (see Lampinen, 2016; Wixted & Mickes, 2014)....
[...]
TL;DR: It is demonstrated that a structural phenomenon of lineups, differential filler siphoning, and not the psychological phenomenon of diagnostic-feature detection, explains why lineups are superior to showups and why fair lineups is superior to biased lineups.
Abstract: Receiver Operating Characteristic (ROC) analysis has recently come in vogue for assessing the underlying discriminability and the applied utility of lineup procedures. Two primary assumptions underlie recommendations that ROC analysis be used to assess the applied utility of lineup procedures: (a) ROC analysis of lineups measures underlying discriminability, and (b) the procedure that produces superior underlying discriminability produces superior applied utility. These same assumptions underlie a recently derived diagnostic-feature detection theory, a theory of discriminability, intended to explain recent patterns observed in ROC comparisons of lineups. We demonstrate, however, that these assumptions are incorrect when ROC analysis is applied to lineups. We also demonstrate that a structural phenomenon of lineups, differential filler siphoning, and not the psychological phenomenon of diagnostic-feature detection, explains why lineups are superior to showups and why fair lineups are superior to biased lineups. In the process of our proofs, we show that computational simulations have assumed, unrealistically, that all witnesses share exactly the same decision criteria. When criterial variance is included in computational models, differential filler siphoning emerges. The result proves dissociation between ROC curves and underlying discriminability: Higher ROC curves for lineups than for showups and for fair than for biased lineups despite no increase in underlying discriminability. (PsycINFO Database Record
44 citations
Cites methods from "ROC analyses in eyewitness identifi..."
...We began by running a simulation of lineups versus showups using methods very similar to Wixted and Mickes’ simulation (2015b; also see Lampinen, 2016) so as to replicate their finding that ROC curves for lineups were not superior to showups using such a simulation. Like Wixted and Mickes (2015b), we set PPVd=, decision criteria, and other relevant parameters to be the same for lineups and showups (see Appendix A for a full description of the parameter settings)....
[...]
...Variance Is Allowed We began by running a simulation of lineups versus showups using methods very similar to Wixted and Mickes’ simulation (2015b; also see Lampinen, 2016) so as to replicate their finding that ROC curves for lineups were not superior to showups using such a simulation....
[...]
TL;DR: Simulations of different possible identification tasks and response strategies show that ROCs are an essential tool for distinguishing memory-based processes from decisional aspects of a response; they offer important constraints on theory development.
Abstract: How should the accuracy of eyewitness identification decisions be measured, so that best practices for identification can be determined? This fundamental question is under intense debate. One side advocates for continued use of a traditional measure of identification accuracy, known as the diagnosticity ratio, whereas the other side argues that receiver operating characteristic curves (ROCs) should be used instead because diagnosticity is confounded with response bias. Diagnosticity proponents have offered several criticisms of ROCs, which we show are either false or irrelevant to the assessment of eyewitness accuracy. We also show that, like diagnosticity, Bayesian measures of identification accuracy confound response bias with witnesses’ ability to discriminate guilty from innocent suspects. ROCs are an essential tool for distinguishing memory-based processes from decisional aspects of a response; simulations of different possible identification tasks and response strategies show that they offer important constraints on theory development.
40 citations
Cites background or methods from "ROC analyses in eyewitness identifi..."
...ROCs do not “compare hit rates after equating false alarm rates” (Lampinen, 2016, p. 32)....
[...]
...4Our conclusions are restricted to the specific model that we, and Lampinen (2016), simulated....
[...]
...Lampinen (2016) showed that the simulated ROC for the showup procedure fell above the simulated ROC for the lineup procedure for each of the true d′ values he considered....
[...]
...Lampinen (2016) claims ROCs invite inappropriate comparison of accuracy at different levels of response bias....
[...]
...One recent paper, by Lampinen (2016), has the potential to play an important role in the debate about diagnosticity and the AUC, because it appears to offer a sophisticated modeling approach and has been interpreted as providing “strong additional evidence that ROC analyses on lineups are not measures of discriminability” (Wells et al....
[...]
TL;DR: Three competing signal-detection models of lineup memory are described, their likelihood functions are derived, and they are fitted to empirical ROC data on the notion that memory signals generated by the faces in a lineup are likely to be correlated because, by design, those faces share features.
36 citations
Cites methods from "ROC analyses in eyewitness identifi..."
...…2011) in the context of the WITNESS model and has often been used to frame a recent debate about the utility of ROC analysis in eyewitness identification (e.g., Lampinen, 2016; Rotello & Chen, 2016; Smith, Wells, Lindsay, & Penrod, 2017; Wixted, Mickes, Wetmore, Gronlund, & Neuschatz, 2017)....
[...]
References
More filters
TL;DR: A representation and interpretation of the area under a receiver operating characteristic (ROC) curve obtained by the "rating" method, or by mathematical predictions based on patient characteristics, is presented and it is shown that in such a setting the area represents the probability that a randomly chosen diseased subject is (correctly) rated or ranked with greater suspicion than a random chosen non-diseased subject.
Abstract: A representation and interpretation of the area under a receiver operating characteristic (ROC) curve obtained by the "rating" method, or by mathematical predictions based on patient characteristics, is presented. It is shown that in such a setting the area represents the probability that a randomly chosen diseased subject is (correctly) rated or ranked with greater suspicion than a randomly chosen non-diseased subject. Moreover, this probability of a correct ranking is the same quantity that is estimated by the already well-studied nonparametric Wilcoxon statistic. These two relationships are exploited to (a) provide rapid closed-form expressions for the approximate magnitude of the sampling variability, i.e., standard error that one uses to accompany the area under a smoothed ROC curve, (b) guide in determining the size of the sample required to provide a sufficiently reliable estimate of this area, and (c) determine how large sample sizes should be to ensure that one can statistically detect difference...
19,398 citations
Book•
01 Jan 1944
TL;DR: Theory of games and economic behavior as mentioned in this paper is the classic work upon which modern-day game theory is based, and it has been widely used to analyze a host of real-world phenomena from arms races to optimal policy choices of presidential candidates, from vaccination policy to major league baseball salary negotiations.
Abstract: This is the classic work upon which modern-day game theory is based. What began more than sixty years ago as a modest proposal that a mathematician and an economist write a short paper together blossomed, in 1944, when Princeton University Press published "Theory of Games and Economic Behavior." In it, John von Neumann and Oskar Morgenstern conceived a groundbreaking mathematical theory of economic and social organization, based on a theory of games of strategy. Not only would this revolutionize economics, but the entirely new field of scientific inquiry it yielded--game theory--has since been widely used to analyze a host of real-world phenomena from arms races to optimal policy choices of presidential candidates, from vaccination policy to major league baseball salary negotiations. And it is today established throughout both the social sciences and a wide range of other sciences.
19,337 citations
TL;DR: Evidence is presented that recognition judgments are based on an assessment of familiarity, as is described by signal detection theory, but that a separate recollection process also contributes to performance.
Abstract: Evidence is presented that recognition judgments are based on an assessment of familiarity, as is described by signal detection theory, but that a separate recollection process also contributes to performance. In 3 receiver-operating characteristics (ROC) experiments, the process dissociation procedure was used to examine the contribution of these processes to recognition memory. In Experiments 1 and 2, reducing the length of the study list increased the intercept (d') but decreased the slope of the ROC and increased the probability of recollection but left familiarity relatively unaffected. In Experiment 3, increasing study time increased the intercept but left the slope of the ROC unaffected and increased both recollection and familiarity. In all 3 experiments, judgments based on familiarity produced a symmetrical ROC (slope = 1), but recollection introduced a skew such that the slope of the ROC decreased.
1,062 citations
TL;DR: A theory for the storage and retrieval of item and associative information is presented, and expressions for signal-to-noise ratio and relative efficiency are derived.
Abstract: A theory for the storage and retrieval of item and associative information is presented. In the theory, items or events are represented as random vectors. Convolution is used as the storage operation, and correlation is used as the retrieval operation. A distributed-memory system is assumed; all information is stored in a common memory vector. The theory applies to both recognition and recall and covers both accuracy and latency. Noise in the decision stage necessitates a two-criterion decision system, and over time the criteria converge until a decision is reached. Performance is predicted from the moments (expectation and variance) of the similarity distributions, and these can be derived from the theory. Several alternative models with varying degrees of distributed memory are considered, and expressions for signal-to-noise ratio and relative efficiency are derived.
933 citations
TL;DR: In this paper, three important themes from the scientific literature relevant to lineup methods were identified and reviewed, namely relative-judgment processes, the lineups-as-experiments analogy, and confidence malleability.
Abstract: There is increasing evidence that false eyewitness identification is the primary cause of the conviction of innocent people. In 1996, the American Psychology/Law Society and Division 41 of the American Psychological Association appointed a subcommittee to review scientific evidence and make recommendations regarding the best procedures for constructing and conducting lineups and photospreads. Three important themes from the scientific literature relevant to lineup methods were identified and reviewed, namely relative-judgment processes, the lineups-as-experiments analogy, and confidence malleability. Recommendations are made that double-blind lineup testing should be used, that eyewitnesses should be forewarned that the culprit might not be present, that distractors should be selected based on the eyewitness's verbal description of the perpetrator, and that confidence should be assessed and recorded at the time of identification. The potential costs and benefits of these recommendations are discussed.
705 citations