Evaluating Eyewitness Identification Procedures Using Receiver Operating Characteristic Analysis
03 Feb 2014-Current Directions in Psychological Science (Sage Publications)-Vol. 23, Iss: 1, pp 3-10
TL;DR: In this article, the receiver operating characteristic (ROC) curves are used to trace out discriminability across levels of response bias for each procedure and demonstrate that ROC analysis is the only way to make that determination.
Abstract: Eyewitness identification is a pivotal issue in applied research because, in practice, a correct identification can help to remove a dangerous criminal from society, but a false identification can lead to the erroneous conviction of an innocent suspect. Consequently, psychologists have tried to ascertain the best procedures for collecting identification evidence, evaluating them using measures based on the ratio of correct to false identification rates. Unfortunately, ratio-based measures are ambiguous because they change systematically as a function of a witness's willingness to choose. In other words, a measure thought to index discriminability is instead fully confounded with response bias. A better method involves constructing receiver operating characteristic (ROC) curves. Using ROC curves, researchers can trace out discriminability across levels of response bias for each procedure. We illustrate the shortcomings of ratio- based measures and demonstrate why ROC analysis is required. In recent studies, researchers comparing simultaneous and sequential lineup procedures using ROC analyses have provided no evidence for the sequential superiority effect and instead have shown that the simultaneous procedure may be diagnostically superior. It is not yet clear which lineup procedure will prove to be generally superior, but it is clear that ROC analysis is the only way to make that determination.
Citations
More filters
TL;DR: A signal-detection-based model of eyewitness identification is proposed, one that encourages the use of (and helps to conceptualize) receiver operating characteristic (ROC) analysis to measure discriminability, and a diagnostic feature-detector hypothesis is proposed to account for that result.
Abstract: The theoretical understanding of eyewitness identifications made from a police lineup has long been guided by the distinction between absolute and relative decision strategies. In addition, the accuracy of identifications associated with different eyewitness memory procedures has long been evaluated using measures like the diagnosticity ratio (the correct identification rate divided by the false identification rate). Framed in terms of signal-detection theory, both the absolute/relative distinction and the diagnosticity ratio are mainly relevant to response bias while remaining silent about the key issue of diagnostic accuracy, or discriminability (i.e., the ability to tell the difference between innocent and guilty suspects in a lineup). Here, we propose a signal-detection-based model of eyewitness identification, one that encourages the use of (and helps to conceptualize) receiver operating characteristic (ROC) analysis to measure discriminability. Recent ROC analyses indicate that the simultaneous presentation of faces in a lineup yields higher discriminability than the presentation of faces in isolation, and we propose a diagnostic feature-detection hypothesis to account for that result. According to this hypothesis, the simultaneous presentation of faces allows the eyewitness to appreciate that certain facial features (viz., those that are shared by everyone in the lineup) are non-diagnostic of guilt. To the extent that those non-diagnostic features are discounted in favor of potentially more diagnostic features, the ability to discriminate innocent from guilty suspects will be enhanced.
134 citations
TL;DR: Two graphical techniques, receiver operating characteristic (ROC) analysis and what might be termed "confidence-accuracy characteristic" (CAC) analysis, are important tools for investigating variables that affect the accuracy of eyewitness identifications as discussed by the authors.
Abstract: Two graphical techniques, receiver operating characteristic (ROC) analysis and what might be termed “confidence–accuracy characteristic” (CAC) analysis, are important tools for investigating variables that affect the accuracy of eyewitness identifications (e.g., type of lineup, exposure duration, same-race vs. other-race identifications, etc.). CAC analysis (a close relative of calibration analysis) consists of simply plotting suspect identification accuracy for each level of confidence. Two parties interested in the results of such investigations include (1) legal policymakers (e.g., state legislators and police chiefs) and (2) triers of guilt and innocence (e.g., judges and jurors). Which type of analysis is the most relevant to which party? The answer is largely a matter of whether the variable in question is a system variable or an estimator variable. ROC analysis, which measures discriminability, is critical for understanding system variables that affect eyewitness accuracy (e.g., the best lineup procedures). Thus, policymakers should be particularly attuned to the results of ROC analysis when making decisions about those variables. CAC analysis, which directly measures the confidence–accuracy relationship for suspect IDs, is critical for understanding the effect of estimator variables on eyewitness accuracy (e.g., exposure duration). Thus, triers of guilt and innocence should be particularly attuned to the results of CAC analysis. The utility of both analyses to system and estimator variables is illustrated by examining both types of analyses on previously published experiments and new experiments.
131 citations
Cites background from "Evaluating Eyewitness Identificatio..."
...…that would be most informative to a judge or juror is whether a lineup ID that is made with high confidence means that that ID is likely 1 Gronlund et al. (2014) provide a tutorial on conducting ROC analysis for lineup data. in Memory and Cognition 4 (2015) 93–102 95 a j l T t e v p p p…...
[...]
TL;DR: In this paper, the authors conducted an experiment with both laboratory and online participants to test hypotheses regarding important system and estimator variables for eyewitness identification and found that simultaneous lineups yielded higher discriminability than sequential lineups.
Abstract: We conducted an experiment ( N = 2675) including both laboratory and online participants to test hypotheses regarding important system and estimator variables for eyewitness identification. Simultaneous lineups were compared to sequential lineups with the suspect presented early versus late because there is evidence that suspect position could be an important factor determining a simultaneous versus sequential advantage in guilty-innocent suspect discriminability. We also manipulated whether or not the perpetrator held a weapon or had a distinctive feature on his face, to re-evaluate recent evidence that these factors interact. Overall, the simultaneous lineup yielded higher discriminability than the sequential lineup, and there was no effect of sequential position. Discriminability was higher when the perpetrator had no weapon, but only when no distinctive feature was present. We conclude with a discussion of the importance of exploring interactions between system and estimator variables using Receiver Operating Characteristic (ROC) analysis.
66 citations
TL;DR: In this paper, the authors examined the impact of retention interval on showup identifications, and concluded that immediate showups might be no worse than, and perhaps even better than, a lineup conducted after a delay.
Abstract: Showups – when a single suspect is presented to an eyewitness – are thought to be a more suggestive procedure than traditional lineups by the U.S. Supreme Court and social science researchers. The present experiment examined the impact of retention interval on showup identifications, because immediate showups might be no worse than, and perhaps even better than, a lineup conducted after a delay. Participants (N = 1584) viewed a mock-crime video and then were presented with a showup or a simultaneous lineup, either immediately or a 48 h delay. Receiver operating characteristic (ROC) analyses revealed that a showup never resulted in better identification accuracy than a lineup. We conclude with a discussion of whether showups should ever be used.
64 citations
Cites background or methods from "Evaluating Eyewitness Identificatio..."
...Recently, researchers have demonstrated the utility of ROC curves when comparing eyewitness identification procedures (Mickes et al., 2012; Wixted et al., 2014; Gronlund et al., 2014)....
[...]
...The standard error is estimated by the bootstrap method using 10,000 bootstraps (see Mickes et al., 2012; for a tutorial, see Gronlund et al., 2014)....
[...]
...Recently, researchers have demonstrated the utility of ROC curves when comparing eyewitness identification procedures (Mickes et al., 2012; Wixted et al., 2014; Gronlund et al., 2014)....
[...]
TL;DR: A novel Bayesian treatment of the eyewitness identification problem as it relates to various system variables, such as instruction effects, lineup presentation format, lineup-filler similarity, lineup administrator influence, and show-ups versus lineups is provided.
Abstract: We provide a novel Bayesian treatment of the eyewitness identification problem as it relates to various system variables, such as instruction effects, lineup presentation format, lineup-filler similarity, lineup administrator influence, and show-ups versus lineups. We describe why eyewitness identification is a natural Bayesian problem and how numerous important observations require careful consideration of base rates. Moreover, we argue that the base rate in eyewitness identification should be construed as a system variable (under the control of the justice system). We then use prior-by-posterior curves and information–gain curves to examine data obtained from a large number of published experiments. Next, we show how information–gain curves are moderated by system variables and by witness confidence and we note how information–gain curves reveal that lineups are consistently more proficient at incriminating the guilty than they are at exonerating the innocent. We then introduce a new type of analysis that we developed called base-rate effect–equivalency (BREE) curves. BREE curves display how much change in the base rate is required to match the impact of any given system variable. The results indicate that even relatively modest changes to the base rate can have more impact on the reliability of eyewitness identification evidence than do the traditional system variables that have received so much attention in the literature. We note how this Bayesian analysis of eyewitness identification has implications for the question of whether there ought to be a reasonable-suspicion criterion for placing a person into the jeopardy of an identification procedure.
56 citations
References
More filters
TL;DR: For diagnostic systems used to distinguish between two classes of events, analysis in terms of the "relative operating characteristic" of signal detection theory provides a precise and valid measure of diagnostic accuracy.
Abstract: Diagnostic systems of several kinds are used to distinguish between two classes of events, essentially "signals" and "noise". For them, analysis in terms of the "relative operating characteristic" of signal detection theory provides a precise and valid measure of diagnostic accuracy. It is the only measure available that is uninfluenced by decision biases and prior probabilities, and it places the performances of diverse systems on a common, easily interpreted scale. Representative values of this measure are reported here for systems in medical imaging, materials testing, weather forecasting, information retrieval, polygraph lie detection, and aptitude testing. Though the measure itself is sound, the values obtained from tests of diagnostic systems often require qualification because the test data on which they are based are of unsure quality. A common set of problems in testing is faced in all fields. How well these problems are handled, or can be handled in a given field, determines the degree of confidence that can be placed in a measured value of accuracy. Some fields fare much better than others.
8,569 citations
"Evaluating Eyewitness Identificatio..." refers methods in this paper
...ROC analysis is widely used to measure the accuracy of diagnostic systems in fields as varied as medical imaging, weather forecasting, and materials testing (for reviews, see Swets, 1988; Swets, Dawes, & Monahan, 2000)....
[...]
TL;DR: This paper presents a meta-modelling system that automates the very labor-intensive and therefore time-heavy and therefore expensive and expensive process of manually cataloging and cataloging medical equipment for use in the health care system.
Abstract: John A. Swets, Robyn M. Dawes, and John Monahan BBN Technologies (emeritus), Cambridge, Massachusetts; Radiology Department, Brigham and Women’s Hospital, and Department of Health Care Policy, Harvard Medical School, Boston, Massachusetts, Department of Social and Decision Sciences, Carnegie Mellon University, Pittsburgh, Pennsylvania, and School of Law, University of Virginia, Charlottesville, Virginia
774 citations
TL;DR: In this paper, three important themes from the scientific literature relevant to lineup methods were identified and reviewed, namely relative-judgment processes, the lineups-as-experiments analogy, and confidence malleability.
Abstract: There is increasing evidence that false eyewitness identification is the primary cause of the conviction of innocent people. In 1996, the American Psychology/Law Society and Division 41 of the American Psychological Association appointed a subcommittee to review scientific evidence and make recommendations regarding the best procedures for constructing and conducting lineups and photospreads. Three important themes from the scientific literature relevant to lineup methods were identified and reviewed, namely relative-judgment processes, the lineups-as-experiments analogy, and confidence malleability. Recommendations are made that double-blind lineup testing should be used, that eyewitnesses should be forewarned that the culprit might not be present, that distractors should be selected based on the eyewitness's verbal description of the perpetrator, and that confidence should be assessed and recorded at the time of identification. The potential costs and benefits of these recommendations are discussed.
705 citations
"Evaluating Eyewitness Identificatio..." refers background in this paper
...Presenting lineup members in a sequential manner has been proposed as a partial solution to the unreliability of eyewitness IDs (Wells et al., 1998)....
[...]
TL;DR: This paper presents a brief description of the ROC, and shows how it provides a measure of diagnostic accuracy that is free of judgmental bias.
Abstract: Analysis in terms of the relative operating characteristic (ROC) has recently been applied to several studies of medical decision-making, primarily to decisions based on imaging techniques. This paper presents a brief description of the ROC, and shows how it provides a measure of diagnostic accuracy that is free of judgmental bias. The results of medical studies are reviewed, and the main questions of theory and method that have arisen in the medical context are identified. Certain of these questions are basic to any psychophysical test, in which case an attempt has been made to present the best available answers. Other questions are of special medical importance and relevant reports are reviewed along with a description of current efforts to provide answers.
680 citations
"Evaluating Eyewitness Identificatio..." refers background in this paper
...However, the procedure that yields greater discriminability is always preferred, a point that is well understood in the field of diagnostic medicine (e.g., Swets, 1979)....
[...]
TL;DR: In this paper, a crime was staged for 240 unsuspecting eyewitnesses either individually or in pairs, and one quarter of the eyewitnesses attempted identifications in each of four lineup conditions: six pictures were presented either simultaneously, as used in traditional procedures, or sequentially, in which yes/no judgments were made for each picture; each procedure either contained the photograph of the criminal-confederate or a picture of a similar looking replacement.
Abstract: Staged crime research has demonstrated the utility of controlling the conduct of lineups as a means of reducing false identifications with little or no apparent decline in the rate of correct identifications by eyewitnesses (e.g., Lindsay & Wells, 1980; Malpass & Devine, 1981a; Wells, 1984). A recent variation in lineup procedure shows that a blank lineup, which includes no suspects, can reduce the rate of false identifications if it precedes the actual lineup. However, there are several practical problems that make it unlikely that police will accept this procedure. Sequential lineup presentation is proposed as a means of accomplishing the same goals of reducing false identifications with little or no loss in accurate identifications. A crime was staged for 240 unsuspecting eyewitnesses either individually or in pairs. One quarter of the eyewitnesses attempted identifications in each of four lineup conditions: Six pictures were presented either simultaneously, as used in traditional procedures, or sequentially, in which yes/no judgments were made for each picture; each procedure either contained the photograph of the criminal-confederate or a picture of a similar looking replacement. Sequential lineup presentation significantly reduced false identifications but did not significantly influence correct identifications when compared with the simultaneous procedure. This resulted in an overall increase in diagnosticity ratio (Wells & Lindsay, 1980) using the sequential procedure. The data are interpreted as supporting the conclusion that sequential presentation of lineups can reduce false identifications of innocent suspects by reducing eyewitnesses' reliance on relativejudgment processes.
529 citations
"Evaluating Eyewitness Identificatio..." refers background or methods or result in this paper
...One could simply compute d′ from Lindsay and Wells’s (1985) data as a proxy for a full receiver operating characteristic (ROC) analysis; it shows a sequential advantage (1....
[...]
...Here, we illustrate ROC analysis by showing how it is performed in the eyewitness memory domain, using the results reported by Lindsay and Wells (1985) to guide our illustration....
[...]
...As illustrated in Figure 3, the correct and false ID rate data reported by Lindsay and Wells (1985), which were collapsed over confidence ratings, are compatible with a sequential superiority effect (top panel), a simultaneous superiority effect (middle panel), or a simple criterion shift resulting in more conservative responding (bottom panel), depending on how the ROC data for each procedure actually trace out. With these considerations in mind, we now turn to a key question that the field of eyewitness memory must confront: When put to an empirical test, does the simultaneous or the sequential procedure yield the higher ROC? In other words, does one lineup procedure facilitate the discrimination between innocent versus guilty suspects more than the other? Thus far, researchers have used ROC analysis to compare simultaneous and sequential lineups in only two studies (see Figure 4). Using ROC analysis, we conducted three experiments comparing simultaneous and sequential lineups (Mickes, Flowe, & Wixted, 2012). We found that the sequential lineup procedure was never better Fig. 1. Hypothetical receiver operating characteristic (ROC) data for the sequential procedure. The data represent the seven correct identification (ID) versus false ID rates presented in Table 1. The rightmost point of the ROC represents the correct ID rate and false ID rate obtained by collapsing across all levels of confidence (see Panel 1 of Table 1). This point matches the ID rates reported by Lindsay and Wells (1985) for the sequential procedure (correct ID rate = ....
[...]
...Instead, it has focused on a statistic called the diagnosticity ratio, the ratio of correct IDs to false IDs.2 As noted by Lindsay and Wells (1985), the diagnosticity ratio favored the sequential lineup 2.94 to 1.35....
[...]
...As illustrated in Figure 3, the correct and false ID rate data reported by Lindsay and Wells (1985), which were collapsed over confidence ratings, are compatible with a sequential superiority effect (top panel), a simultaneous superiority effect (middle panel), or a simple criterion shift resulting…...
[...]