An evaluation of lineup presentation, weapon presence, and a distinctive feature using ROC analysis
TL;DR: In this paper, the authors conducted an experiment with both laboratory and online participants to test hypotheses regarding important system and estimator variables for eyewitness identification and found that simultaneous lineups yielded higher discriminability than sequential lineups.
Abstract: We conducted an experiment ( N = 2675) including both laboratory and online participants to test hypotheses regarding important system and estimator variables for eyewitness identification. Simultaneous lineups were compared to sequential lineups with the suspect presented early versus late because there is evidence that suspect position could be an important factor determining a simultaneous versus sequential advantage in guilty-innocent suspect discriminability. We also manipulated whether or not the perpetrator held a weapon or had a distinctive feature on his face, to re-evaluate recent evidence that these factors interact. Overall, the simultaneous lineup yielded higher discriminability than the sequential lineup, and there was no effect of sequential position. Discriminability was higher when the perpetrator had no weapon, but only when no distinctive feature was present. We conclude with a discussion of the importance of exploring interactions between system and estimator variables using Receiver Operating Characteristic (ROC) analysis.
Citations
More filters
TL;DR: Understanding the information value of eyewitness confidence under pristine testing conditions can help the criminal justice system to simultaneously achieve both of its main objectives: to exonerate the innocent and to convict the guilty.
Abstract: The U.S. legal system increasingly accepts the idea that the confidence expressed by an eyewitness who identified a suspect from a lineup provides little information as to the accuracy of that identification. There was a time when this pessimistic assessment was entirely reasonable because of the questionable eyewitness-identification procedures that police commonly employed. However, after more than 30 years of eyewitness-identification research, our understanding of how to properly conduct a lineup has evolved considerably, and the time seems ripe to ask how eyewitness confidence informs accuracy under more pristine testing conditions (e.g., initial, uncontaminated memory tests using fair lineups, with no lineup administrator influence, and with an immediate confidence statement). Under those conditions, mock-crime studies and police department field studies have consistently shown that, for adults, (a) confidence and accuracy are strongly related and (b) high-confidence suspect identifications are remarkably accurate. However, when certain non-pristine testing conditions prevail (e.g., when unfair lineups are used), the accuracy of even a high-confidence suspect ID is seriously compromised. Unfortunately, some jurisdictions have not yet made reforms that would create pristine testing conditions and, hence, our conclusions about the reliability of high-confidence identifications cannot yet be applied to those jurisdictions. However, understanding the information value of eyewitness confidence under pristine testing conditions can help the criminal justice system to simultaneously achieve both of its main objectives: to exonerate the innocent (by better appreciating that initial, low-confidence suspect identifications are error prone) and to convict the guilty (by better appreciating that initial, high-confidence suspect identifications are surprisingly accurate under proper testing conditions).
242 citations
Cites background from "An evaluation of lineup presentatio..."
...For example, Carlson and Carlson (2014) and Carlson, Dias, Weatherford, and Carlson (in press) found that although the presence of a weapon clearly led to worse memory performance overall (the weapon-focus effect), it had virtually no effect on the accuracy of identifications made with high…...
[...]
...…lineup is one in which the suspect stands out from the fillers such that the suspect 100 Brewer, Keast, and Rihworth (2002) Brewer and Wells (2006) Carlson and Carlson (2014) Confidence Confidence Control Average Average Average Reflection Disconfirmation a b c 90 80 70 60 50 0–20 30–40 50–60…...
[...]
TL;DR: Two graphical techniques, receiver operating characteristic (ROC) analysis and what might be termed "confidence-accuracy characteristic" (CAC) analysis, are important tools for investigating variables that affect the accuracy of eyewitness identifications as discussed by the authors.
Abstract: Two graphical techniques, receiver operating characteristic (ROC) analysis and what might be termed “confidence–accuracy characteristic” (CAC) analysis, are important tools for investigating variables that affect the accuracy of eyewitness identifications (e.g., type of lineup, exposure duration, same-race vs. other-race identifications, etc.). CAC analysis (a close relative of calibration analysis) consists of simply plotting suspect identification accuracy for each level of confidence. Two parties interested in the results of such investigations include (1) legal policymakers (e.g., state legislators and police chiefs) and (2) triers of guilt and innocence (e.g., judges and jurors). Which type of analysis is the most relevant to which party? The answer is largely a matter of whether the variable in question is a system variable or an estimator variable. ROC analysis, which measures discriminability, is critical for understanding system variables that affect eyewitness accuracy (e.g., the best lineup procedures). Thus, policymakers should be particularly attuned to the results of ROC analysis when making decisions about those variables. CAC analysis, which directly measures the confidence–accuracy relationship for suspect IDs, is critical for understanding the effect of estimator variables on eyewitness accuracy (e.g., exposure duration). Thus, triers of guilt and innocence should be particularly attuned to the results of CAC analysis. The utility of both analyses to system and estimator variables is illustrated by examining both types of analyses on previously published experiments and new experiments.
131 citations
TL;DR: The findings suggest that confidence in an eyewitness identification from a fair lineup is a highly reliable indicator of accuracy and if there is any difference in diagnostic accuracy between the two lineup formats, it likely favors the simultaneous procedure.
Abstract: Laboratory-based mock crime studies have often been interpreted to mean that (i) eyewitness confidence in an identification made from a lineup is a weak indicator of accuracy and (ii) sequential lineups are diagnostically superior to traditional simultaneous lineups. Largely as a result, juries are increasingly encouraged to disregard eyewitness confidence, and up to 30% of law enforcement agencies in the United States have adopted the sequential procedure. We conducted a field study of actual eyewitnesses who were assigned to simultaneous or sequential photo lineups in the Houston Police Department over a 1-y period. Identifications were made using a three-point confidence scale, and a signal detection model was used to analyze and interpret the results. Our findings suggest that (i) confidence in an eyewitness identification from a fair lineup is a highly reliable indicator of accuracy and (ii) if there is any difference in diagnostic accuracy between the two lineup formats, it likely favors the simultaneous procedure.
83 citations
TL;DR: It is shown how simple unrecognized incompatibilities among dependent measures, analysis tools, and the properties of data can lead to fundamental interpretive errors that may lead to misguided research efforts and policy recommendations.
Abstract: There is a replication crisis in science, to which psychological research has not been immune: Many effects have proven uncomfortably difficult to reproduce. Although the reli- ability of data is a serious concern, we argue that there is a deeper and more insidious problem in the field: the persistent and dramatic misinterpretation of empirical results that replicate eas- ily and consistently. Using a series of four highly studied "text- book" examples from different research domains (eyewitness memory, deductive reasoning, social psychology, and child wel- fare), we show how simple unrecognized incompatibilities among dependent measures, analysis tools, and the properties of data can lead to fundamental interpretive errors. These errors, which are not reduced by additional data collection, may lead to misguided research efforts and policy recommendations. We conclude with a set of recommended strategies and research tools to reduce the probability of these persistent and largely unrecog- nized errors. The use of receiver operating characteristic (ROC) curves is highlighted as one such recommendation.
69 citations
Cites background from "An evaluation of lineup presentatio..."
...…ROCs that look like those predicted by diagnosticity.5 To date, several large-scale studies (total N = 5,411) have compared sequential and simultaneous lineups using ROCs as the primary analytic tool (Carlson & Carlson, 2014; Dobolyi & Dodson, 2013; Gronlund et al., 2012; Mickes et al., 2012)....
[...]
TL;DR: A novel Bayesian treatment of the eyewitness identification problem as it relates to various system variables, such as instruction effects, lineup presentation format, lineup-filler similarity, lineup administrator influence, and show-ups versus lineups is provided.
Abstract: We provide a novel Bayesian treatment of the eyewitness identification problem as it relates to various system variables, such as instruction effects, lineup presentation format, lineup-filler similarity, lineup administrator influence, and show-ups versus lineups. We describe why eyewitness identification is a natural Bayesian problem and how numerous important observations require careful consideration of base rates. Moreover, we argue that the base rate in eyewitness identification should be construed as a system variable (under the control of the justice system). We then use prior-by-posterior curves and information–gain curves to examine data obtained from a large number of published experiments. Next, we show how information–gain curves are moderated by system variables and by witness confidence and we note how information–gain curves reveal that lineups are consistently more proficient at incriminating the guilty than they are at exonerating the innocent. We then introduce a new type of analysis that we developed called base-rate effect–equivalency (BREE) curves. BREE curves display how much change in the base rate is required to match the impact of any given system variable. The results indicate that even relatively modest changes to the base rate can have more impact on the reliability of eyewitness identification evidence than do the traditional system variables that have received so much attention in the literature. We note how this Bayesian analysis of eyewitness identification has implications for the question of whether there ought to be a reasonable-suspicion criterion for placing a person into the jeopardy of an identification procedure.
56 citations
Cites background or methods from "An evaluation of lineup presentatio..."
...…vs. sequential lineups, show-ups vs. lineups) and conditions (e.g., good vs. poor witnessing conditions) in an attempt to separate discriminability from response bias (e.g., Carlson & Carlson, 2014; Gronlund, Wixted, & Mickes, 2014; Mickes, Flowe, & Wixted, 2012; Wixted & Mickes, 2014)....
[...]
...An increasingly dominant view in recent years has been to use Receiver Operating Characteristic (ROC) curves (e.g., Carlson & Carlson, 2014; Gronlund, Wixted, & Mickes, 2014; Mickes, Flowe, & Wixted, 2012; Wixted & Mickes, 2014)....
[...]
References
More filters
TL;DR: pROC as mentioned in this paper is a package for R and S+ that contains a set of tools displaying, analyzing, smoothing and comparing ROC curves in a user-friendly, object-oriented and flexible interface.
Abstract: Receiver operating characteristic (ROC) curves are useful tools to evaluate classifiers in biomedical and bioinformatics applications. However, conclusions are often reached through inconsistent use or insufficient statistical analysis. To support researchers in their ROC curves analysis we developed pROC, a package for R and S+ that contains a set of tools displaying, analyzing, smoothing and comparing ROC curves in a user-friendly, object-oriented and flexible interface. With data previously imported into the R or S+ environment, the pROC package builds ROC curves and includes functions for computing confidence intervals, statistical tests for comparing total or partial area under the curve or the operating points of different classifiers, and methods for smoothing ROC curves. Intermediary and final results are visualised in user-friendly interfaces. A case study based on published clinical and biomarker data shows how to perform a typical ROC analysis with pROC. pROC is a package for R and S+ specifically dedicated to ROC analysis. It proposes multiple statistical tests to compare ROC curves, and in particular partial areas under the curve, allowing proper ROC interpretation. pROC is available in two versions: in the R programming language or with a graphical user interface in the S+ statistical software. It is accessible at http://expasy.org/tools/pROC/
under the GNU General Public License. It is also distributed through the CRAN and CSAN public repositories, facilitating its installation.
8,052 citations
TL;DR: This paper presents a meta-modelling system that automates the very labor-intensive and therefore time-heavy and therefore expensive and expensive process of manually cataloging and cataloging medical equipment for use in the health care system.
Abstract: John A. Swets, Robyn M. Dawes, and John Monahan BBN Technologies (emeritus), Cambridge, Massachusetts; Radiology Department, Brigham and Women’s Hospital, and Department of Health Care Policy, Harvard Medical School, Boston, Massachusetts, Department of Social and Decision Sciences, Carnegie Mellon University, Pittsburgh, Pennsylvania, and School of Law, University of Virginia, Charlottesville, Virginia
774 citations
TL;DR: In this paper, a crime was staged for 240 unsuspecting eyewitnesses either individually or in pairs, and one quarter of the eyewitnesses attempted identifications in each of four lineup conditions: six pictures were presented either simultaneously, as used in traditional procedures, or sequentially, in which yes/no judgments were made for each picture; each procedure either contained the photograph of the criminal-confederate or a picture of a similar looking replacement.
Abstract: Staged crime research has demonstrated the utility of controlling the conduct of lineups as a means of reducing false identifications with little or no apparent decline in the rate of correct identifications by eyewitnesses (e.g., Lindsay & Wells, 1980; Malpass & Devine, 1981a; Wells, 1984). A recent variation in lineup procedure shows that a blank lineup, which includes no suspects, can reduce the rate of false identifications if it precedes the actual lineup. However, there are several practical problems that make it unlikely that police will accept this procedure. Sequential lineup presentation is proposed as a means of accomplishing the same goals of reducing false identifications with little or no loss in accurate identifications. A crime was staged for 240 unsuspecting eyewitnesses either individually or in pairs. One quarter of the eyewitnesses attempted identifications in each of four lineup conditions: Six pictures were presented either simultaneously, as used in traditional procedures, or sequentially, in which yes/no judgments were made for each picture; each procedure either contained the photograph of the criminal-confederate or a picture of a similar looking replacement. Sequential lineup presentation significantly reduced false identifications but did not significantly influence correct identifications when compared with the simultaneous procedure. This resulted in an overall increase in diagnosticity ratio (Wells & Lindsay, 1980) using the sequential procedure. The data are interpreted as supporting the conclusion that sequential presentation of lineups can reduce false identifications of innocent suspects by reducing eyewitnesses' reliance on relativejudgment processes.
529 citations
458 citations
TL;DR: In this paper, the authors presented subject-witnesses with a series of slides depicting an event in a fast-food restaurant, where half of the subjects saw a customer point a gun at the cashier; the other half saw him hand the cashiers a check.
Abstract: “Weapon focus” refers to the concentration of acrime witness's attention on a weapon, and the resultant reduction in ability to remember other details of the crime. We examined this phenomenon by presenting subject-witnesses with a series of slides depicting an event in a fast-food restaurant. Half of the subjects saw a customer point a gun at the cashier; the other half saw him hand the cashier a check. In Experiment 1, eye movements were recorded while subjects viewed the slides. Results showed that subjects made more eye fixations on the weapon than on the check, and fixations on the weapon were of a longer duration than fixations on the check. In Experiment 2, the memory of subjects in the weapon condition was poorer than the memory of subjects in the check condition: In Experiment 1 similar, though only marginally significant, performance effects were obtained. These results provide the first direct empirical support for weapon focus.
458 citations