scispace - formally typeset

Journal ArticleDOI

Policy Regarding the Sequential Lineup Is Not Informed by Probative Value but Is Informed by Receiver Operating Characteristic Analysis

03 Feb 2014-Current Directions in Psychological Science (SAGE PublicationsSage CA: Los Angeles, CA)-Vol. 23, Iss: 1, pp 17-18

Abstract: It is important to determine if switching from simultaneous to sequential lineups affects response bias (the inclination to make an identification from a lineup), discriminability (the ability to distinguish between innocent and guilty suspects), or both. Measures of probative value cannot provide such information; receiver operating characteristic analysis can. Recent receiver operating characteristic analyses indicate that switching to sequential lineups both induces more conservative responding and makes it more difficult to distinguish between innocent and guilty suspects. If more conservative responding is preferred (i.e., if policymakers judge that the harm associated with the reduction of correct identifications is exceeded by the benefit associated with the reduction in false identifications), recent data indicate that this result can be achieved without a loss of discriminability by using the simultaneous lineup procedure in conjunction with a more conservative decision criterion.

Summary (2 min read)

Introduction

  • Wells (2014, this issue) wrote, “For eyewitness-identification evidence, probative value is reflected in likelihood ratios or probabilities that an identification of the defendant offered at trial was accurate or mistaken” (p. 11).
  • Wells argued that a result like this means that an identification made using the sequential procedure is more trustworthy than is one made using the simultaneous procedure.
  • But this conclusion applies only to situations in which confidence is ignored.
  • For both procedures, this new ROC point is associated with a higher probative value than its neighbor to the right.
  • Corresponding Author: John T. Wixted, Department of Psychology, University of California, San Diego, La Jolla, CA 92093 E-mail: jwixted@ucsd.edu.

Keywords

  • Eyewitness memory, simultaneous versus sequential lineups, signal detection theory at University of Oklahoma on January 20, 2016cdp.sagepub.comDownloaded from.

18 Wixted et al.

  • Instead of using a measure of probative value to identify the best lineup procedure, one should ask which lineup procedure is better able to discriminate between innocent and guilty suspects.
  • ROC analysis measures discriminability without recourse to theory, which is why it has long been used in medicine to measure how well a diagnostic test discriminates between the presence or absence of a disease.
  • Wells’s (2014) observation that the authors “were correct to suggest that receiver operating characteristic analyses are the best way to determine if the simultaneous/sequential difference is a criterion shift” (p. 14) missed the main point of their article (Gronlund, Wixted, & Mickes, 2014, this issue).
  • Just as the authors have illustrated in Figure 1 with hypothetical data, the first three published studies using ROC analysis found that simultaneous lineups yield significantly higher discriminability than sequential lineups (Dobolyi & Dodson, in press; Gronlund et al., 2012; Mickes, Flowe, & Wixted, 2012).
  • That approach maximizes discriminability, which is the only way to simultaneously reduce both errors to which Wells referred—mistaken identifications of the innocent and nonidentifications of the guilty.

Author Contributions

  • All authors contributed equally to the manuscript.

Declaration of Conflicting Interests

  • The authors declared that they had no conflicts of interest with respect to their authorship or the publication of this article.

Note

  • 1. Ideally, the confidence statement used to establish an acceptable criterion would be the one that was made at the time of the initial identification, not the one made later during a trial (Technical Working Group for Eyewitness Evidence, 1999).

Did you find this useful? Give us your feedback

Content maybe subject to copyright    Report

Current Directions in Psychological
Science
2014, Vol. 23(1) 17 –18
© The Author(s) 2014
Reprints and permissions:
sagepub.com/journalsPermissions.nav
DOI: 10.1177/0963721413510934
cdps.sagepub.com
Wells (2014, this issue) wrote, “For eyewitness-identification
evidence, probative value is reflected in likelihood ratios
or probabilities that an identification of the defendant
offered at trial was accurate or mistaken” (p. 11). Figure
1 presents hypothetical receiver operating characteristic
(ROC) data that show the full range of correct and false
identification rates associated with a simultaneous lineup
and, separately, a sequential lineup. Next to each ROC
point is a common measure of probative value. When
Wells referred to probative value, he was referring only to
the right-most ROC point for each procedure. In Figure 1,
the probative value of the right-most ROC point is higher
for the sequential lineup (7.4) than for the simultaneous
lineup (6.9). Wells argued that a result like this means
that an identification made using the sequential proce-
dure is more trustworthy than is one made using
the simultaneous procedure. But this conclusion applies
only to situations in which confidence is ignored.
Wells ignored confidence but courts of law do not, and
neither should researchers who want to determine which
lineup procedure is associated with more trustworthy
identifications.
Just as a higher (i.e., more conservative) criterion may
be used in a court of law by attaching less weight to low-
confidence identifications, one can do the same with data
collected in the laboratory. The use of a slightly more
conservative criterion, which is achieved by treating the
lowest-confidence identifications as effective nonidentifi-
cations, generates the next point to the left on the ROC.
For both procedures, this new ROC point is associated
with a higher probative value than its neighbor to the
right. The use of an ever more conservative criterion gen-
erates additional points to the left on the ROC, each of
which is associated with a higher probative value than
the last. It is not known which ROC point is the most
relevant to a court of law, nor which probative value, but
it seems certain that the right-most point (which includes
low-confidence identifications) is the least relevant.
510934CDP
XXX10.1177/0963721413510934Wixted et al.Sequential Lineup Policy—A Response to <xref ref-type="bibr" rid="bibr8-0963721413510934">Wells (2013)</xref>
research-article2014
Corresponding Author:
John T. Wixted, Department of Psychology, University of California,
San Diego, La Jolla, CA 92093
E-mail: jwixted@ucsd.edu
Policy Regarding the Sequential Lineup
Is Not Informed by Probative Value
but Is Informed by Receiver Operating
Characteristic Analysis
John T. Wixted
1
, Scott D. Gronlund
2
, and Laura Mickes
3
1
University of California, San Diego;
2
University of Oklahoma; and
3
Royal Holloway, University
of London
Abstract
It is important to determine if switching from simultaneous to sequential lineups affects response bias (the inclination to
make an identification from a lineup), discriminability (the ability to distinguish between innocent and guilty suspects),
or both. Measures of probative value cannot provide such information; receiver operating characteristic analysis can.
Recent receiver operating characteristic analyses indicate that switching to sequential lineups both induces more
conservative responding and makes it more difficult to distinguish between innocent and guilty suspects. If more
conservative responding is preferred (i.e., if policymakers judge that the harm associated with the reduction of correct
identifications is exceeded by the benefit associated with the reduction in false identifications), recent data indicate that
this result can be achieved without a loss of discriminability by using the simultaneous lineup procedure in conjunction
with a more conservative decision criterion.
Keywords
eyewitness memory, simultaneous versus sequential lineups, signal detection theory
at University of Oklahoma on January 20, 2016cdp.sagepub.comDownloaded from

18 Wixted et al.
Instead of using a measure of probative value to iden-
tify the best lineup procedure, one should ask which
lineup procedure is better able to discriminate between
innocent and guilty suspects. The use of d to measure
discriminability is conceptually the right approach, but,
in the case of lineups, that measure is directly tied to
questionable and untested theoretical assumptions. ROC
analysis measures discriminability without recourse to
theory, which is why it has long been used in medicine
to measure how well a diagnostic test discriminates
between the presence or absence of a disease.
Wells’s (2014) observation that we “were correct to
suggest that receiver operating characteristic analyses are
the best way to determine if the simultaneous/sequential
difference is a criterion shift” (p. 14) missed the main
point of our article (Gronlund, Wixted, & Mickes, 2014,
this issue). The most important function of ROC analysis
is to identify the procedure that yields higher discrim-
inability. Just as we have illustrated in Figure 1 with
hypothetical data, the first three published studies using
ROC analysis found that simultaneous lineups yield sig-
nificantly higher discriminability than sequential lineups
(Dobolyi & Dodson, in press; Gronlund et al., 2012;
Mickes, Flowe, & Wixted, 2012). If that turns out to be the
final story, it would mean that any probative-value gain
that might be achieved from the conservative criterion
induced by a sequential lineup could be exceeded by
using a suitably conservative criterion in conjunction
with a simultaneous lineup.
1
That approach maximizes
discriminability, which is the only way to simultaneously
reduce both errors to which Wells referred—mistaken
identifications of the innocent and nonidentifications of
the guilty.
Recommended Reading
Green, D. M., & Swets, J. A. (1966). Signal detection theory and
psychophysics. New York, NY: Wiley. The original signal
detection classic, providing a mathematically elegant (and
still timely) treatment of topics such as receiver operating
characteristic analysis, decision goals, and the theory of
ideal observers.
Macmillan, N. A., & Creelman, C. D. (2005). Detection theory:
A user’s guide (2nd ed.). Mahwah, NJ: Erlbaum. The mod-
ern classic, providing a more accessible treatment of how
signal detection theory applies to the many kinds of experi-
mental designs that are routinely used by psychologists to
study memory and perception.
Author Contributions
All authors contributed equally to the manuscript.
Declaration of Conflicting Interests
The authors declared that they had no conflicts of interest with
respect to their authorship or the publication of this article.
Note
1. Ideally, the confidence statement used to establish an accept-
able criterion would be the one that was made at the time of
the initial identification, not the one made later during a trial
(Technical Working Group for Eyewitness Evidence, 1999).
References
Dobolyi, D. G., & Dodson, C. S. (in press). Eyewitness confi-
dence in simultaneous and sequential lineups: A criterion
shift account for sequential mistaken identification over-
confidence. Journal of Experimental Psychology: Applied.
Gronlund, S. D., Carlson, C. A., Neuschatz, J. S., Goodsell, C. A.,
Wetmore, S. A., Wooten, A., & Graham, M. (2012). Showups
versus lineups: An evaluation using ROC analysis. Journal
of Applied Research in Memory and Cognition, 1, 221–228.
Gronlund, S. D., Wixted, J. T., & Mickes, L. (2014). Evaluating
eyewitness identification procedures using ROC analyses.
Current Directions in Psychological Science, 23, 3–10.
Mickes, L., Flowe, H. D., & Wixted, J. T. (2012). Receiver
operating characteristic analysis of eyewitness memory:
Comparing the diagnostic accuracy of simultaneous and
sequential lineups. Journal of Experimental Psychology:
Applied, 18, 361–376.
Technical Working Group for Eyewitness Evidence. (1999).
Eyewitness evidence: A guide for law enforcement.
Washington, DC: U.S. Department of Justice, Office of
Justice Programs.
Wells, G. L. (2014). Eyewitness identification: Probative value,
criterion shifts, and policy regarding the sequential lineup.
Current Directions in Psychological Science, 23, 11–16.
Fig. 1. Hypothetical simultaneous and sequential receiver operating
characteristics (ROCs), with probative values (correct identification
rate/false identification rate) indicated next to each data point. The
right-most ROC point represents the overall correct and false identifica-
tion rates that are typically analyzed in an eyewitness memory study.
at University of Oklahoma on January 20, 2016cdp.sagepub.comDownloaded from
Citations
More filters

Journal ArticleDOI
Abstract: Showups – when a single suspect is presented to an eyewitness – are thought to be a more suggestive procedure than traditional lineups by the U.S. Supreme Court and social science researchers. The present experiment examined the impact of retention interval on showup identifications, because immediate showups might be no worse than, and perhaps even better than, a lineup conducted after a delay. Participants (N = 1584) viewed a mock-crime video and then were presented with a showup or a simultaneous lineup, either immediately or a 48 h delay. Receiver operating characteristic (ROC) analyses revealed that a showup never resulted in better identification accuracy than a lineup. We conclude with a discussion of whether showups should ever be used.

54 citations


Cites background from "Policy Regarding the Sequential Lin..."

  • ...Recently, researchers have demonstrated the utility of ROC curves when comparing eyewitness identification procedures (Mickes et al., 2012; Wixted et al., 2014; Gronlund et al., 2014)....

    [...]


Journal ArticleDOI
James Michael Lampinen1Institutions (1)
Abstract: There has been a recent surge of interest in analyzing the results of eyewitness identification experiments using Receiver Operating Characteristic (ROC) curves. Proponents of this approach have argued not only that ROC analyses are useful but that more traditional approaches are deficient and should not be used. Three arguments might be made for why researchers would prefer ROC approaches over other techniques. The first is that ROC analyses can provide an index of underlying memory discriminability. The second is that ROC analyses provide useful information about the practical utility of identification procedures. The third is that ROC analyses are useful for testing theory. In this article, I critically examine each of these arguments and conclude that recent claims that ROC methods provide the only justifiable method of comparing identification procedures are overstated.

32 citations


Journal ArticleDOI
20 Apr 2018-Memory
TL;DR: It is found that sequential lineups suffered from an unwanted position effect, and may give rise to unwanted position effects that have to be considered when conducting police lineups.
Abstract: For decades, sequential lineups have been considered superior to simultaneous lineups in the context of eyewitness identification. However, most of the research leading to this conclusion was based...

13 citations


Cites background from "Policy Regarding the Sequential Lin..."

  • ...Because these assumptions are difficult to test in the context of eyewitness identification, non-parametric measures based on the Receiver Operating Characteristic (ROC) curve are more appropriate for analyzing the performance of eyewitnesses (Mickes et al., 2012; Wixted et al., 2014)....

    [...]


Journal ArticleDOI
Abstract: Much of the literature on eyewitness identification neglects the social context in which identifications are made. As the number of cognitive psychologists conducting eyewitness research increased so did the use of signal detection theory and ROC analyses. With the resulting need for larger sample size, researchers moved toward conducting studies on internet platforms that allow for crowd-sourcing research participants. These methods make it next to impossible to ask research questions that explore the ways in which social interactions influence the identifications made by witnesses. Yet, the possibility of social context effects on witness memory are prevalent in applied contexts and research supports their existence. In addition, some eyewitness identifications may not be governed by memory at all. We argue that a consideration of social context effects is required to fully explore the reliability of witness identifications and propose a number of avenues for future research.

8 citations


Dissertation
18 Aug 2015-
Abstract: ii Co-Authorship iii Acknowledgements iv List of Figures viii List of Tables ix Chapter

References
More filters

Journal ArticleDOI
TL;DR: The basics of ROC analysis are described, explaining why it is needed and how to use it to measure the performance of different lineup procedures, and 3 ROC experiments that were designed to investigate the diagnostic accuracy of simultaneous versus sequential lineups are reported.
Abstract: A police lineup presents a real-world signal-detection problem because there are two possible states of the world (the suspect is either innocent or guilty), some degree of information about the true state of the world is available (the eyewitness has some degree of memory for the perpetrator), and a decision is made (identifying the suspect or not). A similar state of affairs applies to diagnostic tests in medicine because, in a patient, the disease is either present or absent, a diagnostic test yields some degree of information about the true state of affairs, and a decision is made about the presence or absence of the disease. In medicine, receiver operating characteristic (ROC) analysis is the standard method for assessing diagnostic accuracy. By contrast, in the eyewitness memory literature, this powerful technique has never been used. Instead, researchers have attempted to assess the diagnostic performance of different lineup procedures using methods that cannot identify the better procedure (e.g., by computing a diagnosticity ratio). Here, we describe the basics of ROC analysis, explaining why it is needed and showing how to use it to measure the performance of different lineup procedures. To illustrate the unique advantages of this technique, we also report 3 ROC experiments that were designed to investigate the diagnostic accuracy of simultaneous versus sequential lineups. According to our findings, the sequential procedure appears to be inferior to the simultaneous procedure in discriminating between the presence versus absence of a guilty suspect in a lineup.

138 citations


01 Jan 2000-

119 citations


Additional excerpts

  • ...Wells (2014, this issue) wrote, “For eyewitness-identification evidence, probative value is reflected in likelihood ratios or probabilities that an identification of the defendant offered at trial was accurate or mistaken” (p. 11). Figure 1 presents hypothetical receiver operating characteristic (ROC) data that show the full range of correct and false identification rates associated with a simultaneous lineup and, separately, a sequential lineup. Next to each ROC point is a common measure of probative value. When Wells referred to probative value, he was referring only to the right-most ROC point for each procedure. In Figure 1, the probative value of the right-most ROC point is higher for the sequential lineup (7.4) than for the simultaneous lineup (6.9). Wells argued that a result like this means that an identification made using the sequential procedure is more trustworthy than is one made using the simultaneous procedure. But this conclusion applies only to situations in which confidence is ignored. Wells ignored confidence but courts of law do not, and neither should researchers who want to determine which lineup procedure is associated with more trustworthy identifications. Just as a higher (i.e., more conservative) criterion may be used in a court of law by attaching less weight to lowconfidence identifications, one can do the same with data collected in the laboratory. The use of a slightly more conservative criterion, which is achieved by treating the lowest-confidence identifications as effective nonidentifications, generates the next point to the left on the ROC. For both procedures, this new ROC point is associated with a higher probative value than its neighbor to the right. The use of an ever more conservative criterion generates additional points to the left on the ROC, each of which is associated with a higher probative value than the last. It is not known which ROC point is the most relevant to a court of law, nor which probative value, but it seems certain that the right-most point (which includes low-confidence identifications) is the least relevant. 510934 CDPXXX10.1177/0963721413510934Wixted et al.Sequential Lineup Policy—A Response to <xref ref-type="bibr" rid="bibr8-0963721413510934">Wells (2013)</xref> research-article2014...

    [...]


Journal ArticleDOI
Abstract: Eyewitness identification is a pivotal issue in applied research because, in practice, a correct identification can help to remove a dangerous criminal from society, but a false identification can lead to the erroneous conviction of an innocent suspect. Consequently, psychologists have tried to ascertain the best procedures for collecting identification evidence, evaluating them using measures based on the ratio of correct to false identification rates. Unfortunately, ratio-based measures are ambiguous because they change systematically as a function of a witness's willingness to choose. In other words, a measure thought to index discriminability is instead fully confounded with response bias. A better method involves constructing receiver operating characteristic (ROC) curves. Using ROC curves, researchers can trace out discriminability across levels of response bias for each procedure. We illustrate the shortcomings of ratio- based measures and demonstrate why ROC analysis is required. In recent studies, researchers comparing simultaneous and sequential lineup procedures using ROC analyses have provided no evidence for the sequential superiority effect and instead have shown that the simultaneous procedure may be diagnostically superior. It is not yet clear which lineup procedure will prove to be generally superior, but it is clear that ROC analysis is the only way to make that determination.

102 citations


Journal ArticleDOI
Abstract: Showups (a one-person identification) were compared to both simultaneous and sequential lineups that varied in lineup fairness and the position of the suspect in the lineup. We reanalyzed data from a study by Gronlund, Carlson, Dailey, and Goodsell (2009) , which included simultaneous and sequential lineups, and using the same stimuli and procedures, collected new data using showup identifications. Performance was compared using ROC analysis, which is superior to traditional measures such as correct and false identification rates, and probative value measures. ROC analysis showed that simultaneous lineups consistently produced more accurate identification evidence than showups, but sequential lineups were sometimes no more accurate than showups, and were never more accurate than simultaneous lineups. These results supported prior suppositions regarding the suggestiveness of showups, revealed a misconception about the superiority of sequential lineups, and demonstrated why eyewitness identification procedures need to be evaluated using ROC analyses.

78 citations


"Policy Regarding the Sequential Lin..." refers result in this paper

  • ...Just as we have illustrated in Figure 1 with hypothetical data, the first three published studies using ROC analysis found that simultaneous lineups yield significantly higher discriminability than sequential lineups (Dobolyi & Dodson, in press; Gronlund et al., 2012; Mickes, Flowe, & Wixted, 2012)....

    [...]


Journal ArticleDOI
TL;DR: Overall, it is observed that sequential lineups are both less accurate and produce higher confidence false identifications than do simultaneous lineups, and this data argue for increased scrutiny and possibly a wholesale reevaluation of this lineup format.
Abstract: Confidence judgments for eyewitness identifications play an integral role in determining guilt during legal proceedings. Past research has shown that confidence in positive identifications is strongly associated with accuracy. Using a standard lineup recognition paradigm, we investigated accuracy using signal detection and ROC analyses, along with the tendency to choose a face with both simultaneous and sequential lineups. We replicated past findings of reduced rates of choosing with sequential as compared to simultaneous lineups, but notably found an accuracy advantage in favor of simultaneous lineups. Moreover, our analysis of the confidence-accuracy relationship revealed two key findings. First, we observed a sequential mistaken identification overconfidence effect: despite an overall reduction in false alarms, confidence for false alarms that did occur was higher with sequential lineups than with simultaneous lineups, with no differences in confidence for correct identifications. This sequential mistaken identification overconfidence effect is an expected byproduct of the use of a more conservative identification criterion with sequential than with simultaneous lineups. Second, we found a steady drop in confidence for mistaken identifications (i.e., foil identifications and false alarms) from the first to the last face in sequential lineups, whereas confidence in and accuracy of correct identifications remained relatively stable. Overall, we observed that sequential lineups are both less accurate and produce higher confidence false identifications than do simultaneous lineups. Given the increasing prominence of sequential lineups in our legal system, our data argue for increased scrutiny and possibly a wholesale reevaluation of this lineup format.

57 citations