https://ccrma.stanford.edu/workshops/mir2009/references/ROCintro.pdf

An introduction to ROC analysis

A review is presented of the book “Heuristics and Biases: The Psychology of Intuitive Judgment,” edited by Thomas Gilovich, Dale Griffin, and Daniel Kahneman.

Heuristics and Biases: The Psychology of Intuitive Judgment

Much research in the last two decades has demon- strated that human responses deviate from the performance deemed normative according to various models of decision mak- ing and rational judgment (e.g., the basic axioms of utility theory). This gap between the normative and the descriptive can be inter- preted as indicating systematic irrationalities in human cognition. However, four alternative interpretations preserve the assumption that human behavior and cognition is largely rational. These posit that the gap is due to (1) performance errors, (2) computational limitations, (3) the wrong norm being applied by the experi- menter, and (4) a different construal of the task by the subject. In the debates about the viability of these alternative explanations, attention has been focused too narrowly on the modal response. In a series of experiments involving most of the classic tasks in the heuristics and biases literature, we have examined the implica- tions of individual differences in performance for each of the four explanations of the normative/descriptive gap. Performance er- rors are a minor factor in the gap; computational limitations un- derlie non-normative responding on several tasks, particularly those that involve some type of cognitive decontextualization. Un- expected patterns of covariance can suggest when the wrong norm is being applied to a task or when an alternative construal of the task should be considered appropriate.

https://www.cambridge.org/core/services/aop-cambridge-core/content/view/S0140525X00003435

Individual differences in reasoning: Implications for the rationality debate?

In this review, we examine the oft-made claim that peer-relationship difficulties in childhood predict serious adjustment problems in later life. The article begins with a framework for conceptualizing and assessing children's peer difficulties and with a discussion of conceptual and methodological issues in longitudinal risk research. Following this, three indexes of problematic peer relationships (acceptance, aggressiveness, and shyness/withdrawal) are evaluated as predictors of three later outcomes (dropping out of school, criminality, and psychcpathology). The relation between peer difficulties and later maladjustment is examined in terms of both the consistency and strength of prediction. A review and analysis of the literature indicates general support for the hypothesis that children with poor peer adjustment are at risk for later life difficulties. Support is clearest for the outcomes of dropping out and criminality. It is also clearest for low acceptance and aggressiveness as predictors, whereas a link between shyness/withdrawal and later maladjustment has not yet been adequately tested. The article concludes with a critical discussion of the implicit models that have guided past research in this area and a set of recommendations for the next generation of research on the risk

Peer relations and later personal adjustment: Are low-accepted children at risk?

Modern theories in cognitive psychology and neuroscience indicate that there are two fundamental ways in which human beings comprehend risk. The analytic system uses algorithms and normative rules, such as probability calculus, formal logic, and risk assessment. It is relatively slow, effortful, and requires conscious control. The experiential system is intuitive, fast, mostly automatic, and not very accessible to conscious awareness. The experiential system enabled human beings to survive during their long period of evolution and remains today the most natural and most common way to respond to risk. It relies on images and associations, linked by experience to emotion and affect (a feeling that something is good or bad). This system represents risk as a feeling that tells us whether it's safe to walk down this dark street or drink this strange-smelling water. Proponents of formal risk analysis tend to view affective responses to risk as irrational. Current wisdom disputes this view. The rational and the experiential systems operate in parallel and each seems to depend on the other for guidance. Studies have demonstrated that analytic reasoning cannot be effective unless it is guided by emotion and affect. Rational decision making requires proper integration of both modes of thought. Both systems have their advantages, biases, and limitations. Now that we are beginning to understand the complex interplay between emotion and reason that is essential to rational behavior, the challenge before us is to think creatively about what this means for managing risk. On the one hand, how do we apply reason to temper the strong emotions engendered by some risk events? On the other hand, how do we infuse needed "doses of feeling" into circumstances where lack of experience may otherwise leave us too "coldly rational"? This article addresses these important questions.

https://papers.ssrn.com/sol3/Delivery.cfm?abstractid=931686

Risk As Analysis and Risk As Feelings: Some Thoughts About Affect, Reason, Risk, and Rationality

Scientific Authority: The Breast Implant Litigation and Beyond

The Causes of Violence

Scientifically valid instruments are being used for the first time to assess an individual’s risk of violence in criminal sentencing and in the civil commitment of mental patients and sexual predators. Risk factors on these ٭ Henry and Grace Doherty Professor and Class of 1941 Research Professor, University of Virginia School of Law. I thank my colleagues Linda Monahan, Richard Bonnie, Daniel Ortiz, Kenneth Abraham, David Faigman, Stephen Morse, Deborah Denno, Lawrence Fitch, Christopher Slobogin, Michael Tonry, Shari Diamond, and Janice Nadler, and the participants in a University of Virginia and a Northwestern University workshop for their insightful discussions of issues raised in this piece, and my students Gregory Walter, Brian Kinney, Amy Napier, and Mike Gorokhovich for their excellent research assistance.

/pdf/forecasting-harm-the-law-and-science-of-risk-assessment-47259tiuyb.pdf

Forecasting Harm: The Law and Science of Risk Assessment among Prisoners, Predators, and Patients

INTRODUCTIONA mainstay of the law of evidence is the distinction between admissibility and weight. Judges are tasked with the responsibility of determining whether proffered evidence is admissible and, if it is, jurors must decide what weight to give it.1 This division of responsibility holds across all evidence rules, from basic relevance to hearsay. Hence, a "dying declaration" that is hearsay is only admissible if the judge determines, among other things, that the statement was made by a declarant "while believing that the declarant's death was imminent."2 If admitted, the weight, if any, that should be accorded the particular dying declaration is up to the jury to decide. The predicate issue of whether the defendant believed death was imminent is called a "preliminary fact,"3 one that the judge must determine by a preponderance of the evidence.4 This prerogative belongs to the judge on the assumption that juries are likely to attribute significance to an out-of-court statement even if they were to find that death was not imminent.5This division of responsibilities between judge and jury also applies to scientific evidence presented by experts.6 Indeed, because the distinction between admissibility and weight is endemic to the law of evidence, in theory the issue has existed for scientific evidence since experts were first introduced at trial in the nineteenth century.7 Yet the distinction received little attention until the United States Supreme Court's landmark decision in Daubert v. Merrell Dow Pharmaceuticals, Inc.8 The Daubert Court, interpreting Federal Rule of Evidence 702, held that judges are "gatekeepers" and obligated to determine whether the methods and principles underlying proffered expert testimony are-more likely than not-reliable and valid.9 In other words, the Court treated the "evidentiary reliability" of the scientific evidence proffered in the case as a preliminary fact and thus within the judge's purview to determine. Consistent with the preliminary fact rule in other types of cases, the rationale for this requirement is that requiring juries to parse unvetted scientific information and disregard those aspects of it they consider suspect is likely to lead to ill-considered verdicts.10In contrast to the usual preliminary fact determination, however, the complex nature of scientific evidence has created substantial confusion among courts about just where the judge's authority to decide admissibility ends and the jury's responsibility to assess weight begins. The key variable in the latter setting, according to Daubert's original formulation, is whether the fact is a "conclusion" or something else. Under Daubert, in assessing admissibility, the judge's "focus . . . must be solely on principles and methodology, not on the conclusions that they generate."11 This distinction between methodology and principles on the one hand and case-specific conclusions on the other (which we will refer to as the methodology- conclusions distinction for ease of reference)12 has assumed major significance. While the Court abandoned the distinction just four years later in General Electric Co. v. Joiner,13 and while the amendments to Rule 702 three years after Joiner made no mention of it,14 many courts continue to put considerable emphasis on whether the preliminary fact is about "methodology" or instead describes "conclusions."15 Only a minority of courts have required that the judge preliminarily determine that the expert's conclusion was reliably reached using a reliable methodology.16 Most courts hold that the judge's sole concern is whether the expert followed an acceptable methodology, and other decisions have even punted some types of methodological issues to the jury.17In this Article, we argue that the methodology-conclusions distinction has no principled basis in science and thus should have none in law. Since the distinction does not align with the nature of the evidence that scientists proffer in court, it is destined to fail and should be explicitly jettisoned. …

/pdf/gatekeeping-science-using-the-structure-of-scientific-50a74ulngm.pdf

John Monahan

Papers

Scientific Authority: The Breast Implant Litigation and Beyond

The Causes of Violence

Forecasting Harm: The Law and Science of Risk Assessment among Prisoners, Predators, and Patients

Gatekeeping Science: Using the Structure of Scientific Research to Distinguish between Admissibility and Weight in Expert Testimony

The Prediction of Violent Behavior: Developments in Psychology and Law