https://ccrma.stanford.edu/workshops/mir2009/references/ROCintro.pdf

An introduction to ROC analysis

Methods of evaluating and comparing the performance of diagnostic tests are of increasing importance as new tests are developed and marketed. When a test is based on an observed variable that lies on a continuous or graded scale, an assessment of the overall value of the test can be made through the use of a receiver operating characteristic (ROC) curve. The curve is constructed by varying the cutpoint used to determine which values of the observed variable will be considered abnormal and then plotting the resulting sensitivities against the corresponding false positive rates. When two or more empirical curves are constructed based on tests performed on the same individuals, statistical analysis on differences between curves must take into account the correlated nature of the data. This paper presents a nonparametric approach to the analysis of areas under correlated ROC curves, by using the theory on generalized U-statistics to generate an estimated covariance matrix.

Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach.

https://www.cs.princeton.edu/~schapire/papers/ecolmod.pdf

Maximum entropy modeling of species geographic distributions

A novel behavioural screening questionnaire, the Strengths and Difficulties Questionnaire (SDQ), was administered along with Rutter questionnaires to parents and teachers of 403 children drawn from dental and psychiatric clinics. Scores derived from the SDQ and Rutter questionnaires were highly correlated; parent-teacher correlations for the two sets of measures were comparable or favoured the SDQ. The two sets of measures did not differ in their ability to discriminate between psychiatric and dental clinic attenders. These preliminary findings suggest that the SDQ functions as well as the Rutter questionnaires while offering the following additional advantages: a focus on strengths as well as difficulties; better coverage of inattention, peer relationships, and prosocial behaviour; a shorter format; and a single form suitable for both parents and teachers, perhaps thereby increasing parent-teacher correlations.

https://acamh.onlinelibrary.wiley.com/doi/pdf/10.1111/j.1469-7610.1997.tb01545.x

The Strengths and Difficulties Questionnaire: A Research Note

https://caltcm.memberclicks.net/assets/documents/acc_aha_chf_2013_guidelines.pdf

2013 ACCF/AHA Guideline for the Management of Heart Failure: A Report of the American College of Cardiology Foundation/American Heart Association Task Force on Practice Guidelines

A representation and interpretation of the area under a receiver operating characteristic (ROC) curve obtained by the "rating" method, or by mathematical predictions based on patient characteristics, is presented. It is shown that in such a setting the area represents the probability that a randomly chosen diseased subject is (correctly) rated or ranked with greater suspicion than a randomly chosen non-diseased subject. Moreover, this probability of a correct ranking is the same quantity that is estimated by the already well-studied nonparametric Wilcoxon statistic. These two relationships are exploited to (a) provide rapid closed-form expressions for the approximate magnitude of the sampling variability, i.e., standard error that one uses to accompany the area under a smoothed ROC curve, (b) guide in determining the size of the sample required to provide a sufficiently reliable estimate of this area, and (c) determine how large sample sizes should be to ensure that one can statistically detect difference...

The meaning and use of the area under a receiver operating characteristic (ROC) curve.

Receiver operating characteristic (ROC) curves are used to describe and compare the performance of diagnostic technology and diagnostic algorithms. This paper refines the statistical comparison of the areas under two ROC curves derived from the same set of patients by taking into account the correlation between the areas that is induced by the paired nature of the data. The correspondence between the area under an ROC curve and the Wilcoxon statistic is used and underlying Gaussian distributions (binormal) are assumed to provide a table that converts the observed correlations in paired ratings of images into a correlation between the two ROC areas. This between-area correlation can be used to reduce the standard error (uncertainty) about the observed difference in areas. This correction for pairing, analogous to that used in the paired t-test, can produce a considerable increase in the statistical sensitivity (power) of the comparison. For studies involving multiple readers, this method provides a measure...

/pdf/a-method-of-comparing-the-areas-under-receiver-operating-50lpcksrsi.pdf

A method of comparing the areas under receiver operating characteristic curves derived from the same cases.

We investigated how variations in the way information is presented to patients influence their choices between alternative therapies. Data were presented summarizing the results of surgery and radiation therapy for lung cancer to 238 ambulatory patients with different chronic medical conditions and to 491 graduate students and 424 physicians. We asked the subjects to imagine that they had lung cancer and to choose between the two therapies on the basis of both cumulative probabilities and life-expectancy data. Different groups of respondents received input data that differed only in whether or not the treatments were identified and whether the outcomes were framed in terms of the probability of living or the probability of dying. In all three populations, the attractiveness of surgery, relative to radiation therapy, was substantially greater when the treatments were identified rather than unidentified, when the information consisted of life expectancy rather than cumulative probability, and when the problem was framed in terms of the probability of living rather than in terms of the probability of dying. We suggest that an awareness of these effects among physicians and patients could help reduce bias and improve the quality of medical decision making.

On the Elicitation of Preferences for Alternative Therapies

In this review of the theoretical and empirical work on patient satisfaction with care, the most consistent finding is that the characteristics of providers or organizations that result in more "personal" care are associated with higher levels of satisfaction. Some studies suggest that more personal care will result in better communication and more patient involvement, and hence better quality of care, but the data on these issues are weak and inconsistent. Further research is needed to measure specific aspects of medical care and the ways in which patient reports can complement other sources of information about quality. In addition, more research on the determinants of satisfaction and the relationship between quality and satisfaction among hospitalized patients is recommended.

Patient satisfaction as an indicator of quality care.

The value of a diagnostic test lies in its ability to detect patients with disease (its sensitivity) and to exclude patients without disease (its specificity). For tests with binary outcomes, these measures are fixed. For tests with a continuous scale of values, various cutoff points can be selected to adjust the sensitivity and specificity of the test to conform with the physician's goals. Principles of statistical decision theory and information theory suggest technics for objectively determining these cutoff points, depending upon whether the physician is concerned with health costs, with financial costs, or with the information content of the test.

Barbara J. McNeil

Papers

The meaning and use of the area under a receiver operating characteristic (ROC) curve.

A method of comparing the areas under receiver operating characteristic curves derived from the same cases.

On the Elicitation of Preferences for Alternative Therapies

Patient satisfaction as an indicator of quality care.

Primer on Certain Elements of Medical Decision Making