https://ccrma.stanford.edu/workshops/mir2009/references/ROCintro.pdf

An introduction to ROC analysis

Methods of evaluating and comparing the performance of diagnostic tests are of increasing importance as new tests are developed and marketed. When a test is based on an observed variable that lies on a continuous or graded scale, an assessment of the overall value of the test can be made through the use of a receiver operating characteristic (ROC) curve. The curve is constructed by varying the cutpoint used to determine which values of the observed variable will be considered abnormal and then plotting the resulting sensitivities against the corresponding false positive rates. When two or more empirical curves are constructed based on tests performed on the same individuals, statistical analysis on differences between curves must take into account the correlated nature of the data. This paper presents a nonparametric approach to the analysis of areas under correlated ROC curves, by using the theory on generalized U-statistics to generate an estimated covariance matrix.

Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach.

https://www.cs.princeton.edu/~schapire/papers/ecolmod.pdf

Maximum entropy modeling of species geographic distributions

A novel behavioural screening questionnaire, the Strengths and Difficulties Questionnaire (SDQ), was administered along with Rutter questionnaires to parents and teachers of 403 children drawn from dental and psychiatric clinics. Scores derived from the SDQ and Rutter questionnaires were highly correlated; parent-teacher correlations for the two sets of measures were comparable or favoured the SDQ. The two sets of measures did not differ in their ability to discriminate between psychiatric and dental clinic attenders. These preliminary findings suggest that the SDQ functions as well as the Rutter questionnaires while offering the following additional advantages: a focus on strengths as well as difficulties; better coverage of inattention, peer relationships, and prosocial behaviour; a shorter format; and a single form suitable for both parents and teachers, perhaps thereby increasing parent-teacher correlations.

https://acamh.onlinelibrary.wiley.com/doi/pdf/10.1111/j.1469-7610.1997.tb01545.x

The Strengths and Difficulties Questionnaire: A Research Note

XI. STRATEGIES FOR IMPROVING DIABETES CARE D iabetes is a chronic illness that requires continuing medical care and patient self-management education to prevent acute complications and to reduce the risk of long-term complications. Diabetes care is complex and requires that many issues, beyond glycemic control, be addressed. A large body of evidence exists that supports a range of interventions to improve diabetes outcomes. These standards of care are intended to provide clinicians, patients, researchers, payors, and other interested individuals with the components of diabetes care, treatment goals, and tools to evaluate the quality of care. While individual preferences, comorbidities, and other patient factors may require modification of goals, targets that are desirable for most patients with diabetes are provided. These standards are not intended to preclude more extensive evaluation and management of the patient by other specialists as needed. For more detailed information, refer to Bode (Ed.): Medical Management of Type 1 Diabetes (1), Burant (Ed): Medical Management of Type 2 Diabetes (2), and Klingensmith (Ed): Intensive Diabetes Management (3). The recommendations included are diagnostic and therapeutic actions that are known or believed to favorably affect health outcomes of patients with diabetes. A grading system (Table 1), developed by the American Diabetes Association (ADA) and modeled after existing methods, was utilized to clarify and codify the evidence that forms the basis for the recommendations. The level of evidence that supports each recommendation is listed after each recommendation using the letters A, B, C, or E.

Standards of Medical Care in Diabetes

A representation and interpretation of the area under a receiver operating characteristic (ROC) curve obtained by the "rating" method, or by mathematical predictions based on patient characteristics, is presented. It is shown that in such a setting the area represents the probability that a randomly chosen diseased subject is (correctly) rated or ranked with greater suspicion than a randomly chosen non-diseased subject. Moreover, this probability of a correct ranking is the same quantity that is estimated by the already well-studied nonparametric Wilcoxon statistic. These two relationships are exploited to (a) provide rapid closed-form expressions for the approximate magnitude of the sampling variability, i.e., standard error that one uses to accompany the area under a smoothed ROC curve, (b) guide in determining the size of the sample required to provide a sufficiently reliable estimate of this area, and (c) determine how large sample sizes should be to ensure that one can statistically detect difference...

The meaning and use of the area under a receiver operating characteristic (ROC) curve.

Receiver operating characteristic (ROC) curves are used to describe and compare the performance of diagnostic technology and diagnostic algorithms. This paper refines the statistical comparison of the areas under two ROC curves derived from the same set of patients by taking into account the correlation between the areas that is induced by the paired nature of the data. The correspondence between the area under an ROC curve and the Wilcoxon statistic is used and underlying Gaussian distributions (binormal) are assumed to provide a table that converts the observed correlations in paired ratings of images into a correlation between the two ROC areas. This between-area correlation can be used to reduce the standard error (uncertainty) about the observed difference in areas. This correction for pairing, analogous to that used in the paired t-test, can produce a considerable increase in the statistical sensitivity (power) of the comparison. For studies involving multiple readers, this method provides a measure...

/pdf/a-method-of-comparing-the-areas-under-receiver-operating-50lpcksrsi.pdf

A method of comparing the areas under receiver operating characteristic curves derived from the same cases.

The method of generalized estimating equations (GEE) is often used to analyze longitudinal and other correlated response data, particularly if responses are binary. However, few descriptions of the method are accessible to epidemiologists. In this paper, the authors use small worked examples and one real data set, involving both binary and quantitative response data, to help end-users appreciate the essence of the method. The examples are simple enough to see the behind-the-scenes calculations and the essential role of weighted observations, and they allow nonstatisticians to imagine the calculations involved when the GEE method is applied to more complex multivariate data.

https://faculty.washington.edu/heagerty/Courses/VA-longitudinal/private/Hanley-2003AJE.pdf

Statistical Analysis of Correlated Data Using Generalized Estimating Equations: An Orientation

PHYSICIANS frequently must provide patients with estimates of the risk of a particular medical procedure or of the probability of a specific health outcome. To do so, they may use data from their own experiences as well as those available in the literature. A general goal is to locate the estimate within a fairly narrow range. When the procedure or outcome of concern is relatively uncommon, precise estimation, although desirable, may be difficult or impossible. Moreover, when the only data available come from a study in which none of the events of concern actually occurred—ie, a study in which a zero numerator is reported—making inferences seems to be particularly problematic. Because the occurrence of "no events" seems to be viewed as very different both quantitatively and qualitatively from the occurrence of one or more events, it is useful to look into someof the statistical and psychological issues that influence the

https://jamanetwork.com/HttpHandlers/ArticlePdfHandler.ashx?journal=JAMA&articleId=385438&pdfFileName=jama_249_13_031.pdf

If Nothing Goes Wrong, Is Everything All Right?: Interpreting Zero Numerators

Context: The appropriate therapy for men with clinically localized prostate cancer is uncertain. A recent study suggested an increasing prostate cancer mortality rate for men who are alive more than 15 years following diagnosis. Objective: To estimate 20-year survival based on a competing risk analysis of men who were diagnosed with clinically localized prostate cancer and treated with observation or androgen withdrawal therapy alone, stratified by age at diagnosis and histological findings. Design, Setting, and Patients: A retrospective population-based cohort study using Connecticut Tumor Registry data supplemented by hospital record and histology review of 767 men aged 55 to 74 years with clinically localized prostate cancer diagnosed between January 1, 1971, and December 31, 1984. Patients were treated with either observation or immediate or delayed androgen withdrawal therapy, with a median observation of 24 years. Main Outcome Measures: Probability of mortality from prostate cancer or other competing medical conditions, given a patient’s age at diagnosis and tumor grade. Results: The prostate cancer mortality rate was 33 per 1000 person-years during the first 15 years of follow-up (95% confidence interval [CI], 28-38) and 18 per 1000 personyears after 15 years of follow-up (95% CI, 10-29). The mortality rates for these 2 follow-up periods were not statistically different, after adjusting for differences in tumor histology (rate ratio, 1.1; 95% CI, 0.6-1.9). Men with low-grade prostate cancers have a minimal risk of dying from prostate cancer during 20 years of follow-up (Gleason score of 2-4, 6 deaths per 1000 person-years; 95% CI, 2-11). Men with high-grade prostate cancers have a high probability of dying from prostate cancer within 10 years of diagnosis (Gleason score of 8-10, 121 deaths per 1000 person-years; 95% CI, 90-156). Men with Gleason score of 5 or 6 tumors have an intermediate risk of prostate cancer death. Conclusion: The annual mortality rate from prostate cancer appears to remain stable after 15 years from diagnosis, which does not support aggressive treatment for localized low-grade prostate cancer.

/pdf/20-year-outcomes-following-conservative-management-of-8f4fbs2cvt.pdf

James A. Hanley

Papers

The meaning and use of the area under a receiver operating characteristic (ROC) curve.

A method of comparing the areas under receiver operating characteristic curves derived from the same cases.

Statistical Analysis of Correlated Data Using Generalized Estimating Equations: An Orientation

If Nothing Goes Wrong, Is Everything All Right?: Interpreting Zero Numerators

20-year outcomes following conservative management of clinically localized prostate cancer.