Multivariable regression models are powerful tools that are used frequently in studies of clinical outcomes. These models can use a mixture of categorical and continuous variables and can handle partially observed (censored) responses. However, uncritical application of modelling techniques can result in models that poorly fit the dataset at hand, or, even more likely, inaccurately predict outcomes on new subjects. One must know how to measure qualities of a model's fit in order to avoid poorly fitted or overfitted models. Measurement of predictive accuracy can be difficult for survival time data in the presence of censoring. We discuss an easily interpretable index of predictive discrimination as well as methods for assessing calibration of predicted survival probabilities. Both types of predictive accuracy should be unbiasedly validated using bootstrapping or cross-validation, before using predictions in a new data series. We discuss some of the hazards of poorly fitted and overfitted regression models and present one modelling strategy that avoids many of the problems discussed. The methods described are applicable to all regression models, but are particularly needed for binary, ordinal, and time-to-event outcomes. Methods are illustrated with a survival analysis in prostate cancer using Cox regression.

/pdf/multivariable-prognostic-models-issues-in-developing-models-49wjg4tzp4.pdf

Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors

Prognostic/Clinical Prediction Models: Multivariable Prognostic Models: Issues in Developing Models, Evaluating Assumptions and Adequacy, and Measuring and Reducing Errors

SUMMARY Multivariable regression models are powerful tools that are used frequently in studies of clinical outcomes. These models can use a mixture of categorical and continuous variables and can handle partially observed (censored) responses. However, uncritical application of modelling techniques can result in models that poorly fit the dataset at hand, or, even more likely, inaccurately predict outcomes on new subjects. One must know how to measure qualities of a model's fit in order to avoid poorly fitted or overfitted models. Measurement of predictive accuracy can be difficult for survival time data in the presence of censoring. We discuss an easily interpretable index of predictive discrimination as well as methods for assessing calibration of predicted survival probabilities. Both types of predictive accuracy should be unbiasedly validated using bootstrapping or cross-validation, before using predictions in a new data series. We discuss some of the hazards of poorly fitted and overfitted regression models and present one modelling strategy that avoids many of the problems discussed. The methods described are applicable to all regression models, but are particularly needed for binary, ordinal, and time-to-event outcomes. Methods are illustrated with a survival analysis in prostate cancer using Cox regression. Accurate estimation of patient prognosis is important for many reasons. First, prognostic estimates can be used to inform the patient about likely outcomes of her disease. Second, the physician can use estimates of prognosis as a guide for ordering additional tests and selecting appropriate therapies. Third, prognostic assessments are useful in the evaluation of technologies; prognostic estimates derived both with and without using the results of a given test can be compared to measure the incremental prognostic information provided by that test over what is provided by prior information.' Fourth, a researcher may want to estimate the effect of a single factor (for example, treatment given) on prognosis in an observational study in which many uncontrolled confounding factors are also measured. Here the simultaneous effects of the uncontrolled variables must be controlled (held constant mathematically if using a regression model) so that the effect of the factor of interest can be more purely estimated. An analysis of how variables (especially continuous ones) affect the patient outcomes of interest is necessary to

/pdf/tutorial-in-biostatistics-multivariable-prognostic-models-4auy6032o6.pdf

Tutorial in biostatistics multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors

Scoring rules assess the quality of probabilistic forecasts, by assigning a numerical score based on the predictive distribution and on the event or value that materializes. A scoring rule is proper if the forecaster maximizes the expected score for an observation drawn from the distributionF if he or she issues the probabilistic forecast F, rather than G ≠ F. It is strictly proper if the maximum is unique. In prediction problems, proper scoring rules encourage the forecaster to make careful assessments and to be honest. In estimation problems, strictly proper scoring rules provide attractive loss and utility functions that can be tailored to the problem at hand. This article reviews and develops the theory of proper scoring rules on general probability spaces, and proposes and discusses examples thereof. Proper scoring rules derive from convex functions and relate to information measures, entropy functions, and Bregman divergences. In the case of categorical variables, we prove a rigorous version of the ...

/pdf/strictly-proper-scoring-rules-prediction-and-estimation-28cdd1b5ww.pdf

Strictly Proper Scoring Rules, Prediction, and Estimation

A review is presented of the book “Heuristics and Biases: The Psychology of Intuitive Judgment,” edited by Thomas Gilovich, Dale Griffin, and Daniel Kahneman.

Heuristics and Biases: The Psychology of Intuitive Judgment

Verification of forecasts expressed in terms of probability

Superposed epoch analyses, based on solar sector boundary crossings as key times and the Vorticity Area Index as the response variable, are tested for significance using both parametric and randomization techniques. We conclude from a comparison of these techniques that the randomization procedure leads to markedly different results from those obtained from parametric tests. In particular, the results are strongly affected by the modest skewness of the Vorticity Area Index distribution.

A critique of the superposed epoch analysis method: its application to solar-weather relations.

This paper considers the examination of possible differences in monthly sea-level pressure patterns, The satisfactory examination of such differences requires appropriate multi-response parametric methods based on unknown multivariate distributions (i.e., an appropriate parametric technique is probably non-existent). In order to avoid the likely insurmountable difficulties involving parametric methods, the application of multi-response permutation procedures (MRPP) is suggested as an appropriate approach for the examination of such differences.

Application of Multi-Response Permutation Procedures for Examining Seasonal Changes in Monthly Mean Sea-Level Pressure Patterns

Recent investigations indicate that the quasi-biennial oscillation (QBO) in stratospheric winds may be related to variations in the tropospheric circulation in middle and high latitudes Although the QBO was noticed years ago in a number of worldwide atmospheric phenomena, it is still not well understood A logical question arises as to whether the oscillation might be a result of the annual forcing by the solar heating, since it is well known that in some nonlinear systems subharmonic oscillations can be produced of order one-half the frequency of the exciting force A conceptual model is presented to show how a two-year oscillation could be produced by a negative feedback process acting on a two-state system, ie, winter and summer states Even for a relatively weak biennial oscillation with stochastic elements involved, the results show that rather strong links must exist in the chain or feedback loop, indicating potential predictability for periods of six months or more Support for the conc

The Quasi-Biennial Oscillation and Feedback Processes in the Atmosphere-Ocean-Earth System

A reanalysis of the Climax I and II experiments is described. The concern prompting this reanalysis is a suggestion arising from Colorado State University analyses of extended area effects. Those analyses suggested a regionwide pattern of precipitation that, by chance, may have favored the randomly selected seeding days for some of the important meteorological partitions used in earlier analyses. In order to address this concern, this reanalysis employs excellent covariate relationships developed before the initiation of Climax II and which account for over half of the target variability for most meteorological partitions of major interest (e.g., warm 500 mb temperatures and southwest 700 mb wind directions). The statistical evidence of seeding-induced increases for this reanalysis is generally much stronger than in the previous analysis, which did not utilize covariates. For example, the joint one-sided Wilcoxon test statistic P-value for testing the null hypothesis that seeding did not induce a...

Glenn W. Brier

Papers

Verification of forecasts expressed in terms of probability

A critique of the superposed epoch analysis method: its application to solar-weather relations.

Application of Multi-Response Permutation Procedures for Examining Seasonal Changes in Monthly Mean Sea-Level Pressure Patterns

The Quasi-Biennial Oscillation and Feedback Processes in the Atmosphere-Ocean-Earth System

A Statistical Reanalysis of the Replicated Climax I and II Wintertime Orographic Cloud Seeding Experiments