Scoring rules assess the quality of probabilistic forecasts, by assigning a numerical score based on the predictive distribution and on the event or value that materializes. A scoring rule is proper if the forecaster maximizes the expected score for an observation drawn from the distributionF if he or she issues the probabilistic forecast F, rather than G ≠ F. It is strictly proper if the maximum is unique. In prediction problems, proper scoring rules encourage the forecaster to make careful assessments and to be honest. In estimation problems, strictly proper scoring rules provide attractive loss and utility functions that can be tailored to the problem at hand. This article reviews and develops the theory of proper scoring rules on general probability spaces, and proposes and discusses examples thereof. Proper scoring rules derive from convex functions and relate to information measures, entropy functions, and Bregman divergences. In the case of categorical variables, we prove a rigorous version of the ...

/pdf/strictly-proper-scoring-rules-prediction-and-estimation-28cdd1b5ww.pdf

Strictly Proper Scoring Rules, Prediction, and Estimation

Data snooping occurs when a given set of data is used more than once for purposes of inference or model selection When such data reuse occurs, there is always the possibility that any satisfactory results obtained may simply be due to chance rather than to any merit inherent in the method yielding the results This problem is practically unavoidable in the analysis of time-series data, as typically only a single history measuring a given phenomenon of interest is available for analysis It is widely acknowledged by empirical researchers that data snooping is a dangerous practice to be avoided, but in fact it is endemic The main problem has been a lack of sufficiently simple practical methods capable of assessing the potential dangers of data snooping in a given situation Our purpose here is to provide such methods by specifying a straightforward procedure for testing the null hypothesis that the best model encountered in a specification search has no predictive superiority over a given benchmark model This permits data snooping to be undertaken with some degree of confidence that one will not mistake results that could have been generated by chance for genuinely good results

A Reality Check for Data Snooping

Since the early 1980s, a bewildering array of methods for constructing bootstrap confidence intervals have been proposed. In this article, we address the following questions. First, when should bootstrap confidence intervals be used. Secondly, which method should be chosen, and thirdly, how should it be implemented. In order to do this, we review the common algorithms for resampling and methods for constructing bootstrap confidence intervals, together with some less well known ones, highlighting their strengths and weaknesses. We then present a simulation study, a flow chart for choosing an appropriate method and a survival analysis example.

Bootstrap confidence intervals : when, which, what? A practical guide for medical statisticians

Model Selection and Inference

This paper takes a broad, pragmatic view of statistical inference to include all aspects of model formulation. The estimation of model parameters traditionally assumes that a model has a prespecified known form and takes no account of possible uncertainty regarding the model structure. This implicitly assumes the existence of a 'true' model, which many would regard as a fiction. In practice model uncertainty is a fact of life and likely to be more serious than other sources of uncertainty which have received far more attention from statisticians. This is true whether the model is specified on subject-matter grounds or, as is increasingly the case, when a model is formulated, fitted and checked on the same data set in an iterative, interactive way. Modern computing power allows a large number of models to be considered and data-dependent specification searches have become the norm in many areas of statistics. The term data mining may be used in this context when the analyst goes to great lengths to obtain a good fit. This paper reviews the effects of model uncertainty, such as too narrow prediction intervals, and the non-trivial biases in parameter estimates which can follow data-based modelling. Ways of assessing and overcoming the effects of model uncertainty are discussed, including the use of simulation and resampling methods, a Bayesian model averaging approach and collecting additional data wherever possible. Perhaps the main aim of the paper is to ensure that statisticians are aware of the problems and start addressing the issues even if there is no simple, general theoretical fix.

https://rss.onlinelibrary.wiley.com/doi/pdf/10.2307/2983440

Model uncertainty, data mining and statistical inference

Potscher (1991, Econometric Theory7, 163–181) has recently considered the question of how the use of a model selection procedure affects the asymptotic distribution of parameter estimators and related statistics. An important potential application of such results is to the generation of confidence regions for the parameters of interest. It is demonstrated that a great deal of care must be exercised in any attempt at such an application. We also consider the effect of model selection on prediction regions. It is demonstrated that the use of asymptotic results for the construction of prediction regions requires the same sort of care as the use of such results for the construction of confidence regions.

The Effect of Model Selection on Confidence Regions and Prediction Regions

We give a large-sample analysis of the minimal coverage probability of the usual confidence intervals for regression parameters when the underlying model is chosen by a “conservative” (or “overconsistent”) model selection procedure. We derive an upper bound for the large-sample limit minimal coverage probability of such intervals that applies to a large class of model selection procedures including the Akaike information criterion as well as various pretesting procedures. This upper bound can be used as a safeguard to identify situations where the actual coverage probability can be far below the nominal level. We illustrate that the (asymptotic) upper bound can be statistically meaningful even in rather small samples.

On the Large-Sample Minimal Coverage Probability of Confidence Intervals After Model Selection

We consider a linear regression model with regression parameters (θ1,...,θp) and error variance parameter σ2. Our aim is to find a confidence interval with minimum coverage probability 1 − α for a parameter of interest θ1 in the presence of nuisance parameters (θ2,...,θp,σ2). We consider two confidence intervals, the first of which is the standard confidence interval for θ1 with coverage probability 1 − α. The second confidence interval for θ1 is obtained after a variable selection procedure has been applied to θp. This interval is chosen to be as short as possible subject to the constraint that it has minimum coverage probability 1 − α. The confidence intervals are compared using a risk function that is defined as a scaled version of the expected length of the confidence interval. We show that, subject to certain conditions including that [(dimension of response vector) − p] is small, the second confidence interval is preferable to the first when we anticipate (without being certain) that |θp|/σ is small. This comparison of confidence intervals is shown to be mathematically equivalent to a corresponding comparison of prediction intervals.

Valid confidence intervals in regression after variable selection

Summary

It is very common in applied frequentist (“classical”) statistics to carry out a preliminary statistical (i.e. data-based) model selection by, for example, using preliminary hypothesis tests or minimizing AIC. This is usually followed by the inference of interest, using the same data, based on the assumption that the selected model had been given to us a priori. This assumption is false and it can lead to an inaccurate and misleading inference. We consider the important case that the inference of interest is a confidence region. We review the literature that shows that the resulting confidence regions typically have very poor coverage properties. We also briefly review the closely related literature that describes the coverage properties of prediction intervals after preliminary statistical model selection. A possible motivation for preliminary statistical model selection is a wish to utilize uncertain prior information in the inference of interest. We review the literature in which the aim is to utilize uncertain prior information directly in the construction of confidence regions, without requiring the intermediate step of a preliminary statistical model selection. We also point out this aim as a future direction for research.

Resume

En statistiques appliquees de l'approche frequentiste (“classique”), il est courant de proceder a une selection preliminaire du modele statistique (c'est-a-dire basee sur des donnees) en utilisant, par exemple, des tests preliminaires fondes sur des hypotheses ou en minimisant AIC. Ceci est generalement suivi par l'inference d'interet, ou les memes donnees sont utilisees, et qui suppose que le modele choisi nous avait ete donnea priori. Cette supposition est erronee et peut entrainer une inference inexacte et trompeuse. Nous examinons un cas primordial ou l'inference d'interet constitue une region de confiance. Nous etudions la documentation qui indique que les regions de confiance qui en resultent ont en principe des proprietes d'application reduites. Nous examinons egalement de maniere succincte les ecrits en etroite relation qui decrivent les proprietes d'application des intervalles de prediction apres la selection preliminaire du modele statistique. Il est possible que la motivation sous-tendant la selection preliminaire du modele statistique represente un desir d'utilizer des renseignements prealables incertains dans l'inference d'interet. Nous etudions la documentation ou l'objectif est d'utilizer des renseignements prealables incertains directement dans l'elaboration de regions de confiance, sans exiger de recourir a l'etape intermediaire de selection preliminaire du modele statistique. Nous precisons egalement que cet objectif constitue un axe de recherche future.

The Coverage Properties of Confidence Regions After Model Selection

Shibata (1981, Biometrika 68, 45–54) considers data-generating mechanisms belonging to a certain class of linear regressions with errors that are independent and identically normally distributed. He compares the variable selection criteria AIC (Akaike information criterion) and BIC (Bayesian information criterion) using the following type of comparison. For each fixed possible data–generating mechanism, these criteria are compared as the data length increases. The results of this comparison have been interpreted as meaning that, in the context of the data-generating mechanisms considered by Shibata, AIC is better than BIC for large data lengths. Shibata's comparison is pointwise in the space of data–generating mechanisms (as the data length increases). Such comparisons are potentially misleading. We consider a simple class of data-generating mechanisms satisfying Shibata's assumptions and carry out a different type of comparison. For each fixed data length (possibly large) we compare the variable selection criteria for every possible data-generating mechanism in this class. According to this comparison, for this class of data-generating mechanisms no matter how large the data length AIC is not better than BIC.

Paul Kabaila

Papers

The Effect of Model Selection on Confidence Regions and Prediction Regions

On the Large-Sample Minimal Coverage Probability of Confidence Intervals After Model Selection

Valid confidence intervals in regression after variable selection

The Coverage Properties of Confidence Regions After Model Selection

On variable selection in linear regression