Showing papers on "Brier score published in 2000"

Decomposition of the Continuous Ranked Probability Score for Ensemble Prediction Systems

Abstract: Some time ago, the continuous ranked probability score (CRPS) was proposed as a new verification tool for (probabilistic) forecast systems. Its focus is on the entire permissible range of a certain (weather) parameter. The CRPS can be seen as a ranked probability score with an infinite number of classes, each of zero width. Alternatively, it can be interpreted as the integral of the Brier score over all possible threshold values for the parameter under consideration. For a deterministic forecast system the CRPS reduces to the mean absolute error. In this paper it is shown that for an ensemble prediction system the CRPS can be decomposed into a reliability part and a resolution/uncertainty part, in a way that is similar to the decomposition of the Brier score. The reliability part of the CRPS is closely connected to the rank histogram of the ensemble, while the resolution/ uncertainty part can be related to the average spread within the ensemble and the behavior of its outliers. The usefulness of such a decomposition is illustrated for the ensemble prediction system running at the European Centre for Medium-Range Weather Forecasts. The evaluation of the CRPS and its decomposition proposed in this paper can be extended to systems issuing continuous probability forecasts, by realizing that these can be interpreted as the limit of ensemble forecasts with an infinite number of members.

A probability and decision-model analysis of PROVOST seasonal multi-model ensemble integrations

Abstract: A probabilistic analysis is made of seasonal ensemble integrations from the PROVOST project (PRediction Of climate Variations On Seasonal to interannual Time-scales), with emphasis on the Brier score and related Murphy decomposition, and the relative operating characteristic. To illustrate the significance of these results to potential users, results from the analysis of the relative operating characteristic are input to a simple decision model. The decision-model analysis is used to define a user-specific objective measure of the economic value of seasonal forecasts. The analysis is made for two simple meteorological forecast conditions or ‘events’, E, based on 850 hPa temperature. The ensemble integrations result from integrating four different models over the period 1979–93. For each model a set of 9-member ensembles is generated by running from consecutive analyses. Results from the Brier skill score analysis taken over all northern hemisphere grid points indicate that, whilst the skill of individual-model ensembles is only marginally higher than a probabilistic forecast of climatological frequencies, the multi-model ensemble is substantially more skilful than climatology. Both reliability and resolution are better for the multi-model ensemble than for the individual-model ensembles. This improvement arises both from the use of different models in the ensemble, and from the enhanced ensemble size obtained by combining individual-model ensembles; the latter reason was found to be the more important. Brier skill scores are higher for years in which there were moderate or strong El Nino Southern Oscillation (ENSO) events. Over Europe, only the multi-model ensembles showed skill over climatology. Similar conclusions are reached from an analysis of the relative operating characteristic. Results from the decision-model analysis show that the economic value of seasonal forecasts is strongly dependent on the cost, C, to the user of taking precautionary action against E, in relation to the potential loss, L, if precautionary action is not taken and E occurs. However, based on the multi-model ensemble data, the economic value can be as much as 50% of the value of a hypothetical perfect deterministic forecast. For the hemisphere as a whole, value is enhanced by restriction to ENSO years. It is shown that there is potential economic value in seasonal forecasts for European users. However, the impact of ENSO on economic value over Europe is mixed; value is enhanced by El Nino only for some potential users with specific C/L. The techniques developed are applicable to complex E for arbitrary regions. Hence these techniques are proposed as the basis of an objective probabilistic and decision-model evaluation of operational seasonal ensemble forecasts.

Comments on “Probabilistic Predictions of Precipitation Using the ECMWF Ensemble Prediction System”

Abstract: In their paper, Buizza et al. (1999, hereinafter referred to as BHLG) present and discuss various attributes of the quantitative precipitation forecast (QPF) performance of the European Centre for Medium-Range Weather Forecasts ensemble system. The verification tools used are signal detection theory measures, reliability diagrams, the Brier score and skill score, the threat score, and the root-mean-square error. The verification data consisted of short-range (0‐24 h) forecasts from the full-resolution model. In an appendix, BHLG justify the use of model data as ‘‘observations’’ on the basis that point precipitation observations are representative of smaller scales than model QPF output. Comparative summary results are presented for four 3-month seasons, for a model grid domain covering Europe. Variability in the performance is assessed for two smaller domains, as a function of time, and as a function of projection time. The variability in performance is also assessed as a function of the threshold chosen for probability estimation from the ensemble output. An interesting feature of the paper is the presentation of several case studies, which facilitate the synoptic interpretation of the relative operating characteristic (ROC) values. My comments on this paper relate mostly to the ROC curve and its use and interpretation. The ROC is relatively new to meteorology, having been brought into the field as a verification tool by Mason (1982). Its use has become more widespread since the advent of ensemble forecasting. In Murphy and Winkler’s (1987) framework for probability forecast verification, the ROC fits into the ‘‘likelihood-base rate’’ factorization of the joint distribution of forecasts and observations, which implies a stratification of the joint distribution according to the observation. Specifically, the ROC curve and two as