The Area Skill Score Statistic for Evaluating Earthquake Predictability Experiments

TLDR: A preliminary procedure for comparing earthquake prediction strategies based on alarm functions and for numerical earthquake predictability experiments that involve discretization of the study region and observations is described.

Abstract: Rigorous predictability experimentation requires a statistical characterization of the performance metric used to evaluate the participating models. We explore the properties of the area skill score measure and consider issues related to experimental discretization. For the case of continuous alarm functions and continuous observations, we present exact analytical solutions that describe the distribution of the area skill score for unskilled predictors, and we also describe how a Gaussian distribution with known mean and variance can be used to approximate the area skill score distribution. We quantify the deviation of the exact distribution from the Gaussian approximation by specifying the kurtosis excess as a function of the number of observed target earthquakes. For numerical earthquake predictability experiments that involve discretization of the study region and observations, we explore simulation procedures for estimating the area skill score distribution, and we present efficient algorithms for various experimental scenarios. When more than one target earthquake occurs within a given space/time/magnitude bin, the probabilities of predicting individual events are not independent, and this requires special consideration. Having presented the statistical properties of the area skill score, we describe and illustrate a preliminary procedure for comparing earthquake prediction strategies based on alarm functions.