Leave-one-out cross-validation (LOO) and the widely applicable information criterion (WAIC) are methods for estimating pointwise out-of-sample prediction accuracy from a fitted Bayesian model using the log-likelihood evaluated at the posterior simulations of the parameter values. LOO and WAIC have various advantages over simpler estimates of predictive error such as AIC and DIC but are less used in practice because they involve additional computational steps. Here we lay out fast and stable computations for LOO and WAIC that can be performed using existing simulation draws. We introduce an efficient computation of LOO using Pareto-smoothed importance sampling (PSIS), a new procedure for regularizing importance weights. Although WAIC is asymptotically equal to LOO, we demonstrate that PSIS-LOO is more robust in the finite case with weak priors or influential observations. As a byproduct of our calculations, we also obtain approximate standard errors for estimated predictive errors and for comparing of predictive errors between two models. We implement the computations in an R package called 'loo' and demonstrate using models fit with the Bayesian inference package Stan.

Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC

Leave-one-out cross-validation (LOO) and the widely applicable information criterion (WAIC) are methods for estimating pointwise out-of-sample prediction accuracy from a fitted Bayesian model using the log-likelihood evaluated at the posterior simulations of the parameter values. LOO and WAIC have various advantages over simpler estimates of predictive error such as AIC and DIC but are less used in practice because they involve additional computational steps. Here we lay out fast and stable computations for LOO and WAIC that can be performed using existing simulation draws. We introduce an efficient computation of LOO using Pareto-smoothed importance sampling (PSIS), a new procedure for regularizing importance weights. Although WAIC is asymptotically equal to LOO, we demonstrate that PSIS-LOO is more robust in the finite case with weak priors or influential observations. As a byproduct of our calculations, we also obtain approximate standard errors for estimated predictive errors and for comparison of predictive errors between two models. We implement the computations in an R package called loo and demonstrate using models fit with the Bayesian inference package Stan.

/pdf/practical-bayesian-model-evaluation-using-leave-one-out-3baxpqtpqi.pdf

Survival Analysis: Techniques for Censored and Truncated Data (2nd ed.) (Book)

Service-based systems that are dynamically composed at runtime to provide complex, adaptive functionality are currently one of the main development paradigms in software engineering. However, the Quality of Service (QoS) delivered by these systems remains an important concern, and needs to be managed in an equally adaptive and predictable way. To address this need, we introduce a novel, tool-supported framework for the development of adaptive service-based systems called QoSMOS (QoS Management and Optimization of Service-based systems). QoSMOS can be used to develop service-based systems that achieve their QoS requirements through dynamically adapting to changes in the system state, environment, and workload. QoSMOS service-based systems translate high-level QoS requirements specified by their administrators into probabilistic temporal logic formulae, which are then formally and automatically analyzed to identify and enforce optimal system configurations. The QoSMOS self-adaptation mechanism can handle reliability and performance-related QoS requirements, and can be integrated into newly developed solutions or legacy systems. The effectiveness and scalability of the approach are validated using simulations and a set of experiments based on an implementation of an adaptive service-based system for remote medical assistance.

https://hal.inria.fr/hal-00663216/document

Dynamic QoS Management and Optimization in Service-Based Systems

To date, several methods exist in the statistical literature for model assessment, which purport themselves specifically as Bayesian predictive methods. The decision theoretic assumptions on which these methods are based are not always clearly stated in the original articles, however. The aim of this survey is to provide a unified review of Bayesian predictive model assessment and selection methods, and of methods closely related to them. We review the various assumptions that are made in this context and discuss the connections between different approaches, with an emphasis on how each method approximates the expected utility of using a Bayesian model for the purpose of predicting future data.

/pdf/a-survey-of-bayesian-predictive-methods-for-model-assessment-g031kf5t7k.pdf

A survey of Bayesian predictive methods for model assessment, selection and comparison

Models can help software engineers to reason about design-time decisions before implementing a system. This paper focuses on models that deal with non-functional properties, such as reliability and performance. To build such models, one must rely on numerical estimates of various parameters provided by domain experts or extracted by other similar systems. Unfortunately, estimates are seldom correct. In addition, in dynamic environments, the value of parameters may change over time. We discuss an approach that addresses these issues by keeping models alive at run time and feeding a Bayesian estimator with data collected from the running system, which produces updated parameters. The updated model provides an increasingly better representation of the system. By analyzing the updated model at run time, it is possible to detect or predict if a desired property is, or will be, violated by the running implementation. Requirement violations may trigger automatic reconfigurations or recovery actions aimed at guaranteeing the desired goals. We illustrate a working framework supporting our methodology and apply it to an example in which a Web service orchestrated composition is modeled through a Discrete Time Markov Chain. Numerical simulations show the effectiveness of the approach.

Model evolution by run-time parameter adaptation

BankSealer: A decision support system for online banking fraud analysis and investigation

Case-deleted analysis is a popular method for evaluating the influence of a subset of cases on inference. The use of Monte Carlo estima- tion strategies in complicated Bayesian settings leads naturally to the use of importance sampling techniques to assess the divergence between full-data and case-deleted posteriors and to provide estimates under the case-deleted posteriors. However, the dependability of the importance sampling estima- tors depends critically on the variability of the case-deleted weights. We provide theoretical results concerning the assessment of the dependability of case-deleted importance sampling estimators in several Bayesian models. In particular, these results allow us to establish whether or not the esti- mators satisfy a central limit theorem. Because the conditions we derive are of a simple analytical nature, the assessment of the dependability of the estimators can be verified routinely before estimation is performed. We illustrate the use of the results in several examples.

/pdf/case-deletion-importance-sampling-estimators-central-limit-1cu6zzrqrb.pdf

Case-deletion importance sampling estimators: Central limit theorems and related results

SUMMARY The mean of a random distribution chosen from a neutral-to-the-right prior can be represented as the exponential functional of an increasing additive process. This fact is exploited in order to give sufficient conditions for the existence of the mean of a neutralto-the-right prior and for the absolute continuity of its probability distribution. Moreover, expressions for its moments, of any order, are provided. For illustrative purposes we consider a generalisation of the neutral-to-the-right prior based on the gamma process and the beta-Stacy process. Finally, by resorting to the maximum entropy algorithm, we obtain an approximation to the probability density function of the mean of a neutral-tothe-right prior. The arguments are easily extended to examine means of posterior quantities. The numerical results obtained are compared to those yielded by the application of some well-established simulation algorithms.

Exponential functionals and means of neutral‐to‐the‐right priors

We propose a semi-supervised online banking fraud analysis and decision support approach. During a training phase, it builds a profile for each customer based on past transactions. At runtime, it supports the analyst by ranking unforeseen transactions that deviate from the learned profiles. It uses methods whose output has a immediate statistical meaning that provide the analyst with an easy-to-understand model of each customer’s spending habits. First, we quantify the anomaly of each transaction with respect to the customer historical profile. Second, we find global clusters of customers with similar spending habits. Third, we use a temporal threshold system that measures the anomaly of the current spending pattern of each customer, with respect to his or her past spending behavior. As a result, we mitigate the undertraining due to the lack of historical data for building of well-trained profiles (of fresh users), and the users that change their (spending) habits over time. Our evaluation on real-world data shows that our approach correctly ranks complex frauds as “top priority”.

/pdf/banksealer-an-online-banking-fraud-analysis-and-decision-2lm0nswk0l.pdf

Ilenia Epifani

Papers

Model evolution by run-time parameter adaptation

BankSealer: A decision support system for online banking fraud analysis and investigation

Case-deletion importance sampling estimators: Central limit theorems and related results

Exponential functionals and means of neutral‐to‐the‐right priors

BankSealer: An Online Banking Fraud Analysis and Decision Support System