We study rare events data, binary dependent variables with dozens to thousands of times fewer ones (events, such as wars, vetoes, cases of political activism, or epidemiological infections) than zeros ("nonevents"). In many literatures, these variables have proven difficult to explain and predict, a problem that seems to have at least two sources. First, popular statistical procedures, such as logistic regression, can sharply underestimate the probability of rare events. We recommend corrections that outperform existing methods and change the estimates of absolute and relative risks by as much as some estimated effects reported in the literature. Second, commonly used data collection strategies are grossly inefficient for rare events data. The fear of collecting data with too few events has led to data collections with huge numbers of observations but relatively few, and poorly measured, explanatory variables, such as in international conflict data with more than a quarter-million dyads, only a few of which are at war. As it turns out, more efficient sampling designs exist for making valid inferences, such as sampling all variable events (e.g., wars) and a tiny fraction of nonevents (peace). This enables scholars to save as much as 99% of their (nonfixed) data collection costs or to collect much more meaningful explanatory variables. We provide methods that link these two results, enabling both types of corrections to work simultaneously, and software that implements the methods developed.

Logistic Regression in Rare Events Data

Statistical Analysis with Missing Data

/pdf/modern-applied-statistics-with-s-fourth-edition-fw5louzblu.pdf

Modern Applied Statistics with S Fourth edition

Link prediction in complex networks has attracted increasing attention from both physical and computer science communities. The algorithms can be used to extract missing information, identify spurious interactions, evaluate network evolving mechanisms, and so on. This article summaries recent progress about link prediction algorithms, emphasizing on the contributions from physical perspectives and approaches, such as the random-walk-based methods and the maximum likelihood methods. We also introduce three typical applications: reconstruction of networks, evaluation of network evolving mechanism and classification of partially labeled networks. Finally, we introduce some applications and outline future challenges of link prediction algorithms.

/pdf/link-prediction-in-complex-networks-a-survey-fcztp3ymq3.pdf

Link prediction in complex networks: A survey

to the DLM: The First-Order Polynomial Model.- to the DLM: The Dynamic Regression Model.- The Dynamic Linear Model.- Univariate Time Series DLM Theory.- Model Specification and Design.- Polynomial Trend Models.- Seasonal Models.- Regression, Autoregression, and Related Models.- Illustrations and Extensions of Standard DLMs.- Intervention and Monitoring.- Multi-Process Models.- Non-Linear Dynamic Models: Analytic and Numerical Approximations.- Exponential Family Dynamic Models.- Simulation-Based Methods in Dynamic Models.- Multivariate Modelling and Forecasting.- Distribution Theory and Linear Algebra.

Bayesian Forecasting and Dynamic Models

8. Predictive Inference: An Introduction

The occurrence of future record values based on data from a sequence of independent, identically distributed random variables is considered. Two situations are analysed, namely (i) where only the firstm record observations have been noted, and (ii) where all the observations have been noted up to themth, record. Tolerance regions and Bayesian predictive distributions are derived for the increase in size of the (m+r)th record value over the observedmth record value for two exponential models. Predictive distributions are also given for the additional number of observations required after themth record value until the (m+1)th record value occurs.

The Future Occurrence of Records

On presente une approche bayesienne a travers l'utilisation de probabilites predictives et on l'applique a un modele parametrique dans lequel (Y,X) a une distribution normale bivariable

Screening in a Normal Model

The concept of predictive distributions has been much discussed recently and its importance recognized. Lindley (1971, pp. 56-9) and Aitchison & Dunsmore (1975, Chapter 2) provide details of the development. Our problem concerns the making of predictions about the outcome y of a future experimentf which is described by a probability density function p(y I 0) (y E Y, 0G E)). The predictions are to be based on the information in) = (x1, ..., Xn) obtained from an independent informative experiment e which consists of n replicates of an experiment described by a probability density function p(x I 0) (x E X, Ge E)). The true, but unknown, value 0* of the underlying parameter is the same in e and f, although the forms of p(x I 0) and p(y I 0) may well be different. In this paper we consider an asymptotic Bayesian approach to prediction analysis in the sense that n is large, that is, there is a considerable amount of data available from e. Cox (1975) has derived an asymptotic method of obtaining prediction intervals by classical procedures. For the standard distributions with conjugate prior density functions p(O) (OG(Ee) the inference and decision problems associated with the corresponding predictive distributions are tractable, and prediction regions can be determined by use of standard tables; see, for example, Aitchison & Dunsmore (1975, Chapters 2, 3). One possible use of an asymptotic approach would be to provide approximations for situations which fall outside the ranges of the tables. A more important aspect, however, emerges for situations in which the predictive distributions involved are not so tractable. Some such examples occur in a life testing context (Dunsmore, 1974). Whilst the form of the underlying densityp(y I G) for the future experiment may be known and the maximum likelihood estimate G easily calculable from the informative experiment, the predictive density function may be nonstandard and the only means of evaluating it or deriving prediction intervals is by tedious computation.

Asymptotic prediction analysis

SUMMARY Screening methods are developed which raise the proportion of successes in the retained population to a predetermined level. Specification regions for the screening variables are derived within both the diagnostic and sampling paradigms (Dawid, 1976). Logistic and normal models are used for the two frameworks and the results are illustrated by an example of screening for malnourishment.

Ian R. Dunsmore

Papers

8. Predictive Inference: An Introduction

The Future Occurrence of Records

Screening in a Normal Model

Asymptotic prediction analysis

Diagnostic and sampling models in screening