Showing papers on "Probability distribution published in 2015"

Weight Uncertainty in Neural Networks

Abstract: We introduce a new, efficient, principled and backpropagation-compatible algorithm for learning a probability distribution on the weights of a neural network, called Bayes by Backprop. It regularises the weights by minimising a compression cost, known as the variational free energy or the expected lower bound on the marginal likelihood. We show that this principled kind of regularisation yields comparable performance to dropout on MNIST classification. We then demonstrate how the learnt uncertainty in the weights can be used to improve generalisation in non-linear regression problems, and how this weight uncertainty can be used to drive the exploration-exploitation trade-off in reinforcement learning.

Deep Unsupervised Learning using Nonequilibrium Thermodynamics

Abstract: A central problem in machine learning involves modeling complex data-sets using highly flexible families of probability distributions in which learning, sampling, inference, and evaluation are still analytically or computationally tractable. Here, we develop an approach that simultaneously achieves both flexibility and tractability. The essential idea, inspired by non-equilibrium statistical physics, is to systematically and slowly destroy structure in a data distribution through an iterative forward diffusion process. We then learn a reverse diffusion process that restores structure in data, yielding a highly flexible and tractable generative model of the data. This approach allows us to rapidly learn, sample from, and evaluate probabilities in deep generative models with thousands of layers or time steps, as well as to compute conditional and posterior probabilities under the learned model. We additionally release an open source reference implementation of the algorithm.

Weight Uncertainty in Neural Network

Abstract: We introduce a new, efficient, principled and backpropagation-compatible algorithm for learning a probability distribution on the weights of a neural network, called Bayes by Backprop. It regularises the weights by minimising a compression cost, known as the variational free energy or the expected lower bound on the marginal likelihood. We show that this principled kind of regularisation yields comparable performance to dropout on MNIST classification. We then demonstrate how the learnt uncertainty in the weights can be used to improve generalisation in non-linear regression problems, and how this weight uncertainty can be used to drive the exploration-exploitation trade-off in reinforcement learning.

Data-driven Distributionally Robust Optimization Using the Wasserstein Metric: Performance Guarantees and Tractable Reformulations

Abstract: We consider stochastic programs where the distribution of the uncertain parameters is only observable through a finite training dataset. Using the Wasserstein metric, we construct a ball in the space of (multivariate and non-discrete) probability distributions centered at the uniform distribution on the training samples, and we seek decisions that perform best in view of the worst-case distribution within this Wasserstein ball. The state-of-the-art methods for solving the resulting distributionally robust optimization problems rely on global optimization techniques, which quickly become computationally excruciating. In this paper we demonstrate that, under mild assumptions, the distributionally robust optimization problems over Wasserstein balls can in fact be reformulated as finite convex programs---in many interesting cases even as tractable linear programs. Leveraging recent measure concentration results, we also show that their solutions enjoy powerful finite-sample performance guarantees. Our theoretical results are exemplified in mean-risk portfolio optimization as well as uncertainty quantification.

Generating simple random graphs with prescribed degree distribution

Abstract: Let $F$ be a probability distribution with support on the non-negative integers. Four methods for generating a simple undirected graph with (approximate) degree distribution $F$ are described and compared. Two methods are based on the so called configuration model with modifications ensuring a simple graph, one method is an extension of the classical Erd\H{o}s-R\'{e}nyi graph where the edge probabilities are random variables, and the last method starts with a directed random graph which is then modified to a simple undirected graph. All methods are shown to give the correct distribution in the limit of large graph size, but under different assumptions on the degree distribution $F$ and also using different order of operations.

Polynomial-Chaos-based Kriging

Abstract: Computer simulation has become the standard tool in many engineering fields for designing and optimizing systems, as well as for assessing their reliability. Optimization and uncertainty quantification problems typically require a large number of runs of the computational model at hand, which may not be feasible with high-fidelity models directly. Thus surrogate models (a.k.a metamodels) have been increasingly investigated in the last decade. Polynomial Chaos Expansions (PCE) and Kriging are two popular non-intrusive metamodelling techniques. PCE surrogates the computational model with a series of orthonormal polynomials in the input variables where polynomials are chosen in coherency with the probability distributions of those input variables. A least-square minimization technique may be used to determine the coefficients of the PCE. On the other hand, Kriging assumes that the computer model behaves as a realization of a Gaussian random process whose parameters are estimated from the available computer runs, i.e. input vectors and response values. These two techniques have been developed more or less in parallel so far with little interaction between the researchers in the two fields. In this paper, PC-Kriging is derived as a new non-intrusive meta-modeling approach combining PCE and Kriging. A sparse set of orthonormal polynomials (PCE) approximates the global behavior of the computational model whereas Kriging manages the local variability of the model output. An adaptive algorithm similar to the least angle regression algorithm determines the optimal sparse set of polynomials. PC-Kriging is validated on various benchmark analytical functions which are easy to sample for reference results. From the numerical investigations it is concluded that PC-Kriging performs better than or at least as good as the two distinct meta-modeling techniques. A larger gain in accuracy is obtained when the experimental design has a limited size, which is an asset when dealing with demanding computational models.

The Recovery Theorem

Abstract: We can only estimate the distribution of stock returns, but from option prices we observe the distribution of state prices. State prices are the product of risk aversion—the pricing kernel—and the natural probability distribution. The Recovery Theorem enables us to separate these to determine the market's forecast of returns and risk aversion from state prices alone. Among other things, this allows us to recover the pricing kernel, market risk premium, and probability of a catastrophe and to construct model-free tests of the efficient market hypothesis.

Order-Optimal Rate of Caching and Coded Multicasting with Random Demands

Abstract: We consider the canonical {\em shared link network} formed by a source node, hosting a library of $m$ information messages (files), connected via a noiseless common link to $n$ destination nodes (users), each with a cache of size M files. Users request files at random and independently, according to a given a-priori demand distribution $\qv$. A coding scheme for this network consists of a caching placement (i.e., a mapping of the library files into the user caches) and delivery scheme (i.e., a mapping for the library files and user demands into a common multicast codeword) such that, after the codeword transmission, all users can retrieve their requested file. The rate of the scheme is defined as the {\em average} codeword length normalized with respect to the length of one file, where expectation is taken over the random user demands. For the same shared link network, in the case of deterministic demands, the optimal min-max rate has been characterized within a uniform bound, independent of the network parameters. In particular, fractional caching (i.e., storing file segments) and using linear network coding has been shown to provide a min-max rate reduction proportional to 1/M with respect to standard schemes such as unicasting or "naive" uncoded multicasting. The case of random demands was previously considered by applying the same order-optimal min-max scheme separately within groups of files requested with similar probability. However, no order-optimal guarantee was provided for random demands under the average rate performance criterion. In this paper, we consider the random demand setting and provide general achievability and converse results. In particular, we consider a family of schemes that combine random fractional caching according to a probability distribution $\pv$ that depends on the demand distribution $\qv$, with a linear coded delivery scheme based on ...

Meta-analysis using effect size distributions of only statistically significant studies.

Abstract: Publication bias threatens the validity of meta-analytic results and leads to overestimation of the effect size in traditional meta-analysis. This particularly applies to meta-analyses that feature small studies, which are ubiquitous in psychology. Here we develop a new method for meta-analysis that deals with publication bias. This method, p-uniform, enables (a) testing of publication bias, (b) effect size estimation, and (c) testing of the null-hypothesis of no effect. No current method for meta-analysis possesses all 3 qualities. Application of p-uniform is straightforward because no additional data on missing studies are needed and no sophisticated assumptions or choices need to be made before applying it. Simulations show that p-uniform generally outperforms the trim-and-fill method and the test of excess significance (TES; Ioannidis & Trikalinos, 2007b) if publication bias exists and population effect size is homogenous or heterogeneity is slight. For illustration, p-uniform and other publication bias analyses are applied to the meta-analysis of McCall and Carriger (1993) examining the association between infants' habituation to a stimulus and their later cognitive ability (IQ). We conclude that p-uniform is a valuable technique for examining publication bias and estimating population effects in fixed-effect meta-analyses, and as sensitivity analysis to draw inferences about publication bias.

Distributionally Robust Logistic Regression

Abstract: This paper proposes a distributionally robust approach to logistic regression. We use the Wasserstein distance to construct a ball in the space of probability distributions centered at the uniform distribution on the training samples. If the radius of this ball is chosen judiciously, we can guarantee that it contains the unknown data-generating distribution with high confidence. We then formulate a distributionally robust logistic regression model that minimizes a worst-case expected logloss function, where the worst case is taken over all distributions in the Wasserstein ball. We prove that this optimization problem admits a tractable reformulation and encapsulates the classical as well as the popular regularized logistic regression problems as special cases. We further propose a distributionally robust approach based on Wasserstein balls to compute upper and lower confidence bounds on the misclassification probability of the resulting classifier. These bounds are given by the optimal values of two highly tractable linear programs. We validate our theoretical out-of-sample guarantees through simulated and empirical experiments.

Hilbert maps: scalable continuous occupancy mapping with stochastic gradient descent

Abstract: The vast amount of data robots can capture today motivates the development of fast and scalable statistical tools to model the environment the robot operates in. We devise a new technique for environment representation through continuous occupancy mapping that improves on the popular occupancy grip maps in two fundamental aspects: 1) it does not assume an a priori discretisation of the world into grid cells and therefore can provide maps at an arbitrary resolution; 2) it captures statistical relationships between measurements naturally, thus being more robust to outliers and possessing better generalisation performance. The technique, named Hilbert maps, is based on the computation of fast kernel approximations that project the data in a Hilbert space where a logistic regression classifier is learnt. We show that this approach allows for efficient stochastic gradient optimisation where each measurement is only processed once during learning in an online manner. We present results with three types of approximations, Random Fourier, Nystrom and a novel sparse projection. We also show how to extend the approach to accept probability distributions as inputs, i.e. when there is uncertainty over the position of laser scans due to sensor or localisation errors. Experiments demonstrate the benefits of the approach in popular benchmark datasets with several thousand laser scans.

Distributionally robust logistic regression

Abstract: This paper proposes a distributionally robust approach to logistic regression. We use the Wasserstein distance to construct a ball in the space of probability distributions centered at the uniform distribution on the training samples. If the radius of this ball is chosen judiciously, we can guarantee that it contains the unknown data-generating distribution with high confidence. We then formulate a distribution-ally robust logistic regression model that minimizes a worst-case expected logloss function, where the worst case is taken over all distributions in the Wasserstein ball. We prove that this optimization problem admits a tractable reformulation and encapsulates the classical as well as the popular regularized logistic regression problems as special cases. We further propose a distributionally robust approach based on Wasserstein balls to compute upper and lower confidence bounds on the misclassification probability of the resulting classifier. These bounds are given by the optimal values of two highly tractable linear programs. We validate our theoretical out-of-sample guarantees through simulated and empirical experiments.

Measuring Political Polarization: Twitter shows the two sides of Venezuela

Abstract: We say that a population is perfectly polarized when divided in two groups of the same size and opposite opinions. In this paper, we propose a methodology to study and measure the emergence of polarization from social interactions. We begin by proposing a model to estimate opinions in which a minority of influential individuals propagate their opinions through a social network. The result of the model is an opinion probability density function. Next, we propose an index to quantify the extent to which the resulting distribution is polarized. Finally, we apply the proposed methodology to a Twitter conversation about the late Venezuelan president, Hugo Chavez, finding a good agreement between our results and offline data. Hence, we show that our methodology can detect different degrees of polarization, depending on the structure of the network.

Towards a Learning Theory of Cause-Effect Inference

Abstract: We pose causal inference as the problem of learning to classify probability distributions. In particular, we assume access to a collection {(Si, li)}in=1, where each Si is a sample drawn from the probability distribution of Xi×Yi, and li is a binary label indicating whether "Xi→Yi" or "Xi←Yi". Given these data, we build a causal inference rule in two steps. First, we featurize each Si using the kernel mean embedding associated with some characteristic kernel. Second, we train a binary classifier on such embeddings to distinguish between causal directions. We present generalization bounds showing the statistical consistency and learning rates of the proposed approach, and provide a simple implementation that achieves state-of-the-art cause-effect inference. Furthermore, we extend our ideas to infer causal relationships between more than two variables.

Group multi-criteria supplier selection using combined grey systems theory and uncertainty theory

Abstract: Developing framework for reducing purchasing risks associated with suppliers.Combination of grey system theory and uncertainty theory is used.It neither requires any probability distribution nor fuzzy membership function.It selects the most appropriate suppliers and allocates optimal purchase quantity. Supplier selection in supply chain is critical strategic decision for organization's success and has attracted much attention of both academic researchers and practitioners. Supplier selection problem consists of stochastic and recognitive uncertainties. However, the requirement of large sample size and strong subject knowledge to build suitable fuzzy membership function restrict the applicability of probability and fuzzy theories in supplier selection problem. In response, this study proposed a new tool for supplier selection. In this paper, we applied the combination of grey system theory and uncertainty theory which neither requires any probability distribution nor fuzzy membership function. The objective of this paper is to develop framework for reducing the purchasing risks associated with suppliers. The proposed supplier selection method not only selects the most appropriate supplier(s) but also allocate optimal purchase quantity under stochastic and recognitive uncertainties. An example is shown to highlight the procedure of the proposed model at the end of this paper.

Convergence Theory and Applications of the Factorized Distribution Algorithm

Abstract: The paper investigates the optimization of additively decomposable functions (ADF) by a new evolutionary algorithm called Factorized Distribution Algorithm (FDA). FDA is based on a factorization of the distribution to generate search points. First separable ADFs are considered. These are mapped to generalized linear functions with metavariables defined for multiple alleles. The mapping transforms FDA into an Univariate Marginal Frequency Algorithm (UMDA). For UMDA the exact equation for the response to selection is.computed under the assumption of proportionate selection. For truncation selection an approximate equation for the time to convergence is used, derived from an analysis of the OneMax function. FDA is also numerically investigated for non separable functions. The time to convergence is very similar to separable ADFs. FDA outpe1iorms the genetic algorithm with recombination of strings by far.

Improved Bounds on Sample Size for Implicit Matrix Trace Estimators

Abstract: This article is concerned with Monte Carlo methods for the estimation of the trace of an implicitly given matrix $$A$$A whose information is only available through matrix-vector products. Such a method approximates the trace by an average of $$N$$N expressions of the form $$ \mathbf{w} ^t (A \mathbf{w} )$$wt(Aw), with random vectors $$ \mathbf{w} $$w drawn from an appropriate distribution. We prove, discuss and experiment with bounds on the number of realizations $$N$$N required to guarantee a probabilistic bound on the relative error of the trace estimation upon employing Rademacher (Hutchinson), Gaussian and uniform unit vector (with and without replacement) probability distributions. In total, one necessary bound and six sufficient bounds are proved, improving upon and extending similar estimates obtained in the seminal work of Avron and Toledo (JACM 58(2). Article 8, 2011) in several dimensions. We first improve their bound on $$N$$N for the Hutchinson method, dropping a term that relates to $$\mathrm{rank}(A)$$rank(A) and making the bound comparable with that for the Gaussian estimator. We further prove new sufficient bounds for the Hutchinson, Gaussian and unit vector estimators, as well as a necessary bound for the Gaussian estimator, which depend more specifically on properties of matrix $$A$$A. As such, they may suggest the type of matrix for which one distribution or another provides a particularly effective or relatively ineffective stochastic estimation method.

New Insights into the Problem with a Singular Drift Term in the Complex Langevin Method

Abstract: The complex Langevin method aims at performing path integral with a complex action numerically based on complexification of the original real dynamical variables. One of the poorly understood issues concerns occasional failure in the presence of logarithmic singularities in the action, which appear, for instance, from the fermion determinant in finite density QCD. We point out that the failure should be attributed to the breakdown of the relation between the complex weight that satisfies the Fokker-Planck equation and the probability distribution associated with the stochastic process. In fact, this problem can occur, in general, when the stochastic process involves a singular drift term. We show, however, in a simple example that there exists a parameter region in which the method works, although the standard reweighting method is hardly applicable.

Transmission Line Overload Risk Assessment for Power Systems With Wind and Load-Power Generation Correlation

Abstract: In the risk-based security assessment, probability and severity of events are the two main factors for measuring the security level of power systems. This paper presents a method for assessing line overload risk of wind-integrated power systems with the consideration of wind and load-power generation correlation. The established risk assessment model fully considers the probability and the consequence of wind uncertainties and line flow fluctuations. The point estimate method is employed to deal with the probability of line overload and the severity function is applied to quantify line flow fluctuations. Moreover, with the Cholesky decomposition, the correlation between loads and power generations are simulated by the spatial transformation of probability distributions of random variables. In addition, Nataf transformation is used to address wind resource correlation. Finally, the line overload risk index is obtained, which can be used as an indicator for quantifying power system security. Numerical results on the modified IEEE 30-bus system and the modified IEEE 118-bus system show that the types and the parameters of the wind speed distribution would affect the risk indices of line overload, and the risk indices obtained with the consideration of wind resource correlation and load correlation would reflect the system security more accurately.

Probability Distribution in the SABR Model of Stochastic Volatility

Abstract: We study the SABR model of stochastic volatility (Wilmott Mag, 2003 [10]). This model is essentially an extension of the local volatility model (Risk 7(1):18–20 [4], Risk 7(2):32–39, 1994 [6]), in which a suitable volatility parameter is assumed to be stochastic. The SABR model admits a large variety of shapes of volatility smiles, and it performs remarkably well in the swaptions and caps/floors markets. We refine the results of (Wilmott Mag, 2003 [10]) by constructing an accurate and efficient asymptotic form of the probability distribution of forwards. Furthermore, we discuss the impact of boundary conditions at zero forward on the volatility smile. Our analysis is based on a WKB type expansion for the heat kernel of a perturbed Laplace-Beltrami operator on a suitable hyperbolic Riemannian manifold.

Temporal Gillespie Algorithm: Fast Simulation of Contagion Processes on Time-Varying Networks

Abstract: Stochastic simulations are one of the cornerstones of the analysis of dynamical processes on complex networks, and are often the only accessible way to explore their behavior. The development of fast algorithms is paramount to allow large-scale simulations. The Gillespie algorithm can be used for fast simulation of stochastic processes, and variants of it have been applied to simulate dynamical processes on static networks. However, its adaptation to temporal networks remains non-trivial. We here present a temporal Gillespie algorithm that solves this problem. Our method is applicable to general Poisson (constant-rate) processes on temporal networks, stochastically exact, and up to multiple orders of magnitude faster than traditional simulation schemes based on rejection sampling. We also show how it can be extended to simulate non-Markovian processes. The algorithm is easily applicable in practice, and as an illustration we detail how to simulate both Poissonian and non-Markovian models of epidemic spreading. Namely, we provide pseudocode and its implementation in C++ for simulating the paradigmatic Susceptible-Infected-Susceptible and Susceptible-Infected-Recovered models and a Susceptible-Infected-Recovered model with non-constant recovery rates. For empirical networks, the temporal Gillespie algorithm is here typically from 10 to 100 times faster than rejection sampling.

Exponential $\mathcal {H}_{\infty }$ Filtering for Discrete-Time Switched Neural Networks With Random Delays

Abstract: This paper addresses the exponential $\mathcal {H}_{\infty }$ filtering problem for a class of discrete-time switched neural networks with random time-varying delays. The involved delays are assumed to be randomly time-varying which are characterized by introducing a Bernoulli stochastic variable. Effects of both variation range and distribution probability of the time delays are considered. The nonlinear activation functions are assumed to satisfy the sector conditions. Our aim is to estimate the state by designing a full order filter such that the filter error system is globally exponentially stable with an expected decay rate and a $\mathcal {H}_{\infty }$ performance attenuation level. The filter is designed by using a piecewise Lyapunov–Krasovskii functional together with linear matrix inequality (LMI) approach and average dwell time method. First, a set of sufficient LMI conditions are established to guarantee the exponential mean-square stability of the augmented system and then the parameters of full-order filter are expressed in terms of solutions to a set of LMI conditions. The proposed LMI conditions can be easily solved by using standard software packages. Finally, numerical examples by means of practical problems are provided to illustrate the effectiveness of the proposed filter design.

Inferring Pairwise Interactions from Biological Data Using Maximum-Entropy Probability Models

Abstract: Maximum entropy-based inference methods have been successfully used to infer direct interactions from biological datasets such as gene expression data or sequence ensembles Here, we review undirected pairwise maximum-entropy probability models in two categories of data types, those with continuous and categorical random variables As a concrete example, we present recently developed inference methods from the field of protein contact prediction and show that a basic set of assumptions leads to similar solution strategies for inferring the model parameters in both variable types These parameters reflect interactive couplings between observables, which can be used to predict global properties of the biological system Such methods are applicable to the important problems of protein 3-D structure prediction and association of gene–gene networks, and they enable potential applications to the analysis of gene alteration patterns and to protein design