Showing papers on "Gaussian process published in 2009"

Variational Learning of Inducing Variables in Sparse Gaussian Processes

Abstract: Sparse Gaussian process methods that use inducing variables require the selection of the inducing inputs and the kernel hyperparameters. We introduce a variational formulation for sparse approximations that jointly infers the inducing inputs and the kernel hyperparameters by maximizing a lower bound of the true log marginal likelihood. The key property of this formulation is that the inducing inputs are defined to be variational parameters which are selected by minimizing the Kullback-Leibler divergence between the variational distribution and the exact posterior distribution over the latent function values. We apply this technique to regression and we compare it with other approaches in the literature.

The Cardinality Balanced Multi-Target Multi-Bernoulli Filter and Its Implementations

Abstract: It is shown analytically that the multitarget multiBernoulli (MeMBer) recursion, proposed by Mahler, has a significant bias in the number of targets. To reduce the cardinality bias, a novel multiBernoulli approximation to the multi-target Bayes recursion is derived. Under the same assumptions as the MeMBer recursion, the proposed recursion is unbiased. In addition, a sequential Monte Carlo (SMC) implementation (for generic models) and a Gaussian mixture (GM) implementation (for linear Gaussian models) are proposed. The latter is also extended to accommodate mildly nonlinear models by linearization and the unscented transform.

Choosing the Sample Size of a Computer Experiment: A Practical Guide

Abstract: We provide reasons and evidence supporting the informal rule that the number of runs for an effective initial computer experiment should be about 10 times the input dimension. Our arguments quantify two key characteristics of computer codes that affect the sample size required for a desired level of accuracy when approximating the code via a Gaussian process (GP). The first characteristic is the total sensitivity of a code output variable to all input variables; the second corresponds to the way this total sensitivity is distributed across the input variables, specifically the possible presence of a few prominent input factors and many impotent ones (i.e., effect sparsity). Both measures relate directly to the correlation structure in the GP approximation of the code. In this way, the article moves toward a more formal treatment of sample size for a computer experiment. The evidence supporting these arguments stems primarily from a simulation study and via specific codes modeling climate and ligand activa...

Level sets and extrema of random processes and fields

Abstract: Introduction. Reading diagram. Chapter 1: Classical results on the regularity of the paths. 1. Kolmogorov's Extension Theorem. 2. Reminder on the Normal Distribution. 3. 0-1 law for Gaussian processes. 4. Regularity of the paths. Exercises. Chapter 2: Basic Inequalities for Gaussian Processes. 1. Slepian type inequalities. 2. Ehrhard's inequality. 3. Gaussian isoperimetric inequality. 4. Inequalities for the tails of the distribution of the supremum. 5. Dudley's inequality. Exercises. Chapter 3: Crossings and Rice formulas for 1-dimensional parameter processes. 1. Rice Formulas. 2. Variants and Examples. Exercises. Chapter 4: Some Statistical Applications. 1. Elementary bounds for P{M > u}. 2. More detailed computation of the first two moments. 3. Maximum of the absolute value. 4. Application to quantitative gene detection. 5. Mixtures of Gaussian distributions. Exercises. Chapter 5: The Rice Series. 1. The Rice Series. 2. Computation of Moments. 3. Numerical aspects of Rice Series. 4. Processes with Continuous Paths. Chapter 6: Rice formulas for random fields. 1. Random fields from Rd to Rd. 2. Random fields from Rd to Rd!, d> d!. Exercises. Chapter 7: Regularity of the Distribution of the Maximum. 1. The implicit formula for the density of the maximum. 2. One parameter processes. 3. Continuity of the density of the maximum of random fields. Exercises. Chapter 8: The tail of the distribution of the maximum. 1. One-dimensional parameter: asymptotic behavior of the derivatives of FM. 2. An Application to Unbounded Processes. 3. A general bound for pM. 4. Computing p(x) for stationary isotropic Gaussian fields. 5. Asymptotics as x! +". 6. Examples. Exercises. Chapter 9: The record method. 1. Smooth processes with one dimensional parameter. 2. Non-smooth Gaussian processes. 3. Two-parameter Gaussian processes. Exercises. Chapter 10: Asymptotic methods for infinite time horizon. 1. Poisson character of "high" up-crossings. 2. Central limit theorem for non-linear functionals. Exercises. Chapter 11: Geometric characteristics of random sea-waves. 1. Gaussian model for infinitely deep sea. 2. Some geometric characteristics of waves. 3. Level curves, crests and velocities for space waves. 4. Real Data. 5. Generalizations of the Gaussian model. Exercises. Chapter 12: Systems of random equations. 1. The Shub-Smale model. 2. More general models. 3. Non-centered systems (smoothed analysis). 4. Systems having a law invariant under orthogonal transformations and translations. Chapter 13: Random fields and condition numbers of random matrices. 1. Condition numbers of non-Gaussian matrices. 2. Condition numbers of centered Gaussian matrices. 3. Non-centered Gaussian matrices. Notations. References.

Stein's method on Wiener chaos

Abstract: We combine Malliavin calculus with Stein’s method, in order to derive explicit bounds in the Gaussian and Gamma approximations of random variables in a fixed Wiener chaos of a general Gaussian process. Our approach generalizes, refines and unifies the central and non-central limit theorems for multiple Wiener–Ito integrals recently proved (in several papers, from 2005 to 2007) by Nourdin, Nualart, Ortiz-Latorre, Peccati and Tudor. We apply our techniques to prove Berry–Esseen bounds in the Breuer–Major CLT for subordinated functionals of fractional Brownian motion. By using the well-known Mehler’s formula for Ornstein–Uhlenbeck semigroups, we also recover a technical result recently proved by Chatterjee, concerning the Gaussian approximation of functionals of finite-dimensional Gaussian vectors.

Stationary max-stable fields associated to negative definite functions.

Abstract: Let Wi, i∈ℕ, be independent copies of a zero-mean Gaussian process {W(t), t∈ℝd} with stationary increments and variance σ2(t). Independently of Wi, let ∑i=1∞δUi be a Poisson point process on the real line with intensity e−y dy. We show that the law of the random family of functions {Vi(⋅), i∈ℕ}, where Vi(t)=Ui+Wi(t)−σ2(t)/2, is translation invariant. In particular, the process η(t)=⋁i=1∞Vi(t) is a stationary max-stable process with standard Gumbel margins. The process η arises as a limit of a suitably normalized and rescaled pointwise maximum of n i.i.d. stationary Gaussian processes as n→∞ if and only if W is a (nonisotropic) fractional Brownian motion on ℝd. Under suitable conditions on W, the process η has a mixed moving maxima representation.

Diagnostics for Gaussian Process Emulators

Abstract: Mathematical models, usually implemented in computer programs known as simulators, are widely used in all areas of science and technology to represent complex real-world phenomena. Simulators are often so complex that they take appreciable amounts of computer time or other resources to run. In this context, a methodology has been developed based on building a statistical representation of the simulator, known as an emulator. The principal approach to building emulators uses Gaussian processes. This work presents some diagnostics to validate and assess the adequacy of a Gaussian process emulator as surrogate for the simulator. These diagnostics are based on comparisons between simulator outputs and Gaussian process emulator outputs for some test data, known as validation data, defined by a sample of simulator runs not used to build the emulator. Our diagnostics take care to account for correlation between the validation data. To illustrate a validation procedure, we apply these diagnostics to two different...

Gaussian Process Optimization in the Bandit Setting: No Regret and Experimental Design

Abstract: Many applications require optimizing an unknown, noisy function that is expensive to evaluate. We formalize this task as a multi-armed bandit problem, where the payoff function is either sampled from a Gaussian process (GP) or has low RKHS norm. We resolve the important open problem of deriving regret bounds for this setting, which imply novel convergence rates for GP optimization. We analyze GP-UCB, an intuitive upper-confidence based algorithm, and bound its cumulative regret in terms of maximal information gain, establishing a novel connection between GP optimization and experimental design. Moreover, by bounding the latter in terms of operator spectra, we obtain explicit sublinear regret bounds for many commonly used covariance functions. In some important cases, our bounds have surprisingly weak dependence on the dimensionality. In our experiments on real sensor data, GP-UCB compares favorably with other heuristical GP optimization approaches.

An informational approach to the global optimization of expensive-to-evaluate functions

Abstract: In many global optimization problems motivated by engineering applications, the number of function evaluations is severely limited by time or cost. To ensure that each evaluation contributes to the localization of good candidates for the role of global minimizer, a sequential choice of evaluation points is usually carried out. In particular, when Kriging is used to interpolate past evaluations, the uncertainty associated with the lack of information on the function can be expressed and used to compute a number of criteria accounting for the interest of an additional evaluation at any given point. This paper introduces minimizers entropy as a new Kriging-based criterion for the sequential choice of points at which the function should be evaluated. Based on stepwise uncertainty reduction, it accounts for the informational gain on the minimizer expected from a new evaluation. The criterion is approximated using conditional simulations of the Gaussian process model behind Kriging, and then inserted into an algorithm similar in spirit to the Efficient Global Optimization (EGO) algorithm. An empirical comparison is carried out between our criterion and expected improvement, one of the reference criteria in the literature. Experimental results indicate major evaluation savings over EGO. Finally, the method, which we call IAGO (for Informational Approach to Global Optimization), is extended to robust optimization problems, where both the factors to be tuned and the function evaluations are corrupted by noise.

The variational gaussian approximation revisited

Abstract: The variational approximation of posterior distributions by multivariate gaussians has been much less popular in the machine learning community compared to the corresponding approximation by factorizing distributions. This is for a good reason: the gaussian approximation is in general plagued by an number of variational parameters to be optimized, being the number of random variables. In this letter, we discuss the relationship between the Laplace and the variational approximation, and we show that for models with gaussian priors and factorizing likelihoods, the number of variational parameters is actually . The approach is applied to gaussian process regression with nongaussian likelihoods.

GP-BayesFilters: Bayesian filtering using Gaussian process prediction and observation models

Abstract: Bayesian filtering is a general framework for recursively estimating the state of a dynamical system. Key components of each Bayes filter are probabilistic prediction and observation models. This paper shows how non-parametric Gaussian process (GP) regression can be used for learning such models from training data. We also show how Gaussian process models can be integrated into different versions of Bayes filters, namely particle filters and extended and unscented Kalman filters. The resulting GP-BayesFilters can have several advantages over standard (parametric) filters. Most importantly, GP-BayesFilters do not require an accurate, parametric model of the system. Given enough training data, they enable improved tracking accuracy compared to parametric models, and they degrade gracefully with increased model uncertainty. These advantages stem from the fact that GPs consider both the noise in the system and the uncertainty in the model. If an approximate parametric model is available, it can be incorporated into the GP, resulting in further performance improvements. In experiments, we show different properties of GP-BayesFilters using data collected with an autonomous micro-blimp as well as synthetic data.

Parameter estimation from time-series data with correlated errors: a wavelet-based method and its application to transit light curves

Abstract: We consider the problem of fitting a parametric model to time-series data that are afflicted by correlated noise. The noise is represented by a sum of two stationary Gaussian processes: one that is uncorrelated in time, and another that has a power spectral density varying as 1/f γ. We present an accurate and fast [O(N)] algorithm for parameter estimation based on computing the likelihood in a wavelet basis. The method is illustrated and tested using simulated time-series photometry of exoplanetary transits, with particular attention to estimating the mid-transit time. We compare our method to two other methods that have been used in the literature, the time-averaging method and the residual-permutation method. For noise processes that obey our assumptions, the algorithm presented here gives more accurate results for mid-transit times and truer estimates of their uncertainties.

Elliptical slice sampling

Abstract: Many probabilistic models introduce strong dependencies between variables using a latent multivariate Gaussian distribution or a Gaussian process. We present a new Markov chain Monte Carlo algorithm for performing inference in models with multivariate Gaussian priors. Its key properties are: 1) it has simple, generic code applicable to many models, 2) it has no free parameters, 3) it works well for a variety of Gaussian process based models. These properties make our method ideal for use while model building, removing the need to spend time deriving and tuning updates for more complex algorithms.

Non-linear matrix factorization with Gaussian processes

Abstract: A popular approach to collaborative filtering is matrix factorization. In this paper we develop a non-linear probabilistic matrix factorization using Gaussian process latent variable models. We use stochastic gradient descent (SGD) to optimize the model. SGD allows us to apply Gaussian processes to data sets with millions of observations without approximate methods. We apply our approach to benchmark movie recommender data sets. The results show better than previous state-of-the-art performance.

An efficient algorithm for Co-segmentation

Abstract: This paper is focused on the Co-segmentation problem [1] - where the objective is to segment a similar object from a pair of images. The background in the two images may be arbitrary; therefore, simultaneous segmentation of both images must be performed with a requirement that the appearance of the two sets of foreground pixels in the respective images are consistent. Existing approaches [1, 2] cast this problem as a Markov Random Field (MRF) based segmentation of the image pair with a regularized difference of the two histograms - assuming a Gaussian prior on the foreground appearance [1] or by calculating the sum of squared differences [2]. Both are interesting formulations but lead to difficult optimization problems, due to the presence of the second (histogram difference) term. The model proposed here bypasses measurement of the histogram differences in a direct fashion; we show that this enables obtaining efficient solutions to the underlying optimization model. Our new algorithm is similar to the existing methods in spirit, but differs substantially in that it can be solved to optimality in polynomial time using a maximum flow procedure on an appropriately constructed graph. We discuss our ideas and present promising experimental results.

Tractable nonparametric Bayesian inference in Poisson processes with Gaussian process intensities

Abstract: The inhomogeneous Poisson process is a point process that has varying intensity across its domain (usually time or space). For nonparametric Bayesian modeling, the Gaussian process is a useful way to place a prior distribution on this intensity. The combination of a Poisson process and GP is known as a Gaussian Cox process, or doubly-stochastic Poisson process. Likelihood-based inference in these models requires an intractable integral over an infinite-dimensional random function. In this paper we present the first approach to Gaussian Cox processes in which it is possible to perform inference without introducing approximations or finitedimensional proxy distributions. We call our method the Sigmoidal Gaussian Cox Process, which uses a generative model for Poisson data to enable tractable inference via Markov chain Monte Carlo. We compare our methods to competing methods on synthetic data and apply it to several real-world data sets.

Data Association and Track Management for the Gaussian Mixture Probability Hypothesis Density Filter

Abstract: The Gaussian mixture probability hypothesis density (GM-PHD) recursion is a closed-form solution to the probability hypothesis density (PHD) recursion, which was proposed for jointly estimating the time-varying number of targets and their states from a sequence of noisy measurement sets in the presence of data association uncertainty, clutter, and miss-detection. However the GM-PHD filter does not provide identities of individual target state estimates, that are needed to construct tracks of individual targets. In this paper, we propose a new multi-target tracker based on the GM-PHD filter, which gives the association amongst state estimates of targets over time and provides track labels. Various issues regarding initiating, propagating and terminating tracks are discussed. Furthermore, we also propose a technique for resolving identities of targets in close proximity, which the PHD filter is unable to do on its own.

Regression and Classification Using Gaussian Process Priors

Abstract: Gaussian processes are a natural way of specifying prior distributions over functions of one or more input variables. When such a function defines the mean response in a regression model with Gaussian errors, inference can be done using matrix computations, which are feasible for datasets of up to about a thousand cases. The covariance function of the Gaussian process can be given a hierarchical prior, which allows the model to discover high-level properties of the data, such as which inputs are relevant to predicting the response. Inference for these covariance hyperparameters can be done using Markov chain sampling. Classification models can be defined using Gaussian processes for underlying latent values, which can also be sampled within the Markov chain. Gaussian processes are in my view the simplest and most obvious way of defining flexible Bayesian regression and classification models, but despite some past usage, they appear to have been rather neglected as a general-purpose technique. This may be partly due to a confusion between the properties of the function being modeled and the properties of the best predictor for this unknown function.

Weighted ℓ 1 minimization for sparse recovery with prior information

Abstract: In this paper we study the compressed sensing problem of recovering a sparse signal from a system of underdetermined linear equations when we have prior information about the probability of each entry of the unknown signal being nonzero. In particular, we focus on a model where the entries of the unknown vector fall into two sets, each with a different probability of being nonzero. We propose a weighted l 1 minimization recovery algorithm and analyze its performance using a Grassman angle approach. We compute explicitly the relationship between the system parameters (the weights, the number of measurements, the size of the two sets, the probabilities of being non-zero) so that an iid random Gaussian measurement matrix along with weighted l 1 minimization recovers almost all such sparse signals with overwhelming probability as the problem dimension increases. This allows us to compute the optimal weights. We also provide simulations to demonstrate the advantages of the method over conventional l 1 optimization.

A Gaussian Mixture PHD Filter for Jump Markov System Models

Abstract: The probability hypothesis density (PHD) filter is an attractive approach to tracking an unknown and time-varying number of targets in the presence of data association uncertainty, clutter, noise, and detection uncertainty. The PHD filter admits a closed-form solution for a linear Gaussian multi-target model. However, this model is not general enough to accommodate maneuvering targets that switch between several models. In this paper, we generalize the notion of linear jump Markov systems to the multiple target case to accommodate births, deaths, and switching dynamics. We then derive a closed-form solution to the PHD recursion for the proposed linear Gaussian jump Markov multi-target model. Based on this an efficient method for tracking multiple maneuvering targets that switch between a set of linear Gaussian models is developed. An analytic implementation of the PHD filter using statistical linear regression technique is also proposed for targets that switch between a set of nonlinear models. We demonstrate through simulations that the proposed PHD filters are effective in tracking multiple maneuvering targets.

Fast Approximate Joint Diagonalization Incorporating Weight Matrices

Abstract: We propose a new low-complexity approximate joint diagonalization (AJD) algorithm, which incorporates nontrivial block-diagonal weight matrices into a weighted least-squares (WLS) AJD criterion. Often in blind source separation (BSS), when the sources are nearly separated, the optimal weight matrix for WLS-based AJD takes a (nearly) block-diagonal form. Based on this observation, we show how the new algorithm can be utilized in an iteratively reweighted separation scheme, thereby giving rise to fast implementation of asymptotically optimal BSS algorithms in various scenarios. In particular, we consider three specific (yet common) scenarios, involving stationary or block-stationary Gaussian sources, for which the optimal weight matrices can be readily estimated from the sample covariance matrices (which are also the target-matrices for the AJD). Comparative simulation results demonstrate the advantages in both speed and accuracy, as well as compliance with the theoretically predicted asymptotic optimality of the resulting BSS algorithms based on the weighted AJD, both on large scale problems with matrices of the size 100times100.

Sample Path Properties of Anisotropic Gaussian Random Fields

Abstract: Anisotropic Gaussian random flelds arise in probability theory and in various applications. Typical examples are fractional Brownian sheets, operator-scaling Gaussian flelds with stationary increments, and the solution to the stochastic heat equation. This paper is concerned with sample path properties of anisotropic Gaussian random flelds in general. Let X = fX(t); t 2 R N g be a Gaussian random fleld with values in R d and with parameters H1;:::;HN. Our goal is to characterize the anisotropic nature of X in terms of its parameters explicitly. Under some general conditions, we establish results on the modulus of continuity, small ball probabilities, fractal dimensions, hitting probabilities and local times of anisotropic Gaussian random flelds. An important tool for our study is the various forms of strong local nondeterminism.

Linear Neighborhood Propagation and Its Applications

Abstract: In this paper, a novel graph-based transductive classification approach, called linear neighborhood propagation, is proposed. The basic idea is to predict the label of a data point according to its neighbors in a linear way. This method can be cast into a second-order intrinsic Gaussian Markov random field framework. Its result corresponds to a solution to an approximate inhomogeneous biharmonic equation with Dirichlet boundary conditions. Different from existing approaches, our approach provides a novel graph structure construction method by introducing multiple-wise edges instead of pairwise edges, and presents an effective scheme to estimate the weights for such multiple-wise edges. To the best of our knowledge, these two contributions are novel for semi-supervised classification. The experimental results on image segmentation and transductive classification demonstrate the effectiveness and efficiency of the proposed approach.

Anomaly detection in sea traffic - A comparison of the Gaussian Mixture Model and the Kernel Density Estimator

Abstract: This paper presents a first attempt to evaluate two previously proposed methods for statistical anomaly detection in sea traffic, namely the Gaussian Mixture Model (GMM) and the adaptive Kernel Density Estimator (KDE). A novel performance measure related to anomaly detection, together with an intermediate performance measure related to normalcy modeling, are proposed and evaluated using recorded AIS data of vessel traffic and simulated anomalous trajectories. The normalcy modeling evaluation indicates that KDE more accurately captures finer details of normal data. Yet, results from anomaly detection show no significant difference between the two techniques and the performance of both is considered suboptimal. Part of the explanation is that the methods are based on a rather artificial division of data into geographical cells. The paper therefore discusses other clustering approaches based on more informed features of data and more background knowledge regarding the structure and natural classes of the data.

Modelling pedestrian trajectory patterns with Gaussian processes

Abstract: We propose a non-parametric model for pedestrian motion based on Gaussian Process regression, in which trajectory data are modelled by regressing relative motion against current position. We show how the underlying model can be learned in an unsupervised fashion, demonstrating this on two databases collected from static surveillance cameras. We furthermore exemplify the use of model for prediction, comparing the recently proposed GP-Bayesfilters with a Monte Carlo method. We illustrate the benefit of this approach for long term motion prediction where parametric models such as Kalman Filters would perform poorly.

Gaussian process modeling of large-scale terrain

Abstract: Building a model of large-scale terrain that can adequately handle uncertainty and incompleteness in a statistically sound way is a challenging problem. This work proposes the use of Gaussian processes as models of large-scale terrain. The proposed model naturally provides a multiresolution representation of space, incorporates and handles uncertainties aptly, and copes with incompleteness of sensory information. Gaussian process regression techniques are applied to estimate and interpolate (to fill gaps in occluded areas) elevation information across the field. The estimates obtained are the best linear unbiased estimates for the data under consideration. A single nonstationary (neural network) Gaussian process is shown to be powerful enough to model large and complex terrain, effectively handling issues relating to discontinuous data. A local approximation method based on a “moving window” methodology and implemented using k-dimensional (KD)-trees is also proposed. This enables the approach to handle extremely large data sets, thereby completely addressing its scalability issues. Experiments are performed on large-scale data sets taken from real mining applications. These data sets include sparse mine planning data, which are representative of a global positioning system–based survey, as well as dense laser scanner data taken at different mine sites. Further, extensive statistical performance evaluation and benchmarking of the technique has been performed through cross-validation experiments. They conclude that for dense and-or flat data, the proposed approach will perform very competitively with grid-based approaches using standard interpolation techniques and triangulated irregular networks using triangle-based interpolation techniques; for sparse and-or complex data, however, it would significantly outperform them. © 2009 Wiley Periodicals, Inc.

Analytic moment-based Gaussian process filtering

Abstract: We propose an analytic moment-based filter for nonlinear stochastic dynamic systems modeled by Gaussian processes. Exact expressions for the expected value and the covariance matrix are provided for both the prediction step and the filter step, where an additional Gaussian assumption is exploited in the latter case. Our filter does not require further approximations. In particular, it avoids finite-sample approximations. We compare the filter to a variety of Gaussian filters, that is, the EKF, the UKF, and the recent GP-UKF proposed by Ko et al. (2007).