Showing papers on "Gaussian process published in 2007"

Gaussian basis sets for accurate calculations on molecular systems in gas and condensed phases.

Abstract: We present a library of Gaussian basis sets that has been specifically optimized to perform accurate molecular calculations based on density functional theory. It targets a wide range of chemical environments, including the gas phase, interfaces, and the condensed phase. These generally contracted basis sets, which include diffuse primitives, are obtained minimizing a linear combination of the total energy and the condition number of the overlap matrix for a set of molecules with respect to the exponents and contraction coefficients of the full basis. Typically, for a given accuracy in the total energy, significantly fewer basis functions are needed in this scheme than in the usual split valence scheme, leading to a speedup for systems where the computational cost is dominated by diagonalization. More importantly, binding energies of hydrogen bonded complexes are of similar quality as the ones obtained with augmented basis sets, i.e., have a small (down to 0.2 kcal/mol) basis set superposition error, and the monomers have dipoles within 0.1 D of the basis set limit. However, contrary to typical augmented basis sets, there are no near linear dependencies in the basis, so that the overlap matrix is always well conditioned, also, in the condensed phase. The basis can therefore be used in first principles molecular dynamics simulations and is well suited for linear scaling calculations.

Multi-task Gaussian Process Prediction

Abstract: In this paper we investigate multi-task learning in the context of Gaussian Processes (GP). We propose a model that learns a shared covariance function on input-dependent features and a "free-form" covariance matrix over tasks. This allows for good flexibility when modelling inter-task dependencies while avoiding the need for large amounts of data for training. We show that under the assumption of noise-free observations and a block design, predictions for a given task only depend on its target values and therefore a cancellation of inter-task transfer occurs. We evaluate the benefits of our model on two practical applications: a compiler performance prediction problem and an exam score prediction task. Additionally, we make use of GP approximations and properties of our model in order to provide scalability to large data sets.

Voice Conversion Based on Maximum-Likelihood Estimation of Spectral Parameter Trajectory

Abstract: In this paper, we describe a novel spectral conversion method for voice conversion (VC). A Gaussian mixture model (GMM) of the joint probability density of source and target features is employed for performing spectral conversion between speakers. The conventional method converts spectral parameters frame by frame based on the minimum mean square error. Although it is reasonably effective, the deterioration of speech quality is caused by some problems: 1) appropriate spectral movements are not always caused by the frame-based conversion process, and 2) the converted spectra are excessively smoothed by statistical modeling. In order to address those problems, we propose a conversion method based on the maximum-likelihood estimation of a spectral parameter trajectory. Not only static but also dynamic feature statistics are used for realizing the appropriate converted spectrum sequence. Moreover, the oversmoothing effect is alleviated by considering a global variance feature of the converted spectra. Experimental results indicate that the performance of VC can be dramatically improved by the proposed method in view of both speech quality and conversion accuracy for speaker individuality.

Multi-fidelity optimization via surrogate modelling

Abstract: This paper demonstrates the application of correlated Gaussian process based approximations to optimization where multiple levels of analysis are available, using an extension to the geostatistical method of co-kriging. An exchange algorithm is used to choose which points of the search space to sample within each level of analysis. The derivation of the co-kriging equations is presented in an intuitive manner, along with a new variance estimator to account for varying degrees of computational ‘noise’ in the multiple levels of analysis. A multi-fidelity wing optimization is used to demonstrate the methodology.

Analytic Implementations of the Cardinalized Probability Hypothesis Density Filter

Abstract: The probability hypothesis density (PHD) recursion propagates the posterior intensity of the random finite set (RFS) of targets in time. The cardinalized PHD (CPHD) recursion is a generalization of the PHD recursion, which jointly propagates the posterior intensity and the posterior cardinality distribution. In general, the CPHD recursion is computationally intractable. This paper proposes a closed-form solution to the CPHD recursion under linear Gaussian assumptions on the target dynamics and birth process. Based on this solution, an effective multitarget tracking algorithm is developed. Extensions of the proposed closed-form recursion to accommodate nonlinear models are also given using linearization and unscented transform techniques. The proposed CPHD implementations not only sidestep the need to perform data association found in traditional methods, but also dramatically improve the accuracy of individual state estimates as well as the variance of the estimated number of targets when compared to the standard PHD filter. Our implementations only have a cubic complexity, but simulations suggest favorable performance compared to the standard Joint Probabilistic Data Association (JPDA) filter which has a nonpolynomial complexity.

Bayesian treed Gaussian process models with an application to computer modeling

Abstract: Motivated by a computer experiment for the design of a rocket booster, this paper explores nonstationary modeling methodologies that couple stationary Gaussian processes with treed partitioning. Partitioning is a simple but effective method for dealing with nonstationarity. The methodological developments and statistical computing details which make this approach efficient are described in detail. In addition to providing an analysis of the rocket booster simulator, our approach is demonstrated to be effective in other arenas.

WiFi-SLAM using Gaussian process latent variable models

Abstract: WiFi localization, the task of determining the physical location of a mobile device from wireless signal strengths, has been shown to be an accurate method of indoor and outdoor localization and a powerful building block for location-aware applications. However, most localization techniques require a training set of signal strength readings labeled against a ground truth location map, which is prohibitive to collect and maintain as maps grow large. In this paper we propose a novel technique for solving the WiFi SLAM problem using the Gaussian Process Latent Variable Model (GPLVM) to determine the latent-space locations of unlabeled signal strength data. We show how GPLVM, in combination with an appropriate motion dynamics model, can be used to reconstruct a topological connectivity graph from a signal strength sequence which, in combination with the learned Gaussian Process signal strength model, can be used to perform efficient localization.

Active Learning with Gaussian Processes for Object Categorization

Abstract: Discriminative methods for visual object category recognition are typically non-probabilistic, predicting class labels but not directly providing an estimate of uncertainty. Gaussian Processes (GPs) are powerful regression techniques with explicit uncertainty models; we show here how Gaussian Processes with covariance functions defined based on a Pyramid Match Kernel (PMK) can be used for probabilistic object category recognition. The uncertainty model provided by GPs offers confidence estimates at test points, and naturally allows for an active learning paradigm in which points are optimally selected for interactive labeling. We derive a novel active category learning method based on our probabilistic regression model, and show that a significant boost in classification performance is possible, especially when the amount of training data for a category is ultimately very small.

Most likely heteroscedastic Gaussian process regression

Abstract: This paper presents a novel Gaussian process (GP) approach to regression with input-dependent noise rates. We follow Goldberg et al.'s approach and model the noise variance using a second GP in addition to the GP governing the noise-free output value. In contrast to Goldberg et al., however, we do not use a Markov chain Monte Carlo method to approximate the posterior noise variance but a most likely noise approach. The resulting model is easy to implement and can directly be used in combination with various existing extensions of the standard GPs such as sparse approximations. Extensive experiments on both synthetic and real-world data, including a challenging perception problem in robotics, show the effectiveness of most likely heteroscedastic GP regression.

Local and global sparse Gaussian process approximations.

Abstract: Gaussian process (GP) models are flexible probabilistic nonparametric models for regression, classification and other tasks. Unfortunately they suffer from computational intractability for large data sets. Over the past decade there have been many different approximations developed to reduce this cost. Most of these can be termed global approximations, in that they try to summarize all the training data via a small set of support points. A different approach is that of local regression, where many local experts account for their own part of space. In this paper we start by investigating the regimes in which these different approaches work well or fail. We then proceed to develop a new sparse GP approximation which is a combination of both the global and local approaches. Theoretically we show that it is derived as a natural extension of the framework developed by Quinonero Candela and Rasmussen [2005] for sparse GP approximations. We demonstrate the benefits of the combined approximation on some 1D examples for illustration, and on some large real-world data sets.

Approximate likelihood for large irregularly spaced spatial data.

Abstract: Likelihood approaches for large, irregularly spaced spatial datasets are often very difficult, if not infeasible, to implement due to computational limitations. Even when we can assume normality, exact calculations of the likelihood for a Gaussian spatial process observed at n locations requires O(n3) operations. We present a version of Whittle's approximation to the Gaussian log-likelihood for spatial regular lattices with missing values and for irregularly spaced datasets. This method requires O(nlog2n) operations and does not involve calculating determinants. We present simulations and theoretical results to show the benefits and the performance of the spatial likelihood approximation method presented here for spatial irregularly spaced datasets and lattices with missing values. We apply these methods to estimate the spatial structure of sea surface temperatures using satellite data with missing values.

On the Diffusion of Shape

Abstract: The main objective of this thesis is to study the global geometric properties of a manifold embedded in Euclidean space, as it evolves under a stochastic flow of diffeomorphisms. The processes driving the stochastic flows are chosen to be Gaussian processes with stationary increments (in time). The most common class of Gaussian processes with stationary increments is the family of fractional Brownian motions with Hurst parameter H ∈ (0, 1). This family encompasses a wide variety of processes with applications in the fields of oceanography, finance and telecommunications, to name a few. The fact that these processes possess stationary increments implies that the corresponding noise process is a stationary process, and so one can hope to obtain ergodic estimates. In Part I of the dissertation, we study the evolution of a codimension one manifold embedded in Euclidean space, under an isotropic and volume preserving Brownian flow. In particular we obtain expressions describing the expected rate of growth of the Lipschitz-Killing curvatures, or intrinsic volumes, of the manifold evolving under the flow. These results shed new light on the some of the intriguing growth properties of flows from a global perspective, rather than the local perspective, on which there is much larger literature. In Part II, we deviate from the setting of standard Brownian flows, whose analysis was primarily based on the Markovian character of the flow, and move to stochastic flows driven by fractional Brownian motion with Hurst parameter H ∈ ( 2 , 1). Adopting a pathwise approach, we obtain estimates for the growth of the Hausdorff measure of an m dimensional manifold embedded in R.

Optimal Signal Design for Detection of Gaussian Point Targets in Stationary Gaussian Clutter/Reverberation

Abstract: In this paper, we address the design of an optimal transmit signal and its corresponding optimal detector for a radar or active sonar system. The focus is on the temporal aspects of the waveform with the spatial aspects to be described in a future paper. The assumptions involved in modeling the clutter/reverberation return are crucial to the development of the optimal detector and its consequent optimal signal design. In particular, the target is assumed to be a Gaussian point target and the clutter/reverberation a stationary Gaussian random process. In practice, therefore, the modeling will need to be assessed and possibly extended, and additionally a means of measuring the "in-situ" clutter/reverberation spectrum will be required. The advantages of our approach are that a simple analytical result is obtained which is guaranteed to be optimal, and also the extension to spatial-temporal signal design is immediate using ideas of frequency-wavenumber representations. Some examples are given to illustrate the signal design procedure as well as the calculation of the increase in processing gain. Finally, the results are shown to be an extension of the usual procedure which places the signal energy in the noise band having minimum power

Using Deep Belief Nets to Learn Covariance Kernels for Gaussian Processes

Abstract: We show how to use unlabeled data and a deep belief net (DBN) to learn a good covariance kernel for a Gaussian process. We first learn a deep generative model of the unlabeled data using the fast, greedy algorithm introduced by [7]. If the data is high-dimensional and highly-structured, a Gaussian kernel applied to the top layer of features in the DBN works much better than a similar kernel applied to the raw input. Performance at both regression and classification can then be further improved by using backpropagation through the DBN to discriminatively fine-tune the covariance kernel.

Nonmyopic active learning of Gaussian processes: an exploration-exploitation approach

Abstract: When monitoring spatial phenomena, such as the ecological condition of a river, deciding where to make observations is a challenging task. In these settings, a fundamental question is when an active learning, or sequential design, strategy, where locations are selected based on previous measurements, will perform significantly better than sensing at an a priori specified set of locations. For Gaussian Processes (GPs), which often accurately model spatial phenomena, we present an analysis and efficient algorithms that address this question. Central to our analysis is a theoretical bound which quantifies the performance difference between active and a priori design strategies. We consider GPs with unknown kernel parameters and present a nonmyopic approach for trading off exploration, i.e., decreasing uncertainty about the model parameters, and exploitation, i.e., near-optimally selecting observations when the parameters are (approximately) known. We discuss several exploration strategies, and present logarithmic sample complexity bounds for the exploration phase. We then extend our algorithm to handle nonstationary GPs exploiting local structure in the model. We also present extensive empirical evaluation on several real-world problems.

Hierarchical Gaussian process latent variable models

Abstract: The Gaussian process latent variable model (GP-LVM) is a powerful approach for probabilistic modelling of high dimensional data through dimensional reduction. In this paper we extend the GP-LVM through hierarchies. A hierarchical model (such as a tree) allows us to express conditional independencies in the data as well as the manifold structure. We first introduce Gaussian process hierarchies through a simple dynamical model, we then extend the approach to a more complex hierarchy which is applied to the visualisation of human motion data sets.

tgp: An R Package for Bayesian Nonstationary, Semiparametric Nonlinear Regression and Design by Treed Gaussian Process Models

Abstract: The tgp package for R is a tool for fully Bayesian nonstationary, semiparametric nonlinear regression and design by treed Gaussian processes with jumps to the limiting linear model. Special cases also implemented include Bayesian linear models, linear CART, stationary separable and isotropic Gaussian processes. In addition to inference and posterior prediction, the package supports the (sequential) design of experiments under these models paired with several objective criteria. 1-d and 2-d plotting, with higher dimension projection and slice capabilities, and tree drawing functions (requiring maptree and combinat packages), are also provided for visualization of tgp objects.

Gaussian Processes and Reinforcement Learning for Identification and Control of an Autonomous Blimp

Abstract: Blimps are a promising platform for aerial robotics and have been studied extensively for this purpose. Unlike other aerial vehicles, blimps are relatively safe and also possess the ability to loiter for long periods. These advantages, however, have been difficult to exploit because blimp dynamics are complex and inherently non-linear. The classical approach to system modeling represents the system as an ordinary differential equation (ODE) based on Newtonian principles. A more recent modeling approach is based on representing state transitions as a Gaussian process (GP). In this paper, we present a general technique for system identification that combines these two modeling approaches into a single formulation. This is done by training a Gaussian process on the residual between the non-linear model and ground truth training data. The result is a GP-enhanced model that provides an estimate of uncertainty in addition to giving better state predictions than either ODE or GP alone. We show how the GP-enhanced model can be used in conjunction with reinforcement learning to generate a blimp controller that is superior to those learned with ODE or GP models alone.

Gaussian random number generators

Abstract: Rapid generation of high quality Gaussian random numbers is a key capability for simulations across a wide range of disciplines. Advances in computing have brought the power to conduct simulations with very large numbers of random numbers and with it, the challenge of meeting increasingly stringent requirements on the quality of Gaussian random number generators (GRNG). This article describes the algorithms underlying various GRNGs, compares their computational requirements, and examines the quality of the random numbers with emphasis on the behaviour in the tail region of the Gaussian probability density function.

Maximum likelihood estimation for compound-gaussian clutter with inverse gamma texture

Abstract: The inverse gamma distributed texture is important for modeling compound-Gaussian clutter (e.g. for sea reflections), due to the simplicity of estimating its parameters. We develop maximum-likelihood (ML) and method of fractional moments (MoFM) estimates to find the parameters of this distribution. We compute the Cramer-Rao bounds (CRBs) on the estimate variances and present numerical examples. We also show examples demonstrating the applicability of our methods to real lake-clutter data. Our results illustrate that, as expected, the ML estimates are asymptotically efficient, and also that the real lake-clutter data can be very well modeled by the inverse gamma distributed texture compound-Gaussian model.

Gaussian Mean-Shift Is an EM Algorithm

Abstract: The mean-shift algorithm, based on ideas proposed by Fukunaga and Hosteller, is a hill-climbing algorithm on the density defined by a finite mixture or a kernel density estimate Mean-shift can be used as a nonparametric clustering method and has attracted recent attention in computer vision applications such as image segmentation or tracking We show that, when the kernel is Gaussian, mean-shift is an expectation-maximization (EM) algorithm and, when the kernel is non-Gaussian, mean-shift is a generalized EM algorithm This implies that mean-shift converges from almost any starting point and that, in general, its convergence is of linear order For Gaussian mean-shift, we show: 1) the rate of linear convergence approaches 0 (superlinear convergence) for very narrow or very wide kernels, but is often close to 1 (thus, extremely slow) for intermediate widths and exactly 1 (sublinear convergence) for widths at which modes merge, 2) the iterates approach the mode along the local principal component of the data points from the inside of the convex hull of the data points, and 3) the convergence domains are nonconvex and can be disconnected and show fractal behavior We suggest ways of accelerating mean-shift based on the EM interpretation

Generalized spatial dirichlet process models

Abstract: SUMMARY Many models for the study of point-referenced data explicitly introduce spatial random effects to capture residual spatial association. These spatial effects are customarily modelled as a zeromean stationary Gaussian process. The spatial Dirichlet process introduced by Gelfand et al. (2005) produces a random spatial process which is neither Gaussian nor stationary. Rather, it varies about a process that is assumed to be stationary and Gaussian. The spatial Dirichlet process arises as a probability-weighted collection of random surfaces. This can be limiting for modelling and inferential purposes since it insists that a process realization must be one of these surfaces. We introduce a random distribution for the spatial effects that allows different surface selection at different sites. Moreover, we can specify the model so that the marginal distribution of the effect at each site still comes from a Dirichlet process. The development is offered constructively, providing a multivariate extension of the stick-breaking representation of the weights. We then introduce mixing using this generalized spatial Dirichlet process. We illustrate with a simulated dataset of independent replications and note that we can embed the generalized process within a dynamic model specification to eliminate the independence assumption.

Gaussian Processes: A Method for Automatic QSAR Modeling of ADME Properties

Abstract: In this article, we discuss the application of the Gaussian Process method for the prediction of absorption, distribution, metabolism, and excretion (ADME) properties. On the basis of a Bayesian probabilistic approach, the method is widely used in the field of machine learning but has rarely been applied in quantitative structure-activity relationship and ADME modeling. The method is suitable for modeling nonlinear relationships, does not require subjective determination of the model parameters, works for a large number of descriptors, and is inherently resistant to overtraining. The performance of Gaussian Processes compares well with and often exceeds that of artificial neural networks. Due to these features, the Gaussian Processes technique is eminently suitable for automatic model generation-one of the demands of modern drug discovery. Here, we describe the basic concept of the method in the context of regression problems and illustrate its application to the modeling of several ADME properties: blood-brain barrier, hERG inhibition, and aqueous solubility at pH 7.4. We also compare Gaussian Processes with other modeling techniques.

Learning Gaussian Conditional Random Fields for Low-Level Vision

Abstract: Markov random field (MRF) models are a popular tool for vision and image processing. Gaussian MRF models are particularly convenient to work with because they can be implemented using matrix and linear algebra routines. However, recent research has focused on on discrete-valued and non-convex MRF models because Gaussian models tend to over-smooth images and blur edges. In this paper, we show how to train a Gaussian conditional random field (GCRF) model that overcomes this weakness and can outperform the non-convex field of experts model on the task of denoising images. A key advantage of the GCRF model is that the parameters of the model can be optimized efficiently on relatively large images. The competitive performance of the GCRF model and the ease of optimizing its parameters make the GCRF model an attractive option for vision and image processing applications.

Active Policy Learning for Robot Planning and Exploration under Uncertainty

Abstract: This paper proposes a simulation-based active policy learning algorithm for finite-horizon, partially-observed sequential decision processes. The algorithm is tested in the domain of robot navigation and exploration under uncertainty. In such a setting, the expected cost, that must be minimized, is a function of the belief state (filtering distribution). This filtering distribution is in turn nonlinear and subject to discontinuities, which arise because constraints in the robot motion and control models. As a result, the expected cost is non-differentiable and very expensive to simulate. The new algorithm overcomes the first difficulty and reduces the number of required simulations as follows. First, it assumes that we have carried out previous simulations which returned values of the expected cost for different corresponding policy parameters. Second, it fits a Gaussian process (GP) regression model to these values, so as to approximate the expected cost as a function of the policy parameters. Third, it uses the GP predicted mean and variance to construct a statistical measure that determines which policy parameters should be used in the next simulation. The process is then repeated using the new parameters and the newly gathered expected cost observation. Since the objective is to find the policy parameters that minimize the expected cost, this iterative active learning approach effectively trades-off between exploration (in regions where the GP variance is large) and exploitation (where the GP mean is low). In our experiments, a robot uses the proposed algorithm to plan an optimal path for accomplishing a series of tasks, while maximizing the information about its pose and map estimates. These estimates are obtained with a standard filter for simultaneous localization and mapping. Upon gathering new observations, the robot updates the state estimates and is able to replan a new path in the spirit of open-loop feedback control.

A non‐stationary covariance‐based Kriging method for metamodelling in engineering design

Abstract: Metamodels are widely used to facilitate the analysis and optimization of engineering systems that involve computationally expensive simulations. Kriging is a metamodeling technique that is well known for its ability to build surrogate models of responses with nonlinear behavior. However, the assumption of a stationary covariance structure underlying Kriging does not hold in situations where the level of smoothness of a response varies significantly. Although nonstationary Gaussian process models have been studied for years in statistics and geostatistics communities, this has largely been for physical experimental data in relatively low dimensions. In this paper, the nonstationary covariance structure is incorporated into Kriging modeling for computer simulations. To represent the nonstationary covariance structure, we adopt a nonlinear mapping approach based on a parameterized density functions. To avoid over-parameterizing for the high dimension problems typical of engineering design, we propose a modified version of the nonlinear map approach, with a sparser, yet flexible, parameterization. The effectiveness of the proposed method is demonstrated through both mathematical and engineering examples. The robustness of the method is verified by testing multiple functions under various sampling settings. We also demonstrate that our method is effective in quantifying prediction uncertainty associated with the use of metamodels. Nomenclature

Simulation of Nonstationary Stochastic Processes by Spectral Representation

Abstract: This paper presents a rigorous derivation of a previously known formula for simulation of one-dimensional, univariate, nonstationary stochastic processes integrating Priestly's evolutionary spectral representation theory. Applying this formula, sample functions can be generated with great computational efficiency. The simulated stochastic process is asymptotically Gaussian as the number of terms tends to infinity. This paper shows that (1) these sample functions accurately reflect the prescribed probabilistic characteristics of the stochastic process when the number of terms in the cosine series is large, i.e., the ensemble averaged evolutionary power spectral density function (PSDF) or autocorrelation function approaches the corresponding target function as the sample size increases, and (2) the simulation formula, under certain conditions, can be reduced to that for nonstationary white noise process or Shinozuka's spectral representation of stationary process. In addition to derivation of simulation formula, three methods are developed in this paper to estimate the evolutionary PSDF of a given time-history data by means of the short-time Fourier transform (STFT), the wavelet transform (WT), and the Hilbert-Huang transform (HHT). A comparison of the PSDF of the well-known El Centro earthquake record estimated by these methods shows that the STFT and the WT give similar results, whereas the HHT gives more concentrated energy at certain frequencies. Effectiveness of the proposed simulation formula for nonstationary sample functions is demonstrated by simulating time histories from the estimated evolutionary PSDFs. Mean acceleration spectrum obtained by averaging the spectra of generated time histories are then presented and compared with the target spectrum to demonstrate the usefulness of this method.

Gaussian process latent variable models for human pose estimation

Abstract: We describe a method for recovering 3D human body pose from silhouettes. Our model is based on learning a latent space using the Gaussian Process Latent Variable Model (GP-LVM) [1] encapsulating both pose and silhouette features Our method is generative, this allows us to model the ambiguities of a silhouette representation in a principled way. We learn a dynamical model over the latent space which allows us to disambiguate between ambiguous silhouettes by temporal consistency. The model has only two free parameters and has several advantages over both regression approaches and other generative methods. In addition to the application shown in this paper the suggested model is easily extended to multiple observation spaces without constraints on type.