Showing papers on "Gaussian process published in 2004"

Improved adaptive Gaussian mixture model for background subtraction

Abstract: Background subtraction is a common computer vision task. We analyze the usual pixel-level approach. We develop an efficient adaptive algorithm using Gaussian mixture probability density. Recursive equations are used to constantly update the parameters and but also to simultaneously select the appropriate number of components for each pixel.

Gaussian processes for machine learning.

Abstract: Gaussian processes (GPs) are natural generalisations of multivariate Gaussian random variables to infinite (countably or continuous) index sets. GPs have been applied in a large number of fields to a diverse range of ends, and very many deep theoretical analyses of various properties are available. This paper gives an introduction to Gaussian processes on a fairly elementary level with special emphasis on characteristics relevant in machine learning. It draws explicit connections to branches such as spline smoothing models and support vector machines in which similar ideas have been investigated. Gaussian process models are routinely used to solve hard machine learning problems. They are attractive because of their flexible non-parametric nature and computational simplicity. Treated within a Bayesian framework, very powerful statistical methods can be implemented which offer valid estimates of uncertainties in our predictions and generic model selection procedures cast as nonlinear optimization problems. Their main drawback of heavy computational scaling has recently been alleviated by the introduction of generic sparse approximations.13,78,31 The mathematical literature on GPs is large and often uses deep concepts which are not required to fully understand most machine learning applications. In this tutorial paper, we aim to present characteristics of GPs relevant to machine learning and to show up precise connections to other "kernel machines" popular in the community. Our focus is on a simple presentation, but references to more detailed sources are provided.

Generalized Langevin equation with fractional Gaussian noise: subdiffusion within a single protein molecule.

Abstract: By introducing fractional Gaussian noise into the generalized Langevin equation, the subdiffusion of a particle can be described as a stationary Gaussian process with analytical tractability. This model is capable of explaining the equilibrium fluctuation of the distance between an electron transfer donor and acceptor pair within a protein that spans a broad range of time scales, and is in excellent agreement with a single-molecule experiment.

Evolutionary programming using mutations based on the Levy probability distribution

Abstract: Studies evolutionary programming with mutations based on the Levy probability distribution. The Levy probability distribution has an infinite second moment and is, therefore, more likely to generate an offspring that is farther away from its parent than the commonly employed Gaussian mutation. Such likelihood depends on a parameter /spl alpha/ in the Levy distribution. We propose an evolutionary programming algorithm using adaptive as well as nonadaptive Levy mutations. The proposed algorithm was applied to multivariate functional optimization. Empirical evidence shows that, in the case of functions having many local optima, the performance of the proposed algorithm was better than that of classical evolutionary programming using Gaussian mutation.

Mean-square performance of a family of affine projection algorithms

Abstract: Affine projection algorithms are useful adaptive filters whose main purpose is to speed the convergence of LMS-type filters. Most analytical results on affine projection algorithms assume special regression models or Gaussian regression data. The available analysis also treat different affine projection filters separately. This paper provides a unified treatment of the mean-square error, tracking, and transient performances of a family of affine projection algorithms. The treatment relies on energy conservation arguments and does not restrict the regressors to specific models or to a Gaussian distribution. Simulation results illustrate the analysis and the derived performance expressions.

Dependent Gaussian Processes

Abstract: Gaussian processes are usually parameterised in terms of their covariance functions. However, this makes it difficult to deal with multiple outputs, because ensuring that the covariance matrix is positive definite is problematic. An alternative formulation is to treat Gaussian processes as white noise sources convolved with smoothing kernels, and to parameterise the kernel instead. Using this, we extend Gaussian processes to handle multiple, coupled outputs.

Gaussian process model based predictive control

Abstract: Gaussian process models provide a probabilistic non-parametric modelling approach for black-box identification of non-linear dynamic systems. The Gaussian processes can highlight areas of the input space where prediction quality is poor, due to the lack of data or its complexity, by indicating the higher variance around the predicted mean. Gaussian process models contain noticeably less coefficients to be optimized. This paper illustrates possible application of Gaussian process models within model-based predictive control. The extra information provided within Gaussian process model is used in predictive control, where optimization of control signal takes the variance information into account. The predictive control principle is demonstrated on control of pH process benchmark.

Limit theorems for iterated random functions

Abstract: We study geometric moment contracting properties of nonlinear time series that are expressed in terms of iterated random functions. Under a Dini-continuity condition, a central limit theorem for additive functionals of such systems is established. The empirical processes of sample paths are shown to converge to Gaussian processes in the Skorokhod space. An exponential inequality is established. We present a bound for joint cumulants, which ensures the applicability of several asymptotic results in spectral analysis of time series. Our results provide a vehicle for statistical inferences for fractals and many nonlinear time series models.

Learning Gaussian Process Kernels via Hierarchical Bayes

Abstract: We present a novel method for learning with Gaussian process regression in a hierarchical Bayesian framework In a first step, kernel matrices on a fixed set of input points are learned from data using a simple and efficient EM algorithm This step is nonparametric, in that it does not require a parametric form of covariance function In a second step, kernel functions are fitted to approximate the learned covariance matrix using a generalized Nystrom method, which results in a complex, data driven kernel We evaluate our approach as a recommendation engine for art images, where the proposed hierarchical Bayesian method leads to excellent prediction performance

Statistical analysis of real clutter at different range resolutions

Abstract: A statistical analysis is presented of real radar clutter data collected using the McMaster I FIX radar in 1998 and stored in the Grimsby database. We first show the deviations of the amplitude statistics from the Rayleigh model and the suitability of the K- and Weibull-distribution for the first-order amplitude statistical characterization. Thus we focus on the I and Q components of the available data and study their statistical compatibility with the compound Gaussian model. Towards this goal it has been necessary devising appropriate testing procedures; in particular, with reference to the higher order statistics agreement, we have designed a validation procedure involving the clutter representation into generalized spherical coordinates. Remarkably the results have confirmed the suitability of the spherically invariant random processes (SIRPs) for the correct modeling of the radar clutter. Finally we have performed a spectral analysis highlighting the close matching between the estimated clutter spectral density and the exponential model.

Accurate evaluation of multiple-access performance in TH-PPM and TH-BPSK UWB systems

Abstract: A characteristic function method is proposed for precisely calculating the bit-error probability of time-hopping (TH) ultra-wideband (UWB) systems with multiple-access interference in an additive white Gaussian noise environment. The analytical expressions are validated by simulation and used to assess the accuracy of the Gaussian approximation. The Gaussian approximation is shown to be inaccurate for predicting bit-error rates (BERs) for medium and large signal-to-noise ratio (SNR) values. The performances of TH pulse position modulation (PPM) and binary phase-shift keying (BPSK) modulation schemes are accurately compared in terms of the BER. It is shown that the BPSK system outperforms the binary PPM system for all values of SNR. The sensitivity of the performance of the modulation schemes to the system parameters is also addressed through numerical examples.

Time series classification using Gaussian mixture models of reconstructed phase spaces

Abstract: A new signal classification approach is presented that is based upon modeling the dynamics of a system as they are captured in a reconstructed phase space. The modeling is done using full covariance Gaussian mixture models of time domain signatures, in contrast with current and previous work in signal classification that is typically focused on either linear systems analysis using frequency content or simple nonlinear machine learning models such as artificial neural networks. The proposed approach has strong theoretical foundations based on dynamical systems and topological theorems, resulting in a signal reconstruction, which is asymptotically guaranteed to be a complete representation of the underlying system, given properly chosen parameters. The algorithm automatically calculates these parameters to form appropriate reconstructed phase spaces, requiring only the number of mixtures, the signals, and their class labels as input. Three separate data sets are used for validation, including motor current simulations, electrocardiogram recordings, and speech waveforms. The results show that the proposed method is robust across these diverse domains, significantly outperforming the time delay neural network used as a baseline.

Semi-supervised Learning via Gaussian Processes

Abstract: We present a probabilistic approach to learning a Gaussian Process classifier in the presence of unlabeled data. Our approach involves a "null category noise model" (NCNM) inspired by ordered categorical noise models. The noise model reflects an assumption that the data density is lower between the class-conditional densities. We illustrate our approach on a toy problem and present comparative results for the semi-supervised classification of handwritten digits.

An upper bound on the sum-rate distortion function and its corresponding rate allocation schemes for the CEO problem

Abstract: We consider a distributed sensor network in which several observations are communicated to the fusion center using limited transmission rate. The observation must be separately encoded so that the target can be estimated with minimum average distortion. We address the problem from an information theoretic perspective and establish the inner and outer bound of the admissible rate-distortion region. We derive an upper bound on the sum-rate distortion function and its corresponding rate allocation schemes by exploiting the contra-polymatroid structure of the achievable rate region. The quadratic Gaussian case is analyzed in detail and the optimal rate allocation schemes in the achievable rate region are characterized. We show that our upper bound on the sum-rate distortion function is tight for the quadratic Gaussian CEO problem in the case of same signal-to-noise ratios at the sensors.

Inference from Accelerated Degradation and Failure Data Based on Gaussian Process Models

Abstract: An important problem in reliability and survival analysis is that of modeling degradation together with any observed failures in a life test. Here, based on a continuous cumulative damage approach with a Gaussian process describing degradation, a general accelerated test model is presented in which failure times and degradation measures can be combined for inference about system lifetime. Some specific models when the drift of the Gaussian process depends on the acceleration variable are discussed in detail. Illustrative examples using simulated data as well as degradation data observed in carbon-film resistors are presented.

Gaussian entanglement of formation

Abstract: We introduce a Gaussian version of the entanglement of formation adapted to bipartite Gaussian states by considering decompositions into pure Gaussian states only. We show that this quantity is an entanglement monotone under Gaussian operations and provide a simplified computation for states of arbitrary many modes. For the case of one mode per site the remaining variational problem can be solved analytically. If the considered state is in addition symmetric with respect to interchanging the two modes, we prove additivity of the considered entanglement measure. Moreover, in this case and considering only a single copy, our entanglement measure coincides with the true entanglement of formation.

Bayesian inference for non-Gaussian Ornstein–Uhlenbeck stochastic volatility processes

Abstract: We develop Markov chain Monte Carlo methodology for Bayesian inference for non-Gaussian Ornstein–Uhlenbeck stochastic volatility processes. The approach introduced involves expressing the unobserved stochastic volatility process in terms of a suitable marked Poisson process. We introduce two specific classes of Metropolis–Hastings algorithms which correspond to different ways of jointly parameterizing the marked point process and the model parameters. The performance of the methods is investigated for different types of simulated data. The approach is extended to consider the case where the volatility process is expressed as a superposition of Ornstein–Uhlenbeck processes. We apply our methodology to the US dollar–Deutschmark exchange rate.

Learning graphical models for stationary time series

Abstract: Probabilistic graphical models can be extended to time series by considering probabilistic dependencies between entire time series For stationary Gaussian time series, the graphical model semantics can be expressed naturally in the frequency domain, leading to interesting families of structured time series models that are complementary to families defined in the time domain In this paper, we present an algorithm to learn the structure from data for directed graphical models for stationary Gaussian time series We describe an algorithm for efficient forecasting for stationary Gaussian time series whose spectral densities factorize in a graphical model We also explore the relationships between graphical model structure and sparsity, comparing and contrasting the notions of sparsity in the time domain and the frequency domain Finally, we show how to make use of Mercer kernels in this setting, allowing our ideas to be extended to nonlinear models

Entanglement and purity of two-mode Gaussian states in noisy channels

Abstract: We study the evolution of purity, entanglement, and total correlations of general two-mode continuous variable Gaussian states in arbitrary uncorrelated Gaussian environments. The time evolution of purity, von Neumann entropy, logarithmic negativity, and mutual information is analyzed for a wide range of initial conditions. In general, we find that a local squeezing of the bath leads to a faster degradation of purity and entanglement, while it can help to preserve the mutual information between the modes.

Three-dimensional inversion of static-shifted magnetotelluric data

Abstract: A practical method for inverting static-shifted magnetotelluric (MT) data to produce a 3-D resistivity model is presented. Static-shift parameters are incorporated into an iterative, linearized inversion method, with a constraint added on the assumption that static shifts are due to a zero-mean, Gaussian process. A staggered finite-difference scheme is used to evaluate both the forward problem and the ‘pseudo-forward’ problem needed to construct the full sensitivity matrix. The linear system of equations is efficiently solved by alternating the incomplete Cholesky biconjugate gradient (ICBCG) solver with the static divergence correction procedure. Even with this efficiency in the forward modeling, generating the full sensitivity matrix at every iteration is still impractical on modern PCs. To reduce the computer time to a reasonable level, an efficient procedure for updating the sensitivities is implemented: (1) in the first few iterations, the sensitivities for the starting homogeneous half-space are used, (2) the full sensitivity matrix is computed only once (e.g. at the third iteration), and (3) for the subsequent iterations it is updated using Broyden’s algorithm. The synthetic and real data examples show that the method is robust in the presence of static shifts and can be used for 3-D problems of realistic size.

Derrida's Generalized Random Energy models 2: models with continuous hierarchies

Abstract: This is the second of a series of three papers in which we present a rigorous analysis of Derrida's Generalized Random Energy Models (GREM). Here we study the general case of models with a “continuum of hierarchies”. We prove the convergence of the free energy and give explicit formulas for the free energy and the two-replica distribution function in thermodynamical limit. Then we introduce the empirical distance distribution to describe effectively the Gibbs measures. We show that its limit is uniquely determined via the Ghirlanda–Guerra identities up to the mean of the replica distribution function. Finally, we show that suitable discretizations of the limiting random measure can be described by the same objects in suitably constructed GREMs.

On the efficient evaluation of probabilistic similarity functions for image retrieval

Abstract: Probabilistic approaches are a promising solution to the image retrieval problem that, when compared to standard retrieval methods, can lead to a significant gain in retrieval accuracy. However, this occurs at the cost of a significant increase in computational complexity. In fact, closed-form solutions for probabilistic retrieval are currently available only for simple probabilistic models such as the Gaussian or the histogram. We analyze the case of mixture densities and exploit the asymptotic equivalence between likelihood and Kullback-Leibler (KL) divergence to derive solutions for these models. In particular, 1) we show that the divergence can be computed exactly for vector quantizers (VQs) and 2) has an approximate solution for Gauss mixtures (GMs) that, in high-dimensional feature spaces, introduces no significant degradation of the resulting similarity judgments. In both cases, the new solutions have closed-form and computational complexity equivalent to that of standard retrieval approaches.

Mitigation techniques for non-Gaussian sea clutter

Abstract: This paper deals with the problem of coherent radar detection of targets embedded in clutter modeled as a compound-Gaussian process. We first provide a survey on clutter mitigation techniques with a particular emphasis on adaptive detection schemes ensuring the constant false-alarm rate (CFAR) property with respect to all of the clutter parameters. Thus, we propose a novel decision rule based on a recursive covariance estimator, which exploits the persymmetry property of the clutter covariance matrix. Remarkably, the devised receiver is fully CFAR in that its threshold can be set independently of the clutter distribution as well as of its covariance, even if the environment is highly heterogeneous; namely, the disturbance distributional parameters vary from cell to cell. At the analysis stage, we compare the performance of the novel detector with some classical radar receivers such as that of Kelly and the adaptive matched filter both in the presence of simulated as well as on real radar data, which statistical analysis has shown to be compatible with the compound-Gaussian model. The results show that the new receiving structure generally provides higher detection performance than the others and, for a fluctuating target, it uniformly outperforms the counterparts. We also provide a discussion on the CFAR behavior of the analyzed receivers as well as on their computational complexity.

Estimating percentiles of uncertain computer code outputs

Abstract: Summary. A deterministic computer model is to be used in a situation where there is uncertainty about the values of some or all of the input parameters. This uncertainty induces uncertainty in the output of the model. We consider the problem of estimating a specific percentile of the distribution of this uncertain output. We also suppose that the computer code is computationally expensive, so we can run the model only at a small number of distinct inputs. This means that we must consider our uncertainty about the computer code itself at all untested inputs. We model the output, as a function of its inputs, as a Gaussian process, and after a few initial runs of the code use a simulation approach to choose further suitable design points and to make inferences about the percentile of interest itself. An example is given involving a model that is used in sewer design.

Sharp asymptotics of the functional quantization problem for Gaussian processes

Abstract: The sharp asymptotics for the L 2 -quantization errors of Gaussian measures on a Hilbert space and, in particular, for Gaussian processes is derived. The condition imposed is regular variation of the eigenvalues.

Embedded trees: estimation of Gaussian Processes on graphs with cycles

Abstract: Graphical models provide a powerful general framework for encoding the structure of large-scale estimation problems. However, the graphs describing typical real-world phenomena contain many cycles, making direct estimation procedures prohibitively costly. In this paper, we develop an iterative inference algorithm for general Gaussian graphical models. It operates by exactly solving a series of modified estimation problems on spanning trees embedded within the original cyclic graph. When these subproblems are suitably chosen, the algorithm converges to the correct conditional means. Moreover, and in contrast to many other iterative methods, the tree-based procedures we propose can also be used to calculate exact error variances. Although the conditional mean iteration is effective for quite densely connected graphical models, the error variance computation is most efficient for sparser graphs. In this context, we present a modeling example suggesting that very sparsely connected graphs with cycles may provide significant advantages relative to their tree-structured counterparts, thanks both to the expressive power of these models and to the efficient inference algorithms developed herein. The convergence properties of the proposed tree-based iterations are characterized both analytically and experimentally. In addition, by using the basic tree-based iteration to precondition the conjugate gradient method, we develop an alternative, accelerated iteration that is finitely convergent. Simulation results are presented that demonstrate this algorithm's effectiveness on several inference problems, including a prototype distributed sensing application.

Quantum statistical mechanics with Gaussians: Equilibrium properties of van der Waals clusters

Abstract: The variational Gaussian wave-packet method for computation of equilibrium density matrices of quantum many-body systems is further developed. The density matrix is expressed in terms of Gaussian resolution, in which each Gaussian is propagated independently in imaginary time β=(kBT)−1 starting at the classical limit β=0. For an N-particle system a Gaussian exp[(r−q)TG(r−q)+γ] is represented by its center q∈R3N, the width matrix G∈R3N×3N, and the scale γ∈R, all treated as dynamical variables. Evaluation of observables is done by Monte Carlo sampling of the initial Gaussian positions. As demonstrated previously at not-very-low temperatures the method is surprisingly accurate for a range of model systems including the case of double-well potential. Ideally, a single Gaussian propagation requires numerical effort comparable to the propagation of a single classical trajectory for a system with 9(N2+N)/2 degrees of freedom. Furthermore, an approximation based on a direct product of single-particle Gaussians, r...

Gaussian process modeling in conjunction with individual patient simulation modeling: a case study describing the calculation of cost-effectiveness ratios for the treatment of established osteoporosis.

Abstract: Individual patient-level models can simulate more complex disease processes than cohort-based approaches. However, large numbers of patients need to be simulated to reduce 1st-order uncertainty, increasing the computational time required and often resulting in the inability to perform extensive sensitivity analyses. A solution, employing Gaussian process techniques, is presented using a case study, evaluating the cost-effectiveness of a sample of treatments for established osteoporosis. The Gaussian process model accurately formulated a statistical relationship between the inputs to the individual patient model and its outputs. This model reduced the time required for future runs from 150 min to virtually-instantaneous, allowing probabilistic sensitivity analyses-to be undertaken. This reduction in computational time was achieved with minimal loss in accuracy. The authors believe that this case study demonstrates the value of this technique in handling 1st- and 2nd-order uncertainty in the context of health economic modeling, particularly when more widely used techniques are computationally expensive or are unable to accurately model patient histories.