Showing papers on "Gaussian process published in 1999"
01 Feb 1999
TL;DR: The main aim of this paper is to provide a tutorial on regression with Gaussian processes, starting from Bayesian linear regression, and showing how by a change of viewpoint one can see this method as a Gaussian process predictor based on priors over functions, rather than on prior over parameters.
Abstract: The main aim of this paper is to provide a tutorial on regression with Gaussian processes. We start from Bayesian linear regression, and show how by a change of viewpoint one can see this method as a Gaussian process predictor based on priors over functions, rather than on priors over parameters. This leads in to a more general discussion of Gaussian processes in section 4. Section 5 deals with further issues, including hierarchical modelling and the setting of the parameters that control the Gaussian process, the covariance functions for neural network models and the use of Gaussian processes in classification problems.
712 citations
TL;DR: A methodology is developed to derive algorithms for optimal basis selection by minimizing diversity measures proposed by Wickerhauser (1994) and Donoho (1994), which include the p-norm-like (l/sub (p/spl les/1)/) diversity measures and the Gaussian and Shannon entropies.
Abstract: A methodology is developed to derive algorithms for optimal basis selection by minimizing diversity measures proposed by Wickerhauser (1994) and Donoho (1994). These measures include the p-norm-like (l/sub (p/spl les/1)/) diversity measures and the Gaussian and Shannon entropies. The algorithm development methodology uses a factored representation for the gradient and involves successive relaxation of the Lagrangian necessary condition. This yields algorithms that are intimately related to the affine scaling transformation (AST) based methods commonly employed by the interior point approach to nonlinear optimization. The algorithms minimizing the (l/sub (p/spl les/1)/) diversity measures are equivalent to a previously developed class of algorithms called focal underdetermined system solver (FOCUSS). The general nature of the methodology provides a systematic approach for deriving this class of algorithms and a natural mechanism for extending them. It also facilitates a better understanding of the convergence behavior and a strengthening of the convergence results. The Gaussian entropy minimization algorithm is shown to be equivalent to a well-behaved p=0 norm-like optimization algorithm. Computer experiments demonstrate that the p-norm-like and the Gaussian entropy algorithms perform well, converging to sparse solutions. The Shannon entropy algorithm produces solutions that are concentrated but are shown to not converge to a fully sparse solution.
554 citations
TL;DR: In this article, a weak dependence condition for general sequences of centered random variables is proposed, in terms of the decay rate for the covariance of products of the initial random variables subject to the condition that the gap of time between both products tends to infinity.
397 citations
TL;DR: It is obtained that the log-periodogram semiparametric estimate of the memory parameter d for non-stationary time series is asymptotically normal for d and still consistent for d, and the estimates are invariant to the presence of certain deterministic trends, without any need of estimation.
327 citations
TL;DR: In this paper, the generalized spectral density is indexed by frequency and a pair of auxiliary parameters, which can capture all pairwise dependencies, including those with zero autocorrelation.
Abstract: The standardized spectral density completely describes serial dependence of a Gaussian process. For non-Gaussian processes, however, it may become an inappropriate analytic tool, because it misses the nonlinear processes with zero autocorrelation. By generalizing the concept of the standardized spectral density, I propose a new spectral tool suitable for both linear and nonlinear time series analysis. The generalized spectral density is indexed by frequency and a pair of auxiliary parameters. It is well defined for both continuous and discrete random variables, and requires no moment condition. Introduction of the auxiliary parameters renders the spectrum able to capture all pairwise dependencies, including those with zero autocorrelation. The standardized spectral density can be derived by properly differentiating the generalized spectral density with respect to the auxiliary parameters at the origin. The consistency of a class of Parzen's kernel-type estimators for the generalized spectral dens...
228 citations
TL;DR: A novel approach for the problem of estimating the data model of independent component analysis (or blind source separation) in the presence of Gaussian noise is introduced and a modification of the fixed-point (FastICA) algorithm is introduced.
Abstract: A novel approach for the problem of estimating the data model of independent component analysis (or blind source separation) in the presence of Gaussian noise is introduced. We define the Gaussian moments of a random variable as the expectations of the Gaussian function (and some related functions) with different scale parameters, and show how the Gaussian moments of a random variable can be estimated from noisy observations. This enables us to use Gaussian moments as one-unit contrast functions that have no asymptotic bias even in the presence of noise, and that are robust against outliers. To implement the maximization of the contrast functions based on Gaussian moments, a modification of the fixed-point (FastICA) algorithm is introduced.
226 citations
TL;DR: In this article, the quadratic configuration interaction (QCISD) energy calculation is replaced by a coupled cluster (CCSD(T)) energy calculation, which results in little change in the accuracy of the methods as assessed on the G2/97 test set.
213 citations
TL;DR: This work relates the small ball behavior of a Gaussian measure μ on a Banach space E with the metric entropy behavior of K μ, the unit ball of the reproducing kernel Hilbert space of μ in E to enable the application of tools and results from functional analysis to small ball problems.
Abstract: A precise link proved by Kuelbs and Li relates the small ball behavior of a Gaussian measure $\mu$ on a Banach space $E$ with the metric entropy behavior of $K_\mu$, the unit ball of the reproducing kernel Hilbert space of $\mu$ in $E$. We remove the main regularity assumption imposed on the unknown function in the link. This enables the application of tools and results from functional analysis to small ball problems and leads to small ball estimates of general algebraic type as well as to new estimates for concrete Gaussian processes. Moreover, we show that the small ball behavior of a Gaussian process is also tightly connected with the speed of approximation by “finite rank” processes.
210 citations
TL;DR: In this paper, the extreme values of fractional Brownian motions, self-similar Gaussian processes and more general Gaussian process which have a trend − ct β for some constants c, β > 0 and a variance t 2 H were studied.
169 citations
TL;DR: A new algorithm for segmentation of textured images using a multiresolution Bayesian approach which is a natural extension of the single-resolution "maximization of the posterior marginals" (MPM) estimate.
Abstract: We present a new algorithm for segmentation of textured images using a multiresolution Bayesian approach The new algorithm uses a multiresolution Gaussian autoregressive (MGAR) model for the pyramid representation of the observed image, and assumes a multiscale Markov random field model for the class label pyramid The models used in this paper incorporate correlations between different levels of both the observed image pyramid and the class label pyramid The criterion used for segmentation is the minimization of the expected value of the number of misclassified nodes in the multiresolution lattice The estimate which satisfies this criterion is referred to as the "multiresolution maximization of the posterior marginals" (MMPM) estimate, and is a natural extension of the single-resolution "maximization of the posterior marginals" (MPM) estimate Previous multiresolution segmentation techniques have been based on the maximum a posterior (MAP) estimation criterion, which has been shown to be less appropriate for segmentation than the MPM criterion It is assumed that the number of distinct textures in the observed image is known The parameters of the MGAR model-the means, prediction coefficients, and prediction error variances of the different textures-are unknown A modified version of the expectation-maximization (EM) algorithm is used to estimate these parameters The parameters of the Gibbs distribution for the label pyramid are assumed to be known Experimental results demonstrating the performance of the algorithm are presented
166 citations
TL;DR: It is shown that the derived MGLRT of range distributed targets is much more effective in detecting targets distributed in range than an M out of K detector, which is cascaded with a single-point target Kelly (1986) detector.
Abstract: A modified generalized likelihood ratio test (MGLRT) for the adaptive detection of a target or targets that are distributed in range is derived. The unknown parameters associated with the hypothesis test are the complex amplitudes in range of the desired target and the unknown covariance matrix of the additive interference, which is assumed to be characterized as complex zero-mean correlated Gaussian random variables. The target's or targets' complex amplitudes are assumed to be distributed across the entire input data block (sensor/spl times/range). Results on probabilities of false alarm and detection are derived, and a bounded constant false alarm rate (CFAR) detector is developed. Simulation results are presented. It is shown that the derived MGLRT of range distributed targets is much more effective in detecting targets distributed in range than an M out of K detector, which is cascaded with a single-point target Kelly (1986) detector.
29 Nov 1999
TL;DR: Predictive approaches based on Geisser's predictive sample reuse (PSR) methodology and the related Stone's cross-validation (CV) methodology are proposed and investigated and Experimental results show that these approaches are strongly competitive with the existing approaches.
Abstract: Gaussian processes are powerful regression models specified by parameterized mean and covariance functions. Standard approaches to choose these parameters (known by the name hyperparameters) are maximum likelihood and maximum a posteriori. In this article, we propose and investigate predictive approaches based on Geisser's predictive sample reuse (PSR) methodology and the related Stone's cross-validation (CV) methodology. More specifically, we derive results for Geisser's surrogate predictive probability (GPP), Geisser's predictive mean square error (GPE), and the standard CV error and make a comparative study. Within an approximation we arrive at the generalized cross-validation (GCV) and establish its relationship with the GPP and GPE approaches. These approaches are tested on a number of problems. Experimental results show that these approaches are strongly competitive with the existing approaches.
TL;DR: In this article, a polynomial chaos expansion is used to represent a Gaussian process in terms of multidimensional polynomials orthogonal with respect to the Gaussian measure.
Abstract: A procedure is presented in this paper for developing a representation of lognormal stochastic processes via the polynomial chaos expansion. These are processes obtained by applying the exponential operator to a gaussian process. The polynomial chaos expansion results in a representation of a stochastic process in terms of multidimensional polynomials orthogonal with respect to the gaussian measure with the dimension defined through a set of independent normalized gaussian random variables. Such a representation is useful in the context of the spectral stochastic finite element method, as well as for the analytical investigation of the mathematical properties of lognormal processes.
TL;DR: It is shown that the model develops a stationary power-law probability distribution for the relevant variable, whose exponent depends on the model parameters, and the addition of diffusion to the system modifies in a nontrivial way the profile of the stationary distribution.
Abstract: We study a stochastic multiplicative process with reset events. It is shown that the model develops a stationary power-law probability distribution for the relevant variable, whose exponent depends on the model parameters. Two qualitatively different regimes are observed, corresponding to intermittent and regular behavior. In the boundary between them, the mean value of the relevant variable is time independent, and the exponent of the stationary distribution equals $\ensuremath{-}2$. The addition of diffusion to the system modifies in a nontrivial way the profile of the stationary distribution. Numerical and analytical results are presented.
Proceedings Article•
29 Nov 1999TL;DR: A variational Bayesian method for model selection over families of kernels classifiers like Support Vector machines or Gaussian processes that needs no user interaction and is able to adapt a large number of kernel parameters to given data without having to sacrifice training cases for validation.
Abstract: We present a variational Bayesian method for model selection over families of kernels classifiers like Support Vector machines or Gaussian processes. The algorithm needs no user interaction and is able to adapt a large number of kernel parameters to given data without having to sacrifice training cases for validation. This opens the possibility to use sophisticated families of kernels in situations where the small "standard kernel" classes are clearly inappropriate. We relate the method to other work done on Gaussian processes and clarify the relation between Support Vector machines and certain Gaussian process models.
TL;DR: In this paper, a critical examination of the analytical solution presented in the classic paper of Greenwood and Williamson (1966), (GW) on the statistical modeling of nominally flat contacting rough surfaces is undertaken, and it is found that using GW simple exponential distribution to approximate the usually Gaussian height distribution of the asperities is inadequate for most practical cases.
Abstract: A critical examination of the analytical solution presented in the classic paper of Greenwood and Williamson (1966), (GW) on the statistical modeling of nominally flat contacting rough surfaces is undertaken in this study. It is found that using the GW simple exponential distribution to approximate the usually Gaussian height distribution of the asperities is inadequate for most practical cases. Some other exponential type approximations are suggested, which approximate the Gaussian distribution more accurately, and still enable closed form solutions for the real area of contact, the contact load, and the number of contacting asperities. The best-modified exponential approximation is then used in the case of elastic-plastic contacts of Chang et al. (1987) ( CEB model), to obtain closed-form solutions, which favorably compare with the numerical results using the Gaussian distribution.
30 May 1999
TL;DR: A novel approach for the problem of estimating the data model of independent component analysis (or blind source separation) in the presence of Gaussian noise is introduced, and a modification of the FastICA algorithm is introduced for the maximization of the contrast functions based on Gaussian moments.
Abstract: A novel approach for the problem of estimating the data model of independent component analysis (or blind source separation) in the presence of Gaussian noise is introduced. We define the Gaussian moments of a random variable as the expectations of the Gaussian function (and some related functions) with different scale parameters, and show how the Gaussian moments of a random variable can be estimated from noisy observations. This enables us to use gaussian moments as one-unit contrast functions that have no asymptotic bias even in the presence of noise, and that are robust against outliers. To implement efficiently the maximization of the contrast functions based on Gaussian moments, a modification of our FastICA algorithm is introduced.
TL;DR: The autocorrelation function (ACF) of a non-Gaussian random process, obtained by the memoryless nonlinear transformation of a Gaussian process with a known ACF, is calculated as a power series with coefficients expressed as one-dimensional integrals.
Abstract: The autocorrelation function (ACF) of a non-Gaussian random process, obtained by the memoryless nonlinear transformation of a Gaussian process with a known ACF, is calculated as a power series with coefficients expressed as one-dimensional integrals. In general these must be evaluated numerically; two analytically tractable special cases are also considered. In cases of practical interest the series has been found to converge rapidly. These results are then used in the simulation of a non-Gaussian process with a specified ACF, which can take values less than the square of its mean. Our approach is compared with other methods in the open literature. Examples are given of time series and random fields with gamma single-point statistics that provide controlled models of high-resolution radar clutter.
Journal Article•
TL;DR: In this article, the estimation and testing of functional coefficient linear models under dependence was investigated, which includes the functional coefficient autoregressive model of Chen and Tsay (1993), using local linear smoothing to estimate the coefficient functions of a functional-coefficient linear model and derive their asymptotic distributions in terms of Gaussian processes.
Abstract: In this paper we investigate the estimation and testing of the functional coefficient linear models under dependence, which includes the functional coefficient autoregressive model of Chen and Tsay (1993). We use local linear smoothing to estimate the coefficient functions of a functional-coefficient linear model, prove their uniform consistency, and derive their asymptotic distributions in terms of Gaussian processes. From these distributions we can obtain some tests about coefficient functions and the model. Some simulations and a study of real data are reported.
15 Mar 1999
TL;DR: This work proposes an approach for generation of vectors with truncated Gaussian densities based on Gibbs sampling, which is simple to use and does not reject any of the generated vectors.
Abstract: In many Monte Carlo simulations, it is important to generate samples from given densities. Researchers in statistical signal processing and related disciplines have shown increased interest for a generator of random vectors with truncated multivariate normal probability density functions (PDFs). A straightforward method for their generation is to draw samples from the multivariate normal density and reject the ones that are outside the acceptance region. This method, which is known as rejection sampling, can be very inefficient, especially for high dimensions and/or relatively small supports of the random vectors. We propose an approach for generation of vectors with truncated Gaussian densities based on Gibbs sampling, which is simple to use and does not reject any of the generated vectors.
TL;DR: It is shown that the normal equations for a finite-support Volterra system excited by zero mean Gaussian input have a unique solution if, and only if, the power spectral process of the input signal is nonzero at least at m distinct frequencies.
Abstract: In this paper, nonlinear filtering and identification based on finite-support Volterra models are considered. The Volterra kernels are estimated via input-output statistics or directly in terms of input-output data. It is shown that the normal equations for a finite-support Volterra system excited by zero mean Gaussian input have a unique solution if, and only if, the power spectral process of the input signal is nonzero at least at m distinct frequencies, where m is the memory of the system. A multichannel embedding approach is introduced. A set of primary signals defined in terms of the input signal serve to map efficiently the nonlinear process to an equivalent multichannel format. Efficient algorithms for the estimation of the Volterra parameters are derived for batch, as well as for adaptive processing. An efficient order-recursive method is presented for the determination of the Volterra model structure. The proposed methods are illustrated by simulations.
TL;DR: The problem of estimating the parameters of a chirp signal observed in multiplicative noise, i.e., whose amplitude is randomly time-varying, is considered and an unstructured nonlinear least-squares approach (NLS) is proposed, which provides a computationally simpler but suboptimum estimator.
Abstract: We consider the problem of estimating the parameters of a chirp signal observed in multiplicative noise, i.e., whose amplitude is randomly time-varying. Two methods for solving this problem are presented. First, an unstructured nonlinear least-squares approach (NLS) is proposed. It is shown that by minimizing the NLS criterion with respect to all samples of the time-varying amplitude, the problem reduces to a two-dimensional (2-D) maximization problem. A theoretical analysis of the NLS estimator is presented, and an expression for its asymptotic variance is derived. It is shown that the NLS estimator has a variance that is very close to the Cramer-Rao bound. The second approach combines the principles behind the high-order ambiguity function (HBF) and the NLS approach. It provides a computationally simpler but suboptimum estimator. A statistical analysis of the HAF-based estimator is also carried out, and closed-form expressions are derived for the asymptotic variance of the HAF estimators based on the data and on the squared data. Numerical examples attest to the validity of the theoretical analyzes and establish a comparison between the two proposed methods.
TL;DR: In this article, the authors apply the d-separation concept and the ensuing Markov property to graphs which may have, between each two different vertices i and j, any subset of {ij, ij, Ij} as edges.
Abstract: Pearl's d-separation concept and the ensuing Markov property is applied to graphs which may have, between each two different vertices i and j, any subset of {ij, ij, ij} as edges. The class of graphs so obtained is closed under marginalization. Furthermore, the approach permits a direct proof of this theorem: “The distribution of a multivariate normal random vector satisfying a system of linear simultaneous equations is Markov w.r.t. the path diagram of the linear system”.
TL;DR: A semianalytic approximation for the chord-distribution functions of three-dimensional models of microstructure derived from Gaussian random fields based on the assumption that successive chords are independent is obtained.
Abstract: The main result of this paper is a semianalytic approximation for the chord-distribution functions of three-dimensional models of microstructure derived from Gaussian random fields. In the simplest case the chord functions are equivalent to a standard first-passage time problem, i.e., the probability density governing the time taken by a Gaussian random process to first exceed a threshold. We obtain an approximation based on the assumption that successive chords are independent. The result is a generalization of the independent interval approximation recently used to determine the exponent of persistence time decay in coarsening. The approximation is easily extended to more general models based on the intersection and union sets of models generated from the isosurfaces of random fields. The chord-distribution functions play an important role in the characterization of random composite and porous materials. Our results are compared with experimental data obtained from a three-dimensional image of a porous Fontainebleau sandstone and a two-dimensional image of a tungsten-silver composite alloy.
TL;DR: This paper focuses on Gaussian Markov random fields for which two estimation methods are proposed, and applied in a nonstationary framework, and demonstrates that the estimated parameters allow texture discrimination for remote sensing data.
Abstract: In this paper, we tackle the problem of estimating textural parameters. We do not consider the problem of texture synthesis, but the problem of extracting textural features for tasks such as image segmentation. We take into account nonstationarities occurring in the local mean. We focus on Gaussian Markov random fields for which two estimation methods are proposed, and applied in a nonstationary framework. The first one consists of extracting conditional probabilities and performing a least square approximation. This method is applied to a nonstationary framework, dealing with the piecewise constant local mean. This framework is adapted to practical tasks when discriminating several textures on a single image. The blurring effect affecting edges between two different textures is thus reduced. The second proposed method is based on renormalization theory. Statistics involved only concern variances of Gaussian laws, leading to Cramer-Rao estimators. This method is thus especially robust with respect to the size of sampling. Moreover, nonstationarities of the local mean do not affect results. We then demonstrate that the estimated parameters allow texture discrimination for remote sensing data. The first proposed estimation method is applied to extract urban areas from SPOT images. Since discontinuities of the local mean are taken into account, we obtain an accurate urban areas delineation. Finally, we apply the renormalization based on method to segment ice in polar regions from AVHRR data.
TL;DR: In this article, a universal methodology for the analysis of discretely sampled sealed bituminous road profile data is introduced, where road profile spatial acceleration is adopted as the preferred analysis domain, as roughness variations and transient events are identified with greater reliability and accuracy.
Abstract: This paper introduces a universal methodology for the analysis of discretely sampled sealed bituminous road profile data. Several hundred kilometers of Victorian (Australia) road profile data are analyzed in both the frequency and the amplitude domains. Road profile spectral characteristics are shown to be independent of road roughness. However, statistical analysis of the road elevation data shows them to be highly nonstationary, non-Gaussian processes that contain transients. Transients are difficult to locate when the data are analyzed in the road profile elevation domain itself, as they often occur within the Gaussian distribution. The road profile spatial acceleration is adopted as the preferred analysis domain, as roughness variations and transient events are identified with greater reliability and accuracy. Analysis of the spatial acceleration data enables the identification of large amplitude, short duration events (transients) as they occur extremely outside the Gaussian distribution. Higher order statistics, such as skewness and kurtosis, as well as the crest factor, are also used to detect transients. It is shown that the road surface elevation becomes a stationary mean process with a nonstationary root-mean square when analyzed in the spatial acceleration domain.
TL;DR: The PDF of a power estimate is derived for an estimate based on an arbitrary number of frequency bins, overlapping data segments, amount of overlap, and type of data window, given a correlated Gaussian input sequence.
Abstract: Welch's (1967) method for spectral estimation of averaging modified periodograms has been widely used for decades. Because such an estimate relies on random data, the estimate is also a random variable with some probability density function. Here, the PDF of a power estimate is derived for an estimate based on an arbitrary number of frequency bins, overlapping data segments, amount of overlap, and type of data window, given a correlated Gaussian input sequence. The PDFs of several cases are plotted and found to be distinctly non-Gaussian (the asymptotic result of averaging frequency bins and/or data segments), using the Kullback-Leibler distance as a measure. For limited numbers of frequency bins or data segments, the precise PDF is considerably skewed and will be important in applications such as maximum likelihood tests.
TL;DR: The experimental results show that the adopted prior distribution and the proposed techniques help to improve the performance robustness under the examined mismatch conditions.
Abstract: We study a category of robust speech recognition problem in which mismatches exist between training and testing conditions, and no accurate knowledge of the mismatch mechanism is available. The only available information is the test data along with a set of pretrained Gaussian mixture continuous density hidden Markov models (CDHMMs). We investigate the problem from the viewpoint of Bayesian prediction. A simple prior distribution, namely constrained uniform distribution, is adopted to characterize the uncertainty of the mean vectors of the CDHMMs. Two methods, namely a model compensation technique based on Bayesian predictive density and a robust decision strategy called Viterbi Bayesian predictive classification are studied. The proposed methods are compared with the conventional Viterbi decoding algorithm in speaker-independent recognition experiments on isolated digits and TI connected digit strings (TIDTGITS), where the mismatches between training and testing conditions are caused by: (1) additive Gaussian white noise, (2) each of 25 types of actual additive ambient noises, and (3) gender difference. The experimental results show that the adopted prior distribution and the proposed techniques help to improve the performance robustness under the examined mismatch conditions.
NEC1
TL;DR: Agreement of theoretical results with those of simulation in the experiment with some examples of filter convergence shows sufficient accuracy of the theory and assures the usefulness of the difference equations in estimating filter performances, thus facilitating the design of adaptive filters using the NSRA.
Abstract: In this paper, adaptive filters using the normalized signed regressor LMS algorithm (NSRA) with Gaussian reference inputs are proposed and analyzed to yield difference equations for theoretically calculating expected convergence of the filters. A simple difference equation for mean squared error (MSE) is derived when the filter input is a white and Gaussian process, whereas approximate difference equations for colored Gaussian inputs are proposed and tested. Stability conditions and residual MSE after convergence are also obtained. Agreement of theoretical results with those of simulation in the experiment with some examples of filter convergence shows sufficient accuracy of the theory and assures the usefulness of the difference equations in estimating filter performances, thus facilitating the design of adaptive filters using the NSRA.
TL;DR: The key result developed here is an explicit expression for the cross-covariance between the log-periodograms of the clean and noisy signals that is used to show that the covariance matrix of cepstral components representing N signal samples, is a fixed signal independent matrix which approaches a diagonal matrix at a rate of 1/N.
Abstract: Explicit expressions for the second-order statistics of cepstral components representing clean and noisy signal waveforms are derived. The noise is assumed additive to the signal, and the spectral components of each process are assumed statistically independent complex Gaussian random variables. The key result developed here is an explicit expression for the cross-covariance between the log-periodograms of the clean and noisy signals. In the absence of noise, this expression is used to show that the covariance matrix of cepstral components representing N signal samples, is a fixed signal independent matrix, which approaches a diagonal matrix at a rate of 1/N. In addition, the cross-covariance expression is used to develop an explicit linear minimum mean square error estimator for the clean cepstral components given noisy cepstral components. Recognition results on the English digits using the fixed covariance and linear estimator are presented.