Showing papers on "Gaussian process published in 1998"

Wavelet analysis of long-range-dependent traffic

Abstract: A wavelet-based tool for the analysis of long-range dependence and a related semi-parametric estimator of the Hurst parameter is introduced. The estimator is shown to be unbiased under very general conditions, and efficient under Gaussian assumptions. It can be implemented very efficiently allowing the direct analysis of very large data sets, and is highly robust against the presence of deterministic trends, as well as allowing their detection and identification. Statistical, computational, and numerical comparisons are made against traditional estimators including that of Whittle. The estimator is used to perform a thorough analysis of the long-range dependence in Ethernet traffic traces. New features are found with important implications for the choice of valid models for performance evaluation. A study of mono versus multifractality is also performed, and a preliminary study of the stationarity with respect to the Hurst parameter and deterministic trends.

A multistage representation of the Wiener filter based on orthogonal projections

Abstract: The Wiener filter is analyzed for stationary complex Gaussian signals from an information theoretic point of view. A dual-port analysis of the Wiener filter leads to a decomposition based on orthogonal projections and results in a new multistage method for implementing the Wiener filter using a nested chain of scalar Wiener filters. This new representation of the Wiener filter provides the capability to perform an information-theoretic analysis of previous, basis-dependent, reduced-rank Wiener filters. This analysis demonstrates that the cross-spectral metric is optimal in the sense that it maximizes mutual information between the observed and desired processes. A new reduced-rank Wiener filter is developed based on this new structure which evolves a basis using successive projections of the desired signal onto orthogonal, lower dimensional subspaces. The performance is evaluated using a comparative computer analysis model and it is demonstrated that the low-complexity multistage reduced-rank Wiener filter is capable of outperforming the more complex eigendecomposition-based methods.

Bayesian classification with Gaussian processes

Abstract: We consider the problem of assigning an input vector to one of m classes by predicting P(c|x) for c=1,...,m. For a two-class problem, the probability of class one given x is estimated by /spl sigma/(y(x)), where /spl sigma/(y)=1/(1+e/sup -y/). A Gaussian process prior is placed on y(x), and is combined with the training data to obtain predictions for new x points. We provide a Bayesian treatment, integrating over uncertainty in y and in the parameters that control the Gaussian process prior the necessary integration over y is carried out using Laplace's approximation. The method is generalized to multiclass problems (m>2) using the softmax function. We demonstrate the effectiveness of the method on a number of datasets.

Introduction to Gaussian processes

Abstract: Feedforward neural networks such as multilayer perceptrons are popular tools for nonlinear regression and classification problems. From a Bayesian perspective, a choice of a neural network model can be viewed as defining a prior probability distribution over non-linear functions, and the neural network's learning process can be interpreted in terms of the posterior probability distribution over the unknown function. (Some learning algorithms search for the function with maximum posterior probability and other Monte Carlo methods draw samples from this posterior probability). In the limit of large but otherwise standard networks, Neal (1996) has shown that the prior distribution over non-linear functions implied by the Bayesian neural network falls in a class of probability distributions known as Gaussian processes. The hyperparameters of the neural network model determine the characteristic length scales of the Gaussian process. Neal's observation motivates the idea of discarding parameterized networks and working directly with Gaussian processes. Computations in which the parameters of the network are optimized are then replaced by simple matrix operations using the covariance matrix of the Gaussian process. In this chapter I will review work on this idea by Williams and Rasmussen (1996), Neal (1997), Barber and Williams (1997) and Gibbs and MacKay (1997), and will assess whether, for supervised regression and classification tasks, the feedforward network has been superceded.

Blind source separation based on time-frequency signal representations

Abstract: Blind source separation consists of recovering a set of signals of which only instantaneous linear mixtures are observed. Thus far, this problem has been solved using statistical information available on the source signals. This paper introduces a new blind source separation approach exploiting the difference in the time-frequency (t-f) signatures of the sources to be separated. The approach is based on the diagonalization of a combined set of "spatial t-f distributions". In contrast to existing techniques, the proposed approach allows the separation of Gaussian sources with identical spectral shape but with different t-f localization properties. The effects of spreading the noise power while localizing the source energy in the t-f domain amounts to increasing the robustness of the proposed approach with respect to noise and, hence, improved performance. Asymptotic performance analysis and numerical simulations are provided.

The rate-distortion function for the quadratic Gaussian CEO problem

Abstract: A new multiterminal source coding problem called the CEO problem was presented and investigated by Berger, Zhang, and Viswanathan. Recently, Viswanathan and Berger have investigated an extension of the CEO problem to Gaussian sources and call it the quadratic Gaussian CEO problem. They considered this problem from a statistical viewpoint, deriving some interesting results. In this paper, we consider the quadratic Gaussian CEO problem from a standpoint of multiterminal rate-distortion theory. We regard the CEO problem as a certain multiterminal remote source coding problem with infinitely many separate encoders whose observations are conditionally independent if the remote source is given. This viewpoint leads us to a complete solution to the problem. We determine the tradeoff between the total amount of rate and squared distortion, deriving an explicit formula of the rate-distortion function. The derived function has the form of a sum of two nonnegative functions. One is a classical rate-distortion function for single Gaussian source and the other is a new rate-distortion function which dominates the performance of the system for a relatively small distortion. It follows immediately from our result that the conjecture of Viswanathan and Berger on the asymptotic behavior of minimum squared distortion for large rates is true.

Model-based geostatistics. Discussion. Authors' reply

Abstract: Conventional geostatistical methodology solves the problem of predicting the realized value of a linear functional of a Gaussian spatial stochastic process S(x) based on observations (Y i = S(x i ;)+ Z i at sampling locations x i , where the Z i are mutually independent, zero-mean Gaussian random variables. We describe two spatial applications for which Gaussian distributional assumptions are clearly inappropriate. The first concerns the assessment of residual contamination from nuclear weapons testing on a South Pacific island, in which the sampling method generates spatially indexed Poisson counts conditional on an unobserved spatially varying intensity of radioactivity; we conclude that a conventional geostatistical analysis oversmooths the data and underestimates the spatial extremes of the intensity. The second application provides a description of spatial variation in the risk of campylobacter infections relative to other enteric infections in part of north Lancashire and south Cumbria. For this application, we treat the data as binomial counts at unit postcode locations, conditionally on an unobserved relative risk surface which we estimate. The theoretical framework for our extension of geostatistical methods is that, conditionally on the unobserved process S(×), observations at sample locations × i form a generalized linear model with the corresponding values of S(× i ) appearing as an offset term in the linear predictor. We use a Bayesian inferential framework, implemented via the Markov chain Monte Carlo method, to solve the prediction problem for non-linear functionals of S(×), making a proper allowance for the uncertainty in the estimation of any model parameters.

Simulation of stationary non-Gaussian translation processes

Abstract: A simulation algorithm is developed for generating realizations of non-Gaussian stationary translation processes \iX(\it) with a specified marginal distribution and covariance function. Translation processes are memoryless nonlinear transformations \IX(t)=g[Y(t)]\N of stationary Gaussian processes \iY(\it). The proposed simulation algorithm has three steps. First, the memoryless nonlinear transformation \ig and the covariance function of \iY(\it) need to be determined from the condition that the marginal distribution and the covariance functions of \iX(\it) need to be determined from the condition that the marginal distribution and the covariance functions of \iX(\it) coincide with specified target functions. It is shown that there is a transformation \ig giving the target marginal distribution for \iX(\it). However, it is not always possible to find a covariance function of \iY(\it) yielding the target covariance function for \iX(\it). Second, realizations of \iY(\it) have to be generated. Any algorithm for generating samples of Gaussian processes can be used to produce samples of \iY(\it). Third, samples of \iX(\it) can be obtained from samples of \iY(\it) and the mapping of \iX(\it)=\ig[\iY(\it)]. The proposed simulation algorithm is demonstrated by several examples, including the case of non-Gaussian translation random field.

Bayesian pixel classification using spatially variant finite mixtures and the generalized EM algorithm

Abstract: A spatially variant finite mixture model is proposed for pixel labeling and image segmentation. For the case of spatially varying mixtures of Gaussian density functions with unknown means and variances, an expectation-maximization (EM) algorithm is derived for maximum likelihood estimation of the pixel labels and the parameters of the mixture densities, An a priori density function is formulated for the spatially variant mixture weights. A generalized EM algorithm for maximum a posteriori estimation of the pixel labels based upon these prior densities is derived. This algorithm incorporates a variation of gradient projection in the maximization step and the resulting algorithm takes the form of grouped coordinate ascent. Gaussian densities have been used for simplicity, but the algorithm can easily be modified to incorporate other appropriate models for the mixture model component densities. The accuracy of the algorithm is quantitatively evaluated through Monte Carlo simulation, and its performance is qualitatively assessed via experimental images from computerized tomography (CT) and magnetic resonance imaging (MRI).

Computation with infinite neural networks

Abstract: For neural networks with a wide class of weight priors, it can be shown that in the limit of an infinite number of hidden units, the prior over functions tends to a gaussian process. In this article, analytic forms are derived for the covariance function of the gaussian processes corresponding to networks with sigmoidal and gaussian hidden units. This allows predictions to be made efficiently using networks with an infinite number of hidden units and shows, somewhat paradoxically, that it may be easier to carry out Bayesian prediction with infinite networks rather than finite ones.

Accelerated failure time models for counting processes

Abstract: SUMMARY We present a natural extension of the conventional accelerated failure time model for survival data to formulate the effects of covariates on the mean function of the counting process for recurrent events. A class of consistent and asymptotically normal rank estimators is developed for estimating the regression parameters of the proposed model. In addition, a Nelson-Aalen-type estimator for the mean function of the counting process is constructed, which is consistent and, properly normalised, converges weakly to a zeromean Gaussian process. We assess the finite-sample properties of the proposed estimators and the associated inference procedures through Monte Carlo simulation and provide an application to a well-known bladder cancer study.

Hidden Markov modeling of flat fading channels

Abstract: Hidden Markov models (HMMs) are a powerful tool for modeling stochastic random processes. They are general enough to model with high accuracy a large variety of processes and are relatively simple allowing us to compute analytically many important parameters of the process which are very difficult to calculate for other models (such as complex Gaussian processes). Another advantage of using HMMs is the existence of powerful algorithms for fitting them to experimental data and approximating other processes. In this paper, we demonstrate that communication channel fading can be accurately modeled by HMMs, and we find closed-form solutions for the probability distribution of fade duration and the number of level crossings.

Maximum-likelihood synchronization of a single user for code-division multiple-access communication systems

Abstract: Code-division multiple access (CDMA) has emerged as an access protocol well-suited for voice and data transmission. One significant limitation of the conventional CDMA system is the near-far problem where strong signals interfere with the detection of a weak signal. Multiuser detectors assume knowledge of all of the modulation waveforms and channel parameters, and exploit this information to eliminate multiple-access interference (MAI) and to achieve near-far resistance. A major problem in practical application of multiuser detection is the estimation of the signal and channel parameters in a near-far limited system. We consider maximum-likelihood estimation of users delay, amplitude, and phase in a CDMA communication system. We present an approach for decomposing this multiuser estimation problem into a series of single-user problems. In this method the interfering users are treated as colored non-Gaussian noise. The observation vectors are preprocessed to be able to apply a Gaussian model for the MAI. The maximum-likelihood estimate (MLE) of each user's parameters based on the processed observation vectors becomes tractable. The estimator includes a whitening filter derived from the sample covariance matrix which is used to suppress the MAI, thus yielding a near-far resistant estimator.

A central-limit-theorem-based approach for analyzing queue behavior in high-speed networks

Abstract: In this paper, we study P(/spl Qscr/>x), the tail of the steady-state queue length distribution at a high-speed multiplexer. In particular, we focus on the case when the aggregate traffic to the multiplexer can be characterized by a stationary Gaussian process. We provide two asymptotic upper bounds for the tail probability and an asymptotic result that emphasizes the importance of the dominant time scale and the maximum variance. One of our bounds is in a single-exponential form and can be used to calculate an upper bound to the asymptotic constant. However, we show that this bound, being of a single-exponential form, may not accurately capture the tail probability. Our asymptotic result on the importance of the maximum variance and our extensive numerical study on a known lower bound motivate the development of our second asymptotic upper bound. This bound is expressed in terms of the maximum variance of a Gaussian process, and enables the accurate estimation of the tail probability over a wide range of queue lengths. We apply our results to Gaussian as well as multiplexed non-Gaussian input sources, and validate their performance via simulations. Wherever possible, we have conducted our simulation study using importance sampling in order to improve its reliability and to effectively capture rare events. Our analytical study is based on extreme value theory, and therefore different from the approaches using traditional Markovian and large deviations techniques.

Some stationary processes in discrete and continuous time

Abstract: A number of stationary stochastic processes are presented with properties pertinent to modelling time series from turbulence and finance. Specifically, the one-dimensional marginal distributions have log-linear tails and the autocorrelation may have two or more time scales. Discrete time models with a given marginal distribution are constructed as sums of independent autoregressions. A similar construction is made in continuous time by considering sums of Ornstein-Uhlenbeck-type processes. To prepare for this, a new property of self-decomposable distributions is presented. Also another, rather different, construction of stationary processes with generalized logistic marginal distributions as an infinite sum of Gaussian processes is proposed. In this way processes with continuous sample paths can be constructed. Multivariate versions of the various constructions are also given.

Nonlinear blind equalization schemes using complex-valued multilayer feedforward neural networks

Abstract: Among the useful blind equalization algorithms, stochastic-gradient iterative equalization schemes are based on minimizing a nonconvex and nonlinear cost function. However, as they use a linear FIR filter with a convex decision region, their residual estimation error is high. In the paper, four nonlinear blind equalization schemes that employ a complex-valued multilayer perceptron instead of the linear filter are proposed and their learning algorithms are derived. After the important properties that a suitable complex-valued activation function must possess are discussed, a new complex-valued activation function is developed for the proposed schemes to deal with QAM signals of any constellation sizes. It has been further proven that by the nonlinear transformation of the proposed function, the correlation coefficient between the real and imaginary parts of input data decreases when they are jointly Gaussian random variables. Last, the effectiveness of the proposed schemes is verified in terms of initial convergence speed and MSE in the steady state. In particular, even without carrier phase tracking procedure, the proposed schemes correct an arbitrary phase rotation caused by channel distortion.

Simultaneous bootstrap confidence bands in nonparametric regression

Abstract: In the present paper we construct asymptotic confidence bands in non-parametric regression. Our assumptions cover unequal variances of the observations and nonuni-form, possibly considerably clustered design. The confidence band is based on an undersmoothed local polynomial estimator. An appropriate quantile is obtained via the wild bootstrap. We derive certain rates (in the sample size n) for the error in coverage probability, which improves on existing results for methods that rely on the asymptotic distribution of the maximum of some Gaussian process. We propose a practicable rule for a data-dependent choice of the band-width. A small simulation study illustrates the possible gains by our approach over alternative frequently used methods.

A generalized QS-CDMA system and the design of new spreading codes

Abstract: A generalized quasi-synchronous code-division multiple-access (QS-CDMA) system for digital mobile radio communications is proposed. In a QS-CDMA system, the relative time delay between the signals of different users is random and restricted in a certain time range, that is, the signals are quasi-synchronous. The analysis shows that the multiple-access interference (MAI) of the QS-CDMA system is determined by the cross-correlation between spreading codes around the origin. To minimize the MAI of the QS-CDMA system, we design a new set of spreading codes. The performance is evaluated according to the criteria of the bit error rate (BER). Analytic results of the BER are obtained by using two methods: Gaussian approximation and characteristic function approaches, which are checked by modified Monte Carlo computer simulations known as "importance sampling." The results indicate that the performance of the QS-CDMA system using the spreading codes we construct is much improved.

Asymptotic Optimality of Certain Multihypothesis Sequential Tests: Non‐i.i.d. Case

Abstract: It is known that certain combinations of one‐sided sequential probability ratio tests are asymptotically optimal (relative to the expected sample size) for problems involving a finite number of possible distributions when probabilities of errors tend to zero and observations are independent and identically distributed according to one of the underlying distributions. The objective of this paper is to show that two specific constructions of sequential tests asymptotically minimize not only the expected time of observation but also any positive moment of the stopping time distribution under fairly general conditions for a finite number of simple hypotheses. This result appears to be true for general statistical models which include correlated and non‐homogeneous processes observed either in discrete or continuous time. For statistical problems with nuisance parameters, we consider invariant sequential tests and show that the same result is valid for this case. Finally, we apply general results to the solution of several particular problems such as a multi‐sample slippage problem for correlated Gaussian processes and for statistical models with nuisance parameters.

Inversion of large-support ill-posed linear operators using a piecewise Gaussian MRF

Abstract: We propose a method for the reconstruction of signals and images observed partially through a linear operator with a large support (e.g., a Fourier transform on a sparse set). This inverse problem is ill-posed and we resolve it by incorporating the prior information that the reconstructed objects are composed of smooth regions separated by sharp transitions. This feature is modeled by a piecewise Gaussian (PG) Markov random field (MRF), known also as the weak-string in one dimension and the weak-membrane in two dimensions. The reconstruction is defined as the maximum a posteriori estimate. The prerequisite for the use of such a prior is the success of the optimization stage. The posterior energy corresponding to a PG MRF is generally multimodal and its minimization is particularly problematic. In this context, general forms of simulated annealing rapidly become intractable when the observation operator extends over a large support. Global optimization is dealt with by extending the graduated nonconvexity (GNC) algorithm to ill-posed linear inverse problems. GNC has been pioneered by Blake and Zisserman (1987) in the field of image segmentation. The resulting algorithm is mathematically suboptimal but it is seen to be very efficient in practice. We show that the original GNC does not correctly apply to ill-posed problems. Our extension is based on a proper theoretical analysis, which provides further insight into the GNC. The performance of the proposed algorithm is corroborated by a synthetic example in the area of diffraction tomography.

Maximum Likelihood Estimation of Spatial Covariance Parameters

Abstract: In this paper, the maximum likelihood method for inferring the parameters of spatial covariances is examined. The advantages of the maximum likelihood estimation are discussed and it is shown that this method, derived assuming a multivariate Gaussian distribution for the data, gives a sound criterion of fitting covariance models irrespective of the multivariate distribution of the data. However, this distribution is impossible to verify in practice when only one realization of the random function is available. Then, the maximum entropy method is the only sound criterion of assigning probabilities in absence of information. Because the multivariate Gaussian distribution has the maximum entropy property for a fixed vector of means and covariance matrix, the multinormal distribution is the most logical choice as a default distribution for the experimental data. Nevertheless, it should be clear that the assumption of a multivariate Gaussian distribution is maintained only for the inference of spatial covariance parameters and not necessarily for other operations such as spatial interpolation, simulation or estimation of spatial distributions. Various results from simulations are presented to support the claim that the simultaneous use of maximum likelihood method and the classical nonparametric method of moments can considerably improve results in the estimation of geostatistical parameters.

Regression with input-dependent noise: A Gaussian process treatment

Abstract: Gaussian processes provide natural non-parametric prior distributions over regression functions. In this paper we consider regression problems where there is noise on the output, and the variance of the noise depends on the inputs. If we assume that the noise is a smooth function of the inputs, then it is natural to model the noise variance using a second Gaussian process, in addition to the Gaussian process governing the noise-free output value. We show that prior uncertainty about the parameters controlling both processes can be handled and that the posterior distribution of the noise rate can be sampled from using Markov chain Monte Carlo methods. Our results on a synthetic data set give a posterior noise variance that well-approximates the true variance.

Models for the extremes of Markov chains

Abstract: The modelling of extremes of a time series has progressed from the assumption of independent observations to more realistic forms of temporal dependence. In this paper, we focus on Markov chains, deriving a class of models for their joint tail which allows the degree of clustering of extremes to decrease at high levels, overcoming a key Limitation in current methodologies. Theoretical aspects of the model are examined and a simulation algorithm is developed through which the stochastic properties of summaries of the extremal txhaviour of the chain are evaluated. The approach is illustrated through a simulation study of extremal events of Gaussian autoregressive processes and an application to temperature data.

On the blockwise bootstrap for empirical processes for stationary sequences

Abstract: In this paper, we study the weak convergence to an appropriate Gaussian process of the empirical process of the block-based bootstrap estimator proposed by Kunsch for stationary sequences. The classes of processes investigated are weak dependent and associated sequences. We also prove that, differently from the independent situation, the bootstrapped estimator of the mean of certain dependent sequences satisfies the central limit theorem while the mean of the original sequence does not.

Fractional Brownian Motion and the Markov Property

Abstract: Fractional Brownian motion belongs to a class of long memory Gaussian processes that can be represented as linear functionals of an infinite dimensional Markov process. This leads naturally to: An efficient algorithm to approximate the process. An ergodic theorem which applies to functionals of the type $$\int_0^t \phi(V_h(s)),ds \quad\text{where}\quad V_h(s)=\int_0^s h(s-u), dB_u,.$$ where $B$ is a real Brownian motion.

Discovering Hidden Features with Gaussian Processes Regression

Abstract: In Gaussian process regression the covariance between the outputs at input locations x and x′ is usually assumed to depend on the distance (x− x′) W (x− x′), where W is a positive definite matrix. W is often taken to be diagonal, but if we allow W to be a general positive definite matrix which can be tuned on the basis of training data, then an eigen-analysis of W shows that we are effectively creating hidden features, where the dimensionality of the hidden-feature space is determined by the data. We demonstrate the superiority of predictions using the general matrix over those based on a diagonal matrix on two test problems.