A comprehensive and self-contained introduction to Gaussian processes, which provide a principled, practical, probabilistic approach to learning in kernel machines. Gaussian processes (GPs) provide a principled, practical, probabilistic approach to learning in kernel machines. GPs have received increased attention in the machine-learning community over the past decade, and this book provides a long-needed systematic and unified treatment of theoretical and practical aspects of GPs in machine learning. The treatment is comprehensive and self-contained, targeted at researchers and students in machine learning and applied statistics. The book deals with the supervised-learning problem for both regression and classification, and includes detailed algorithms. A wide variety of covariance (kernel) functions are presented and their properties discussed. Model selection is discussed both from a Bayesian and a classical perspective. Many connections to other well-known techniques from machine learning and statistics are discussed, including support-vector machines, neural networks, splines, regularization networks, relevance vector machines and others. Theoretical issues including learning curves and the PAC-Bayesian framework are treated, and several approximation methods for learning with large datasets are discussed. The book contains illustrative examples and exercises, and code and datasets are available on the Web. Appendixes provide mathematical background and a discussion of Gaussian Markov processes.

/pdf/gaussian-processes-for-machine-learning-40fp9c190d.pdf

Gaussian Processes for Machine Learning

Many scientific phenomena are now investigated by complex computer models or codes A computer experiment is a number of runs of the code with various inputs A feature of many computer experiments is that the output is deterministic--rerunning the code with the same inputs gives identical observations Often, the codes are computationally expensive to run, and a common objective of an experiment is to fit a cheaper predictor of the output to the data Our approach is to model the deterministic output as the realization of a stochastic process, thereby providing a statistical basis for designing experiments (choosing the inputs) for efficient prediction With this model, estimates of uncertainty of predictions are also available Recent work in this area is reviewed, a number of applications are discussed, and we demonstrate our methodology with an example

/pdf/the-design-and-analysis-of-computer-experiments-1kfo89lyf2.pdf

The design and analysis of computer experiments

We describe the maximum-likelihood parameter estimation problem and how the ExpectationMaximization (EM) algorithm can be used for its solution. We first describe the abstract form of the EM algorithm as it is often given in the literature. We then develop the EM parameter estimation procedure for two applications: 1) finding the parameters of a mixture of Gaussian densities, and 2) finding the parameters of a hidden Markov model (HMM) (i.e., the Baum-Welch algorithm) for both discrete and Gaussian mixture observation models. We derive the update equations in fairly explicit detail but we do not prove any convergence properties. We try to emphasize intuition rather than mathematical rigor.

/pdf/a-gentle-tutorial-of-the-em-algorithm-and-its-application-to-4o5uraouqh.pdf

A gentle tutorial of the em algorithm and its application to parameter estimation for Gaussian mixture and hidden Markov models

S We consider prediction and uncertainty analysis for complex computer codes which can be run at different levels of sophistication. In particular, we wish to improve efficiency by combining expensive runs of the most complex versions of the code with relatively cheap runs from one or more simpler approximations. A Bayesian approach is described in which prior beliefs about the codes are represented in terms of Gaussian processes. An example is presented using two versions of an oil reservoir simulator. 1. C  Complex mathematical models, implemented in large computer codes, have been used to study real systems in many areas of scientific research (Sacks et al., 1989), usually because physical experimentation is too costly and sometimes impossible, as in the case of large environmental systems. A ‘computer experiment’ involves running the code with various input values for the purpose of learning something about the real system. Often a simulator can be run at different levels of complexity, with versions ranging from the most sophisticated high level code to the most basic. For example, in § 4 we consider two codes which simulate oil pressure at a well of a hydrocarbon reservoir. Both codes use finite element analysis, in which the rocks comprising the reservoir are represented by small interacting grid blocks. The flow of oil within the reservoir can be simulated by considering the interaction between the blocks. The two codes differ in the resolution of the grid, so that we have a very accurate, slow version using many small blocks and a crude approximation using large blocks which runs much faster. Alternatively, a mathematical model could be expanded to include more of the scientific laws underlying the physical processes. Simple, fast versions of the code may well include the most important features, and are useful for preliminary investigations. In real-time applications the number of runs from a high level simulator may be limited by expense. Then there is a need to trade-off the complexity of the expensive code with the availability of the simpler approximations. The purpose of the current paper is to explore ways in which runs from several levels of a code can be used to make inference about the output from the most complex code. We may also have uncertainty about values for the input parameters which apply in any given application. Uncertainty analysis of computer codes describes how this uncertainty on the inputs affects our uncertainty about the output.

https://www.jstor.org/stable/pdfplus/2673557.pdf

Predicting the output from a complex computer code when fast approximations are available

This article is concerned with prediction of a function y(t) over a (multidimensional) domain T, given the function values at a set of “sites” {t (1), t (2), …, t (n)} in T, and with the design, that is, with the selection of those sites. The motivating application is the design and analysis of computer experiments, where t determines the input to a computer model of a physical or behavioral system, and y(t) is a response that is part of the output or is calculated from it. Following a Bayesian formulation, prior uncertainty about the function y is expressed by means of a random function Y, which is taken here to be a Gaussian stochastic process. The mean of the posterior process can be used as the prediction function ŷ(t), and the variance can be used as a measure of uncertainty. This kind of approach has been used previously in Bayesian interpolation and is strongly related to the kriging methods used in geostatistics. Here emphasis is placed on product linear and product cubic correlation func...

Bayesian Prediction of Deterministic Functions, with Applications to the Design and Analysis of Computer Experiments

A method is presented which, in many cases, appears to be an improvement over the standard approach to the polynomial regression problem. This improvement is achieved by focusing attention on the deviation of the polynomial representation from the true underlying function. By fully utilizing the nature of this deviation, a model is constructed in which its properties are represented in terms of a Bayesian prior distribution. The model is analyzed to give parameter estimates and predictions of further observations. Comparisons are made with standard least squares procedures when the true underlying model is (a) quadratic and (b) linear and quadratic with a superimposed sine wave.

L. Ott

Papers

A Bayesian approach to model inadequacy for polynomial regression