https://rss.onlinelibrary.wiley.com/doi/pdf/10.1111/j.2517-6161.1977.tb01600.x

Maximum likelihood from incomplete data via the EM algorithm

A comprehensive and self-contained introduction to Gaussian processes, which provide a principled, practical, probabilistic approach to learning in kernel machines. Gaussian processes (GPs) provide a principled, practical, probabilistic approach to learning in kernel machines. GPs have received increased attention in the machine-learning community over the past decade, and this book provides a long-needed systematic and unified treatment of theoretical and practical aspects of GPs in machine learning. The treatment is comprehensive and self-contained, targeted at researchers and students in machine learning and applied statistics. The book deals with the supervised-learning problem for both regression and classification, and includes detailed algorithms. A wide variety of covariance (kernel) functions are presented and their properties discussed. Model selection is discussed both from a Bayesian and a classical perspective. Many connections to other well-known techniques from machine learning and statistics are discussed, including support-vector machines, neural networks, splines, regularization networks, relevance vector machines and others. Theoretical issues including learning curves and the PAC-Bayesian framework are treated, and several approximation methods for learning with large datasets are discussed. The book contains illustrative examples and exercises, and code and datasets are available on the Web. Appendixes provide mathematical background and a discussion of Gaussian Markov processes.

/pdf/gaussian-processes-for-machine-learning-40fp9c190d.pdf

Gaussian Processes for Machine Learning

Many scientific phenomena are now investigated by complex computer models or codes A computer experiment is a number of runs of the code with various inputs A feature of many computer experiments is that the output is deterministic--rerunning the code with the same inputs gives identical observations Often, the codes are computationally expensive to run, and a common objective of an experiment is to fit a cheaper predictor of the output to the data Our approach is to model the deterministic output as the realization of a stochastic process, thereby providing a statistical basis for designing experiments (choosing the inputs) for efficient prediction With this model, estimates of uncertainty of predictions are also available Recent work in this area is reviewed, a number of applications are discussed, and we demonstrate our methodology with an example

/pdf/the-design-and-analysis-of-computer-experiments-1kfo89lyf2.pdf

The design and analysis of computer experiments

We describe the maximum-likelihood parameter estimation problem and how the ExpectationMaximization (EM) algorithm can be used for its solution. We first describe the abstract form of the EM algorithm as it is often given in the literature. We then develop the EM parameter estimation procedure for two applications: 1) finding the parameters of a mixture of Gaussian densities, and 2) finding the parameters of a hidden Markov model (HMM) (i.e., the Baum-Welch algorithm) for both discrete and Gaussian mixture observation models. We derive the update equations in fairly explicit detail but we do not prove any convergence properties. We try to emphasize intuition rather than mathematical rigor.

/pdf/a-gentle-tutorial-of-the-em-algorithm-and-its-application-to-4o5uraouqh.pdf

A gentle tutorial of the em algorithm and its application to parameter estimation for Gaussian mixture and hidden Markov models

SUMMARY This paper is concerned with the non-parametric estimation of a distribution function F, when the data are incomplete due to grouping, censoring and/or truncation. Using the idea of self-consistency, a simple algorithm is constructed and shown to converge monotonically to yield a maximum likelihood estimate of F. An application to hypothesis testing is indicated.

/pdf/the-empirical-distribution-function-with-arbitrarily-grouped-1bkzs8oi0y.pdf

The Empirical Distribution Function with Arbitrarily Grouped, Censored, and Truncated Data

A method is presented which, in many cases, appears to be an improvement over the standard approach to the polynomial regression problem. This improvement is achieved by focusing attention on the deviation of the polynomial representation from the true underlying function. By fully utilizing the nature of this deviation, a model is constructed in which its properties are represented in terms of a Bayesian prior distribution. The model is analyzed to give parameter estimates and predictions of further observations. Comparisons are made with standard least squares procedures when the true underlying model is (a) quadratic and (b) linear and quadratic with a superimposed sine wave.

A Bayesian approach to model inadequacy for polynomial regression

SUMMARY The likelihood equations are derived for the estimation of the parameters of an exponential family from a Type I censored sample and are shown to have an interpretation which suggests a particular iterative method of solution. Some results relevant to the convergence properties of this method are given and the asymptotic variance-covariance matrix is derived. The method is illustrated by an example in which it is compared with an alternative method in the case of estimation for a doubly censored normal distribution.

Estimation from a censored sample for the exponential family

The efficient estimator and mean square error for the mean of a finite population are determined when a sample is available from a population that is assumed to be generated by a simple Markov process. The optimal sample allocation is derived and the consequences of ignoring the autocorrelation are examined. The best linear estimator and mean square error for random sampling are also determined and compared with the corresponding results for systematic sampling.

https://www.jstor.org/stable/pdf/2334551.pdf

B. J. N. Blight

Papers

A Bayesian approach to model inadequacy for polynomial regression

Estimation from a censored sample for the exponential family

Sampling from an autocorrelated finite population