THE DESIGN AND ANALYSIS OF EXPERIMENTS. By Oscar Kempthorne. New York, John Wiley and Sons, Inc., 1952. 631 pp. $8.50. This book by a teacher of statistics (as well as a consultant for \"experimenters\") is a comprehensive study of the philosophical background for the statistical design of experiment. It is necessary to have some facility with algebraic notation and manipulation to be able to use the volume intelligently. The problems are presented from the theoretical point of view, without such practical examples as would be helpful for those not acquainted with mathematics. The mathematical justification for the techniques is given. As a somewhat advanced treatment of the design and analysis of experiments, this volume will be interesting and helpful for many who approach statistics theoretically as well as practically. With emphasis on the \"why,\" and with description given broadly, the author relates the subject matter to the general theory of statistics and to the general problem of experimental inference. MARGARET J. ROBERTSON

The Design and Analysis of Experiments

A comprehensive and self-contained introduction to Gaussian processes, which provide a principled, practical, probabilistic approach to learning in kernel machines. Gaussian processes (GPs) provide a principled, practical, probabilistic approach to learning in kernel machines. GPs have received increased attention in the machine-learning community over the past decade, and this book provides a long-needed systematic and unified treatment of theoretical and practical aspects of GPs in machine learning. The treatment is comprehensive and self-contained, targeted at researchers and students in machine learning and applied statistics. The book deals with the supervised-learning problem for both regression and classification, and includes detailed algorithms. A wide variety of covariance (kernel) functions are presented and their properties discussed. Model selection is discussed both from a Bayesian and a classical perspective. Many connections to other well-known techniques from machine learning and statistics are discussed, including support-vector machines, neural networks, splines, regularization networks, relevance vector machines and others. Theoretical issues including learning curves and the PAC-Bayesian framework are treated, and several approximation methods for learning with large datasets are discussed. The book contains illustrative examples and exercises, and code and datasets are available on the Web. Appendixes provide mathematical background and a discussion of Gaussian Markov processes.

/pdf/gaussian-processes-for-machine-learning-40fp9c190d.pdf

Gaussian Processes for Machine Learning

Many scientific phenomena are now investigated by complex computer models or codes A computer experiment is a number of runs of the code with various inputs A feature of many computer experiments is that the output is deterministic--rerunning the code with the same inputs gives identical observations Often, the codes are computationally expensive to run, and a common objective of an experiment is to fit a cheaper predictor of the output to the data Our approach is to model the deterministic output as the realization of a stochastic process, thereby providing a statistical basis for designing experiments (choosing the inputs) for efficient prediction With this model, estimates of uncertainty of predictions are also available Recent work in this area is reviewed, a number of applications are discussed, and we demonstrate our methodology with an example

/pdf/the-design-and-analysis-of-computer-experiments-1kfo89lyf2.pdf

The design and analysis of computer experiments

Statistics for Spatial Data.

We consider prediction and uncertainty analysis for systems which are approximated using complex mathematical models. Such models, implemented as computer codes, are often generic in the sense that by a suitable choice of some of the model's input parameters the code can be used to predict the behaviour of the system in a variety of specific applications. However, in any specific application the values of necessary parameters may be unknown. In this case, physical observations of the system in the specific context are used to learn about the unknown parameters. The process of fitting the model to the observed data by adjusting the parameters is known as calibration. Calibration is typically effected by ad hoc fitting, and after calibration the model is used, with the fitted input values, to predict the future behaviour of the system. We present a Bayesian calibration technique which improves on this traditional approach in two respects. First, the predictions allow for all sources of uncertainty, including the remaining uncertainty over the fitted parameters. Second, they attempt to correct for any inadequacy of the model which is revealed by a discrepancy between the observed data and the model predictions from even the best-fitting parameter values. The method is illustrated by using data from a nuclear radiation release at Tomsk, and from a more complex simulated nuclear accident exercise.

https://www.jstor.org/stable/pdfplus/2680584.pdf

Bayesian Calibration of computer models

In the last ten to fifteen years many phenomena that could be studied only using physical experiments can now be studied by computer experiments Advances in the mathematical modeling of many physical processes, in algorithms for solving mathematical systems, and in computer speeds, have combined to make it possible to augment or replace physical experiments with computer experiments In a computer experiment, a response z( x), usually deterministic, is computed for each set of input variables, x, according to an experimental design strategy This strategy is determined by the goal of the experiment and depends, for example, on whether response prediction at unsampled input sites or response optimization is of primary interest 
We are concerned with the commonly occuring situation in which there are two types of input variables: suppose x = ( xc, x e) where xc is a set of “control” (manufacturing) variables and xe is a set of “environmental” (noise) variables Manufacturing variables can be controlled while noise variables are not controllable but have values governed by some probability distribution 
For single response settings, we introduce a sequential experimental design for finding the optimum of e(x c) = E[z(x c, Xe)], where the expectation is taken over the distribution of the environmental variables For bivariate response settings, we introduce a sequential experimental design for finding the constrained optimum of e1( xc)) = E[z( xc, X e)], subject to e2 (x c) = E[z2(x c, Xe)] ≤ U The approach is Bayesian; the prior information is that the responses are a draw from a stationary Gaussian stochastic process with correlation function belonging to a parametric family with unknown parameters The idea of the methods is to compute the posterior expected “improvement” over the current optimum for each untested site; the design selects the next site to maximize the expected improvement Both procedures are illustrated by examples utilizing test functions from the numerical optimization literature

/pdf/sequential-design-of-computer-experiments-to-minimize-16mtuwtcy5.pdf

Sequential design of computer experiments to minimize integrated response functions

Computer simulations have become increasingly popular as a method for studying physical processes that are difficult to study directly. These simulations are based on complex mathematical models that are believed to accurately describe the physical process. We consider the situation where these simulations take a long time to run (several hours or days) and hence can only be conducted a limited number of times. As a result, the inputs (design) at which to run the simulations must be chosen carefully. For the purpose of fitting a response surface to the output from these simulations, a variety of designs based on a fixed number of runs have been proposed. 
In this thesis, we consider sequential adaptive designs as an “efficient” alternative to fixed-point designs. We propose new adaptive design criteria based on a cross validation approach and on an expected improvement criterion, the latter inspired by a criterion originally proposed for global optimization. We compare these new designs with others in the literature in an empirical study and they shown to perform well. 
The issue of robustness for the proposed sequential adaptive designs is also addressed in this thesis. While we find that sequential adaptive designs are potentially more effective and efficient than fixed-point designs, issues such as numerical instability do arise. We address these concerns and also propose a diagnostic tool based on cross validation prediction error to improve the performance of sequential designs. 
We are also interested in the design of computer experiments where there are control variables and environmental (noise) variables. We extend the implementation of the proposed sequential designs to achieve a good fit of the unknown integrated response surface (i.e., the averaged response surface taken over the distributions of the environmental variables) using output from the simulations. The goal is to find an optimal choice of the control variables while taking into account the distributions of the noise variables.

Sequential adaptive designs in computer experiments for response surface model fit

This article introduces a Bayesian methodology for the prediction for computer experiments having quantitative and qualitative inputs. The proposed model is a hierarchical Bayesian model with conditional Gaussian stochastic process components. For each of the qualitative inputs, our model assumes that the outputs corresponding to different levels of the qualitative input have “similar” functional behavior in the quantitative inputs. The predictive accuracy of this method is compared with the predictive accuracies of alternative proposals in examples. The method is illustrated in a biomechanical engineering application.

Prediction for Computer Experiments Having Quantitative and Qualitative Input Variables

This 1998 book describes the sampling and statistical methods used most often by behavioral ecologists and field biologists. Written by a biologist and two statisticians, it provides a rigorous discussion together with worked examples of statistical concepts and methods that are generally not covered in introductory courses, and which are consequently poorly understood and applied by field biologists. The first section reviews important issues such as defining the statistical population and the sampling plan when using non-random methods for sample selection, bias, interpretation of statistical tests, confidence intervals and multiple comparisons. After a detailed discussion of sampling methods and multiple regression, subsequent chapters discuss specialized problems such as pseudoreplication, and their solutions. It will quickly become the statistical handbook for all field biologists.

http://catdir.loc.gov/catdir/samples/cam031/97035086.pdf

William I. Notz

Papers

The design and analysis of computer experiments

Sequential design of computer experiments to minimize integrated response functions

Sequential adaptive designs in computer experiments for response surface model fit

Prediction for Computer Experiments Having Quantitative and Qualitative Input Variables

Sampling and statistical methods for behavioral ecologists