In many engineering optimization problems, the number of function evaluations is severely limited by time or cost. These problems pose a special challenge to the field of global optimization, since existing methods often require more function evaluations than can be comfortably afforded. One way to address this challenge is to fit response surfaces to data collected by evaluating the objective and constraint functions at a few points. These surfaces can then be used for visualization, tradeoff analysis, and optimization. In this paper, we introduce the reader to a response surface methodology that is especially good at modeling the nonlinear, multimodal functions that often occur in engineering. We then show how these approximating functions can be used to construct an efficient global optimization algorithm with a credible stopping rule. The key to using response surfaces for global optimization lies in balancing the need to exploit the approximating surface (by sampling where it is minimized) with the need to improve the approximation (by sampling where prediction error may be high). Striking this balance requires solving certain auxiliary problems which have previously been considered intractable, but we show how these computational obstacles can be overcome.

/pdf/efficient-global-optimization-of-expensive-black-box-4m8fv6oubz.pdf

Efficient Global Optimization of Expensive Black-Box Functions

1 Linear Prediction.- 1.1 Introduction.- 1.2 Best linear prediction.- Exercises.- 1.3 Hilbert spaces and prediction.- Exercises.- 1.4 An example of a poor BLP.- Exercises.- 1.5 Best linear unbiased prediction.- Exercises.- 1.6 Some recurring themes.- The Matern model.- BLPs and BLUPs.- Inference for differentiable random fields.- Nested models are not tenable.- 1.7 Summary of practical suggestions.- 2 Properties of Random Fields.- 2.1 Preliminaries.- Stationarity.- Isotropy.- Exercise.- 2.2 The turning bands method.- Exercise.- 2.3 Elementary properties of autocovariance functions.- Exercise.- 2.4 Mean square continuity and differentiability.- Exercises.- 2.5 Spectral methods.- Spectral representation of a random field.- Bochner's Theorem.- Exercises.- 2.6 Two corresponding Hilbert spaces.- An application to mean square differentiability.- Exercises.- 2.7 Examples of spectral densities on 112.- Rational spectral densities.- Principal irregular term.- Gaussian model.- Triangular autocovariance functions.- Matern class.- Exercises.- 2.8 Abelian and Tauberian theorems.- Exercises.- 2.9 Random fields with nonintegrable spectral densities.- Intrinsic random functions.- Semivariograms.- Generalized random fields.- Exercises.- 2.10 Isotropic autocovariance functions.- Characterization.- Lower bound on isotropic autocorrelation functions.- Inversion formula.- Smoothness properties.- Matern class.- Spherical model.- Exercises.- 2.11 Tensor product autocovariances.- Exercises.- 3 Asymptotic Properties of Linear Predictors.- 3.1 Introduction.- 3.2 Finite sample results.- Exercise.- 3.3 The role of asymptotics.- 3.4 Behavior of prediction errors in the frequency domain.- Some examples.- Relationship to filtering theory.- Exercises.- 3.5 Prediction with the wrong spectral density.- Examples of interpolation.- An example with a triangular autocovariance function.- More criticism of Gaussian autocovariance functions.- Examples of extrapolation.- Pseudo-BLPs with spectral densities misspecified at high frequencies.- Exercises.- 3.6 Theoretical comparison of extrapolation and ointerpolation.- An interpolation problem.- An extrapolation problem.- Asymptotics for BLPs.- Inefficiency of pseudo-BLPs with misspecified high frequency behavior.- Presumed mses for pseudo-BLPs with misspecified high frequency behavior.- Pseudo-BLPs with correctly specified high frequency behavior.- Exercises.- 3.7 Measurement errors.- Some asymptotic theory.- Exercises.- 3.8 Observations on an infinite lattice.- Characterizing the BLP.- Bound on fraction of mse of BLP attributable to a set of frequencies.- Asymptotic optimality of pseudo-BLPs.- Rates of convergence to optimality.- Pseudo-BLPs with a misspecified mean function.- Exercises.- 4 Equivalence of Gaussian Measures and Prediction.- 4.1 Introduction.- 4.2 Equivalence and orthogonality of Gaussian measures.- Conditions for orthogonality.- Gaussian measures are equivalent or orthogonal.- Determining equivalence or orthogonality for periodic random fields.- Determining equivalence or orthogonality for nonperiodic random fields.- Measurement errors and equivalence and orthogonality.- Proof of Theorem 1.- Exercises.- 4.3 Applications of equivalence of Gaussian measures to linear prediction.- Asymptotically optimal pseudo-BLPs.- Observations not part of a sequence.- A theorem of Blackwell and Dubins.- Weaker conditions for asymptotic optimality of pseudo-BLPs.- Rates of convergence to asymptotic optimality.- Asymptotic optimality of BLUPs.- Exercises.- 4.4 Jeffreys's law.- A Bayesian version.- Exercises.- 5 Integration of Random Fields.- 5.1 Introduction.- 5.2 Asymptotic properties of simple average.- Results for sufficiently smooth random fields.- Results for sufficiently rough random fields.- Exercises.- 5.3 Observations on an infinite lattice.- Asymptotic mse of BLP.- Asymptotic optimality of simple average.- Exercises.- 5.4 Improving on the sample mean.- Approximating $$\int_0^1 {\exp } (ivt)dt$$.- Approximating $$\int_{{{[0,1]}^d}} {\exp (i{\omega ^T}x)} dx$$ in more than one dimension.- Asymptotic properties of modified predictors.- Are centered systematic samples good designs?.- Exercises.- 5.5 Numerical results.- Exercises.- 6 Predicting With Estimated Parameters.- 6.1 Introduction.- 6.2 Microergodicity and equivalence and orthogonality of Gaussian measures.- Observations with measurement error.- Exercises.- 6.3 Is statistical inference for differentiable processes possible?.- An example where it is possible.- Exercises.- 6.4 Likelihood Methods.- Restricted maximum likelihood estimation.- Gaussian assumption.- Computational issues.- Some asymptotic theory.- Exercises.- 6.5 Matern model.- Exercise.- 6.6 A numerical study of the Fisher information matrix under the Matern model.- No measurement error and?unknown.- No measurement error and?known.- Observations with measurement error.- Conclusions.- Exercises.- 6.7 Maximum likelihood estimation for a periodic version of the Matern model.- Discrete Fourier transforms.- Periodic case.- Asymptotic results.- Exercises.- 6.8 Predicting with estimated parameters.- Jeffreys's law revisited.- Numerical results.- Some issues regarding asymptotic optimality.- Exercises.- 6.9 An instructive example of plug-in prediction.- Behavior of plug-in predictions.- Cross-validation.- Application of Matern model.- Conclusions.- Exercises.- 6.10 Bayesian approach.- Application to simulated data.- Exercises.- A Multivariate Normal Distributions.- B Symbols.- References.

Interpolation of Spatial Data: Some Theory for Kriging

Species distributional or trait data based on range map (extent-of-occurrence) or atlas survey data often display spatial autocorrelation, i.e. locations close to each other exhibit more similar values than those further apart. If this pattern remains present in the residuals of a statistical model based on such data, one of the key assumptions of standard statistical analyses, that residuals are independent and identically distributed (i.i.d), is violated. The violation of the assumption of i.i.d. residuals may bias parameter estimates and can increase type I error rates (falsely rejecting the null hypothesis of no effect). While this is increasingly recognised by researchers analysing species distribution data, there is, to our knowledge, no comprehensive overview of the many available spatial statistical methods to take spatial autocorrelation into account in tests of statistical significance. Here, we describe six different statistical approaches to infer correlates of species’ distributions, for both presence/absence (binary response) and species abundance data (poisson or normally distributed response), while accounting for spatial autocorrelation in model residuals: autocovariate regression; spatial eigenvector mapping; generalised least squares; (conditional and simultaneous) autoregressive models and generalised estimating equations. A comprehensive comparison of the relative merits of these methods is beyond the scope of this paper. To demonstrate each method’s implementation, however, we undertook preliminary tests based on simulated data. These preliminary tests verified that most of the spatial modeling techniques we examined showed good type I error control and precise parameter estimates, at least when confronted with simplistic simulated data containing

/pdf/methods-to-account-for-spatial-autocorrelation-in-the-5fbbuizhqs.pdf

Methods to account for spatial autocorrelation in the analysis of species distributional data : a review

https://www.clw.csiro.au/aclep/documents/McBratney%20et%20al.%202003.pdf

On digital soil mapping

Covering point process theory, random geometric graphs and coverage processes, this rigorous introduction to stochastic geometry will enable you to obtain powerful, general estimates and bounds of wireless network performance and make good design choices for future wireless architectures and protocols that efficiently manage interference effects. Practical engineering applications are integrated with mathematical theory, with an understanding of probability the only prerequisite. At the same time, stochastic geometry is connected to percolation theory and the theory of random geometric graphs and accompanied by a brief introduction to the R statistical computing language. Combining theory and hands-on analytical techniques with practical examples and exercises, this is a comprehensive guide to the spatial stochastic models essential for modelling and analysis of wireless network performance.

/pdf/stochastic-geometry-for-wireless-networks-50d7xggv33.pdf

Stochastic Geometry for Wireless Networks

5. Statistics for Spatial Data. By N. Cressie. ISBN 0 471 84336 9. Wiley, Chichester, 1991. 900 pp. £71.00.

N. Cressie

Papers

5. Statistics for Spatial Data