Showing papers in "Technometrics in 2009"
TL;DR: In this paper, the authors quantify two key characteristics of computer codes that affect the sample size required for a desired level of accuracy when approximating the code via a Gaussian process (GP) and provide reasons and evidence supporting the informal rule that the number of runs for an effective initial computer experiment should be about 10 times the input dimension.
Abstract: We provide reasons and evidence supporting the informal rule that the number of runs for an effective initial computer experiment should be about 10 times the input dimension. Our arguments quantify two key characteristics of computer codes that affect the sample size required for a desired level of accuracy when approximating the code via a Gaussian process (GP). The first characteristic is the total sensitivity of a code output variable to all input variables; the second corresponds to the way this total sensitivity is distributed across the input variables, specifically the possible presence of a few prominent input factors and many impotent ones (i.e., effect sparsity). Both measures relate directly to the correlation structure in the GP approximation of the code. In this way, the article moves toward a more formal treatment of sample size for a computer experiment. The evidence supporting these arguments stems primarily from a simulation study and via specific codes modeling climate and ligand activa...
591 citations
TL;DR: Diagnostics are presented to validate and assess the adequacy of a Gaussian process emulator as surrogate for the simulator and take care to account for correlation between the validation data.
Abstract: Mathematical models, usually implemented in computer programs known as simulators, are widely used in all areas of science and technology to represent complex real-world phenomena. Simulators are often so complex that they take appreciable amounts of computer time or other resources to run. In this context, a methodology has been developed based on building a statistical representation of the simulator, known as an emulator. The principal approach to building emulators uses Gaussian processes. This work presents some diagnostics to validate and assess the adequacy of a Gaussian process emulator as surrogate for the simulator. These diagnostics are based on comparisons between simulator outputs and Gaussian process emulator outputs for some test data, known as validation data, defined by a sample of simulator runs not used to build the emulator. Our diagnostics take care to account for correlation between the validation data. To illustrate a validation procedure, we apply these diagnostics to two different...
397 citations
TL;DR: This work proposes an approach that automatically explores the space while simultaneously fitting the response surface, using predictive uncertainty to guide subsequent experimental runs, and develops an adaptive sequential design framework to cope with an asynchronous, random, agent–based supercomputing environment.
Abstract: Computer experiments often are performed to allow modeling of a response surface of a physical experiment that can be too costly or difficult to run except by using a simulator. Running the experiment over a dense grid can be prohibitively expensive, yet running over a sparse design chosen in advance can result in insufficient information in parts of the space, particularly when the surface calls for a nonstationary model. We propose an approach that automatically explores the space while simultaneously fitting the response surface, using predictive uncertainty to guide subsequent experimental runs. We use the newly developed Bayesian treed Gaussian process as the surrogate model; a fully Bayesian approach allows explicit measures of uncertainty. We develop an adaptive sequential design framework to cope with an asynchronous, random, agent–based supercomputing environment by using a hybrid approach that melds optimal strategies from the statistics literature with flexible strategies from the active learni...
192 citations
TL;DR: A Bayesian approach to validating computer models that overcomes several difficulties of the frequentist approach proposed by Oberkampf and Barone and provides a clear decomposition of the expected prediction error of the true output.
Abstract: Computer models are mathematical representations of real systems developed for understanding and investigating the systems. They are particularly useful when physical experiments are either cost- prohibitive or time-prohibitive. Before a computer model is used, it often must be validated by comparing the computer outputs with physical experiments. This article proposes a Bayesian approach to validating computer models that overcomes several difficulties of the frequentist approach proposed by Oberkampf and Barone. Kennedy and O’Hagan proposed a similar Bayesian approach. A major difference between their approach and ours is that theirs focuses on directly deriving the posterior of the true output, whereas our approach focuses on first deriving the posteriors of the computer model and model bias (difference between computer and true outputs) separately, then deriving the posterior of the true output. As a result, our approach provides a clear decomposition of the expected prediction error of the true outpu...
131 citations
TL;DR: Risk assessment of rare natural hazards, such as large volcanic block and ash or pyroclastic flows, is addressed through a combination of computer modeling, statistical modeling, and extreme-event probability computation.
Abstract: Risk assessment of rare natural hazards, such as large volcanic block and ash or pyroclastic flows, is addressed. Assessment is approached through a combination of computer modeling, statistical modeling, and extreme-event probability computation. A computer model of the natural hazard is used to provide the needed extrapolation to unseen parts of the hazard space. Statistical modeling of the available data is needed to determine the initializing distribution for exercising the computer model. In dealing with rare events, direct simulations involving the computer model are prohibitively expensive. The solution instead requires a combination of adaptive design of computer model approximations (emulators) and rare event simulation. The techniques that are developed for risk assessment are illustrated on a test-bed example involving volcanic flow.
127 citations
TL;DR: In this article, the sensitivity indexes when the inputs of a model are not independent are derived from local polynomial techniques, which have good theoretical properties, which are illustrated through analytical examples.
Abstract: Sensitivity indexes when the inputs of a model are not independent are derived from local polynomial techniques. Two original estimators based on local polynomial smoothers are proposed. Both have good theoretical properties, which are illustrated through analytical examples. Comparison with the Bayesian approach developed by Oakley and O’Hagan (2004) is also performed. The two proposed estimators are used to carry out a sensitivity analysis on two real case models with correlated parameters.
122 citations
TL;DR: This work presents a clever quadrature scheme that greatly improves the feasibility of using Bayesian design criteria, and illustrates the method on some designed experiments.
Abstract: Experimental design in nonlinear settings is complicated by the fact that the efficiency of a design depends on the unknown parameter values Thus good designs need to be efficient over a range of likely parameter values Bayesian design criteria provide a natural framework for achieving such robustness, by averaging local design criteria over a prior distribution on the parameters A major drawback to the use of such criteria is the heavy computational burden that they impose We present a clever quadrature scheme that greatly improves the feasibility of using Bayesian design criteria We illustrate the method on some designed experiments
121 citations
TL;DR: New small sample-based tests are derived and the Type I error rate and statistical power of these tests are studied via simulation to reveal that in terms of maintaining Type Ierror rate, the new tests perform extremely well as long as the shape parameter is not too small, and even then the results are only slightly conservative.
Abstract: The gamma distribution is relevant to numerous areas of application in the physical, environmental, and biological sciences. The focus of this paper is on testing the shape, scale, and mean of the gamma distribution. Testing the shape parameter of the gamma distribution is relevant to failure time modeling where it can be used to determine if the failure rate is constant, increasing, or decreasing. Testing the scale parameter is also relevant to problems in survival analysis, where when the shape parameter κ=1, the reciprocal of the scale parameter measures the hazard function. Finally, testing the mean of the gamma distribution allows us to determine if the average concentration of an environmental contaminant is higher, lower, or equivalent to a health-based standard. In this paper, we first derive new small sample-based tests and then via simulation, we study the Type I error rate and statistical power of these tests. Results of these simulation studies reveal that in terms of maintaining Type I error ...
98 citations
TL;DR: By combining statistical emulation using treed Gaussian processes with pattern search optimization, this work is able to perform robust local optimization more efficiently and effectively than when using either method alone.
Abstract: Optimization for complex systems in engineering often involves the use of expensive computer simulation. By combining statistical emulation using treed Gaussian processes with pattern search optimization, we are able to perform robust local optimization more efficiently and effectively than when using either method alone. Our approach is based on the augmentation of local search patterns with location sets generated through improvement prediction over the input space. We further develop a computational framework for asynchronous parallel implementation of the optimization algorithm. We demonstrate our methods on two standard test problems and our motivating example of calibrating a circuit device simulator.
90 citations
TL;DR: This work proposes a Bayesian nonparametric regression model for curve fitting and variable selection, uses the smoothing splines ANOVA framework, and uses stochastic search variable selection through Markov chain Monte Carlo sampling to search for models that fit the data well.
Abstract: With many predictors, choosing an appropriate subset of the covariates is a crucial, and difficult, step in nonparametric regression. We propose a Bayesian nonparametric regression model for curve-fitting and variable selection. We use the smoothing spline ANOVA framework to decompose the regression function into interpretable main effect and interaction functions. Stochastic search variable selection via MCMC sampling is used to search for models that fit the data well. Also, we show that variable selection is highly-sensitive to hyperparameter choice and develop a technique to select hyperparameters that control the long-run false positive rate. The method is used to build an emulator for a complex computer model for two-phase fluid flow.
89 citations
TL;DR: A Bayesian methodology for the prediction for computer experiments having quantitative and qualitative inputs is introduced and the predictive accuracy of this method is compared with the predictive accuracies of alternative proposals in examples.
Abstract: This article introduces a Bayesian methodology for the prediction for computer experiments having quantitative and qualitative inputs. The proposed model is a hierarchical Bayesian model with conditional Gaussian stochastic process components. For each of the qualitative inputs, our model assumes that the outputs corresponding to different levels of the qualitative input have “similar” functional behavior in the quantitative inputs. The predictive accuracy of this method is compared with the predictive accuracies of alternative proposals in examples. The method is illustrated in a biomechanical engineering application.
TL;DR: An efficient new estimator is proposed, which is computationally easy, free from the problems observed in traditional approaches, and performs well compared with existing estimators.
Abstract: The generalized Pareto distribution (GPD) is widely used to model extreme values, for example, exceedences over thresholds, in modeling floods. Existing methods for estimating parameters have theoretical or computational defects. An efficient new estimator is proposed, which is computationally easy, free from the problems observed in traditional approaches, and performs well compared with existing estimators. A numerical example involving heights of waves is used to illustrate the various methods and tests of fit are performed to compare them.
TL;DR: A model and methods are described to predict the failure time distribution of a newly designed product with two failure modes and how this model can be used to predict a future component or product operating in the same use environment.
Abstract: Accelerated life tests (ALTs) provide timely assessments of the reliability of materials, components, and subsystems. ALTs can be run at any of these levels or at the full-system level. Sometimes ALTs generate multiple failure modes. A frequently asked question near the end of an ALT program is “what do these test results say about field performance?” ALTs are carefully controlled, whereas the field environment is highly variable. For example, products in the field have different average use rates across the product population. With good characterization of field use conditions, it may be possible to use ALT results to predict the failure time distribution in the field. When such information is not available but both life test data and field data (from, e.g., warranty returns) are available, it may be possible to find a model to relate the two data sets. Under a reasonable set of practical assumptions, this model then can be used to predict the failure time distribution for a future component or product o...
TL;DR: Methods for finding good ADDT plans for an important class of destructive degradation models are described and a more robust and useful compromise plan is proposed.
Abstract: Accelerated destructive degradation tests (ADDTs) provide reliability information quickly. An ADDT plan specifies factor-level combinations of an accelerating variable (e.g., temperature) and evaluation time and the allocations of test units to these combinations. This article describes methods for finding good ADDT plans for an important class of destructive degradation models. First, a collection of optimum plans is derived. These plans minimize the large sample approximate variance of the maximum likelihood (ML) estimator of a specified quantile of the failure-time distribution. The general equivalence theorem is used to verify the optimality of these plans. Because an optimum plan is not robust to the model specification and the planning information used in deriving the plan, a more robust and useful compromise plan is proposed. Sensitivity analyses show the effects that changes in sample size, time duration of the experiment, levels of the accelerating variable, and misspecification of the planning i...
TL;DR: The results indicate that the novel FDR-adjusted approaches are better at identifying the faulty stage than the conventional type I error rate control approach, especially when multiple out-of-control stages are present.
Abstract: Most statistical process control research focuses on single-stage processes. This article considers the problem of multistage process monitoring and fault identification. This problem is formulated as a multiple hypotheses testing problem; however, as the number of stages increases, the detection power of multiple hypotheses testing methods that seek to control the type I error rate decreases dramatically. To maintain the detection power, we use a false discovery rate (FDR) control approach, which is widely used in microarray research. Two multistage process monitoring and fault identification schemes—an FDR-adjusted Shewhart chart and an FDR-adjusted cumulative sum (CUSUM) chart—are established. To apply the FDR approach, the distribution of the CUSUM statistics are obtained based on Markov chain theory and Brownian motion with drift models. The detection and fault identification power of the new schemes are evaluated by the Monte Carlo method. The results indicate that the novel FDR-adjusted approaches ...
TL;DR: A decision-theoretic framework for conducting sensitivity analysis that addresses the problem of traditional variance-based measures of input parameter importance that can be impractical for complex computer models is reviewed and efficient computational tools using Gaussian processes are provided.
Abstract: When using a computer model to inform a decision, it is important to investigate any uncertainty in the model and determine how that uncertainty may impact on the decision. In probabilistic sensitivity analysis, model users can investigate how various uncertain model inputs contribute to the uncertainty in the model output. However, much of the literature focuses only on output uncertainty as measured by variance; the decision problem itself often is ignored, even though uncertainty as measured by variance may not equate to uncertainty about the optimum decision. Consequently, traditional variance-based measures of input parameter importance may not correctly describe the importance of each input. We review a decision-theoretic framework for conducting sensitivity analysis that addresses this problem. Because computation of these decision-theoretic measures can be impractical for complex computer models, we provide efficient computational tools using Gaussian processes. We give an illustration in the fiel...
TL;DR: A statistical methodology for simultaneously determining tuning and calibration parameters in settings where data are available from a computer code and the associated physical experiment and based on a hierarchical Bayesian model is introduced.
Abstract: Tuning and calibration are processes for improving the representativeness of a computer simulation code to a physical phenomenon. This article introduces a statistical methodology for simultaneously determining tuning and calibration parameters in settings where data are available from a computer code and the associated physical experiment. Tuning parameters are set by minimizing a discrepancy measure while the distribution of the calibration parameters are determined based on a hierarchical Bayesian model. The proposed Bayesian model views the output as a realization of a Gaussian stochastic process with hyper-priors. Draws from the resulting posterior distribution are obtained by the Markov chain Monte Carlo simulation. Our methodology is compared with an alternative approach in examples and is illustrated in a biomechanical engineering application. Supplemental materials, including the software and a user manual, are available online and can be requested from the first author.
TL;DR: Models that may be used to assess the dependence on age or usage in heterogeneous populations of products are presented, and how to estimate model parameters based on different types of field data is shown.
Abstract: Failures or other adverse events in systems or products may depend on the age and usage history of the unit. Motivated by motor vehicle reliability and warranty data issues, we present models that may be used to assess the dependence on age or usage in heterogeneous populations of products, and show how to estimate model parameters based on different types of field data. The setting in which the events in question are warranty claims is complicated because of the sparseness and incompleteness of the data, and we examine it in some detail. We consider some North American automobile warranty data and use these data to illustrate the methodology.
TL;DR: This work demonstrates the use of the outer product emulator for efficient calculation, with an emphasis on predictive diagnostics for model choice and model validation, and uses the emulator to “verify” the underlying computer code and to quantify the qualitative physical understanding.
Abstract: The thermosphere–ionosphere electrodynamics general circulation model (TIE-GCM) of the upper atmosphere has a number of features that are a challenge to standard approaches to emulation, including a long run time, multivariate output, periodicity, and strong constraints on the interrelationship between inputs and outputs. These kinds of features are not unusual in models of complex systems. We show how they can be handled in an emulator and demonstrate the use of the outer product emulator for efficient calculation, with an emphasis on predictive diagnostics for model choice and model validation. We use our emulator to “verify” the underlying computer code and to quantify our qualitative physical understanding.
TL;DR: In this article, the authors consider the problem of designing for complex high-dimensional computer models that can be evaluated at different levels of accuracy and propose an approach that combines the information from both the approximate model and the accurate model into a single multiscale emulator.
Abstract: We consider the problem of designing for complex high-dimensional computer models that can be evaluated at different levels of accuracy. Ordinarily, this requires performing many expensive evaluations of the most accurate version of the computer model to obtain a reasonable coverage of the design space. In some cases, it is possible to supplement the information from the accurate model evaluations with a large number of evaluations of a cheap, approximate version of the computer model to enable a more informed design choice. We describe an approach that combines the information from both the approximate model and the accurate model into a single multiscale emulator for the computer model. We then propose a design strategy for selecting a small number of expensive evaluations of the accurate computer model based on our multiscale emulator and a decomposition of the input parameter space. We illustrate our methodology with an example concerning a computer simulation of a hydrocarbon reservoir.
TL;DR: An alternative jump detection procedure is proposed, based on estimation of the (one-sided) first-order derivatives of the true regression curve, which works well in applications and is extended for detecting roofs/valleys of the regression curve.
Abstract: Curve estimation from observed noisy data has broad applications. In certain applications, the underlying regression curve may have singularities, including jumps and roofs/valleys (i.e., jumps in the first-order derivative of the regression curve), at some unknown positions, representing structural changes of the related process. Detection of such singularities is important for understanding the structural changes. In the literature, a number of jump detection procedures have been proposed, most of which are based on estimation of the (one-sided) first-order derivatives of the true regression curve. In this paper, motivated by certain related research in image processing, we propose an alternative jump detection procedure. Besides the first-order derivatives, we suggest using helpful information about jumps in the second-order derivatives as well. Theoretical justifications and numerical studies show that this jump detector works well in applications. This procedure is then extended for detecting roofs/v...
TL;DR: In this article, a construction procedure is proposed that allows a design to be constructed only from its minimum aberration projection in the sequential buildup process and a fast isomorphism checking procedure is developed by matching the factors using their delete-one-factor projections.
Abstract: Fractional factorial (FF) designs are widely used in practice and typically are chosen according to the minimum aberration criterion. A sequential algorithm is developed for constructing efficient FF designs. A construction procedure is proposed that allows a design to be constructed only from its minimum aberration projection in the sequential buildup process. To efficiently identify nonisomorphic designs, designs are categorized according to moment projection pattern. A fast isomorphism checking procedure is developed by matching the factors using their delete-one-factor projections. This algorithm is used to completely enumerate all 128-run designs of resolution 4, all 256-run designs of resolution 4 up to 17 factors, all 512-run designs of resolution 5, all 1024-run designs of resolution 6, and all 2048- and 4,096-run designs of resolution 7. A method is proposed for constructing minimum aberration (MA) designs using only a partial catalog of some good designs. Three approaches to constructing good de...
Journal Article•
TL;DR: This work describes an approach that combines the information from both the approximate model and the accurate model into a single multiscale emulator for the computer model and proposes a design strategy for selecting a small number of expensive evaluations of the accurate computer model based on the authors' multiscales emulator and a decomposition of the input parameter space.
Abstract: Designing for complex high-dimensional computer models ordinarily requires performing expensive evaluations of the most accurate version of the computer model. It is sometimes possible to supplement this information with evaluations of cheaper approxima..
TL;DR: In this article, the authors developed optimal Latin hypercube designs and kriging methods that can accommodate branching and nested factors, which resulted in a remarkable improvement in the machining process.
Abstract: In many experiments, some of the factors exist only within the level of another factor. Such factors are often called nested factors. A factor within which other factors are nested is called a branching factor. Suppose, for example, that we want to experiment with two processing methods. The factors involved in these two methods can be different. Thus in this experiment, the processing method is a branching factor, and the other factors are nested within the branching factor. The design and analysis of experiments with branching and nested factors are challenging and have not received much attention in the literature. Motivated by a computer experiment in a machining process, we have developed optimal Latin hypercube designs and kriging methods that can accommodate branching and nested factors. Through the application of the proposed methods, optimal machining conditions and tool edge geometry are attained, which resulted in a remarkable improvement in the machining process.
TL;DR: The methodology has three components: dimension reduction through a smooth factor model, time series modeling and forecasting of the factor scores, and dynamic updating using penalized least squares.
Abstract: We consider modeling a time series of smooth curves and develop methods for forecasting such curves and dynamically updating the forecasts. The research problem is motivated by efficient operations management of telephone customer service centers, where forecasts of daily call arrival rate profiles are needed for service agent staffing and scheduling purposes. Our methodology has three components: dimension reduction through a smooth factor model, time series modeling and forecasting of the factor scores, and dynamic updating using penalized least squares. The proposed methods are illustrated via the motivating application and two simulation studies.
TL;DR: Procedures to compute the exact minimum and average coverage probabilities of the tolerance intervals for Poisson and binomial variables are proposed and illustrated with examples and real data applications.
Abstract: The construction of tolerance intervals (TIs) for discrete variables, such as binomial and Poisson variables, has been critical in industrial applications in various sectors, including manufacturing and pharmaceuticals. Inaccurate estimation of coverage probabilities leads to improper construction of tolerance intervals and may lead to serious financial losses for the manufacturers. This article proposes procedures to compute the exact minimum and average coverage probabilities of the tolerance intervals for Poisson and binomial variables. These procedures are illustrated with examples and real data applications. Based on these procedures, improved tolerance intervals are proposed that can ensure that the true minimum or average coverage probabilities are very close to the nominal levels.
TL;DR: Novel arrangements for strip-block designs that reduce the experimental effort are presented and catalogs of post-fractionated strip- block designs with 16 and 32 trials are provided to aid the selection of appropriate plans.
Abstract: Novel arrangements for strip-block designs that reduce the experimental effort are presented. Theoretical properties of strip-block designs using post-fractionation are provided, and appropriate data analysis is explained. An experiment on an industrial process is used as an illustration. As a tool to aid the selection of appropriate plans, catalogs of post-fractionated strip-block designs with 16 and 32 trials are provided.
TL;DR: It is argued that effect hierarchy principle should not be altered for achieving the robustness objective of the experiment, and a Bayesian approach to develop single arrays which incorporate the importance of control-by-noise interactions without altering the effect hierarchy is proposed.
Abstract: It is critical to estimate control-by-noise interactions in robust parameter design. This can be achieved by using a cross array, which is a cross product of a design for control factors and another design for noise factors. However, the total run size of such arrays can be prohibitively large. To reduce the run size, single arrays are proposed in the literature, where a modified effect hierarchy principle is used for the optimal selection of the arrays. In this article, we argue that effect hierarchy principle should not be altered for achieving the robustness objective of the experiment. We propose a Bayesian approach to develop single arrays which incorporate the importance of control-by-noise interactions without altering the effect hierarchy. The approach is very general and places no restrictions on the number of runs or levels or type of factors or type of designs. A modified exchange algorithm is proposed for finding the optimal single arrays. MATLAB code for implementing the algorithm is availabl...
Journal Article•
TL;DR: This work applies a branch-and-bound algorithm for selection of the best hierarchically well-formulated subset in second-order polynomial regression to a well-known data set from the regression literature, and reveals that the hierarchical constraints yield only a small penalty in explained variation.
Abstract: Variable selection in multiple regression requires identifying the best subset from among a set of candidate predictors. In the case of polynomial regression, the variable selection process can be further complicated by the need to obtain subsets that are hierarchically well formulated. We present a branch-and-bound algorithm for selection of the best hierarchically well-formulated subset in second-order polynomial regression. We apply the new algorithm to a well-known data set from the regression literature and compare the results with those obtained from a branch-and-bound algorithm that does not impose the hierarchical constraints. This comparison reveals that the hierarchical constraints yield only a small penalty in explained variation. We offer Fortran and MATLAB implementations of the branch-and-bound algorithms as supplemental materials associated with this work.