Many scientific phenomena are now investigated by complex computer models or codes A computer experiment is a number of runs of the code with various inputs A feature of many computer experiments is that the output is deterministic--rerunning the code with the same inputs gives identical observations Often, the codes are computationally expensive to run, and a common objective of an experiment is to fit a cheaper predictor of the output to the data Our approach is to model the deterministic output as the realization of a stochastic process, thereby providing a statistical basis for designing experiments (choosing the inputs) for efficient prediction With this model, estimates of uncertainty of predictions are also available Recent work in this area is reviewed, a number of applications are discussed, and we demonstrate our methodology with an example

/pdf/the-design-and-analysis-of-computer-experiments-1kfo89lyf2.pdf

The design and analysis of computer experiments

2 1 The innovative transport systems and the CityMobil project 10 1.1 The research questions 10 2 The state of art in the field of automatic transport systems 12 2.1 Case studies and demand studies for innovative transport systems 12 3 The design and implementation of surveys 14 3.1 Definition of experimental design 14 3.2 Questionnaire design and delivery 16 3.3 First analyses on the collected sample 18 4 Calibration of Logit Multionomial demand models 21 4.1 Methodology 21 4.2 Calibration of the “full” model. 22 4.3 Calibration of the “final” model 24 4.4 The demand analysis through the final Multinomial Logit model 25 5 The analysis of interaction between the demand and socioeconomic attributes 31 5.1 Methodology 31 5.2 Application of Mixed Logit models to the demand 31 5.3 Analysis of the interactions between demand and socioeconomic attributes through Mixed Logit models 32 5.4 Mixed Logit model and interaction between age and the demand for the CTS 38 5.5 Demand analysis with Mixed Logit model 39 6 Final analyses and conclusions 45 6.1 Comparison between the results of the analyses 45 6.2 Conclusions 48 6.3 Answers to the research questions and future developments 52

The European commission

In this book, Andrew Harvey sets out to provide a unified and comprehensive theory of structural time series models. Unlike the traditional ARIMA models, structural time series models consist explicitly of unobserved components, such as trends and seasonals, which have a direct interpretation. As a result the model selection methodology associated with structural models is much closer to econometric methodology. The link with econometrics is made even closer by the natural way in which the models can be extended to include explanatory variables and to cope with multivariate time series. From the technical point of view, state space models and the Kalman filter play a key role in the statistical treatment of structural time series models. The book includes a detailed treatment of the Kalman filter. This technique was originally developed in control engineering, but is becoming increasingly important in fields such as economics and operations research. This book is concerned primarily with modelling economic and social time series, and with addressing the special problems which the treatment of such series poses. The properties of the models and the methodological techniques used to select them are illustrated with various applications. These range from the modellling of trends and cycles in US macroeconomic time series to to an evaluation of the effects of seat belt legislation in the UK.

Forecasting, Structural Time Series Models and the Kalman Filter

Global sensitivity indices for nonlinear mathematical models and their Monte Carlo estimates

We consider prediction and uncertainty analysis for systems which are approximated using complex mathematical models. Such models, implemented as computer codes, are often generic in the sense that by a suitable choice of some of the model's input parameters the code can be used to predict the behaviour of the system in a variety of specific applications. However, in any specific application the values of necessary parameters may be unknown. In this case, physical observations of the system in the specific context are used to learn about the unknown parameters. The process of fitting the model to the observed data by adjusting the parameters is known as calibration. Calibration is typically effected by ad hoc fitting, and after calibration the model is used, with the fitted input values, to predict the future behaviour of the system. We present a Bayesian calibration technique which improves on this traditional approach in two respects. First, the predictions allow for all sources of uncertainty, including the remaining uncertainty over the fitted parameters. Second, they attempt to correct for any inadequacy of the model which is revealed by a discrepancy between the observed data and the model predictions from even the best-fitting parameter values. The method is illustrated by using data from a nuclear radiation release at Tomsk, and from a more complex simulated nuclear accident exercise.

https://www.jstor.org/stable/pdfplus/2680584.pdf

Bayesian Calibration of computer models

Preface. 1. Introduction to Sensitivity Analysi. 1.1 Models and Sensitivity Analysis. 1.1.1 Definition. 1.1.2 Models. 1.1.3 Models and Uncertainty. 1.1.4 How to Set Up Uncertainty and Sensitivity Analyses. 1.1.5 Implications for Model Quality. 1.2 Methods and Settings for Sensitivity Analysis - An Introduction. 1.2.1 Local versus Global. 1.2.2 A Test Model. 1.2.3 Scatterplots versus Derivatives. 1.2.4 Sigma-normalized Derivatives. 1.2.5 Monte Carlo and Linear Regression. 1.2.6 Conditional Variances - First Path. 1.2.7 Conditional Variances - Second Path. 1.2.8 Application to Model (1.3). 1.2.9 A First Setting: 'Factor Prioritization' 1.2.10 Nonadditive Models. 1.2.11 Higher-order Sensitivity Indices. 1.2.12 Total Effects. 1.2.13 A Second Setting: 'Factor Fixing'. 1.2.14 Rationale for Sensitivity Analysis. 1.2.15 Treating Sets. 1.2.16 Further Methods. 1.2.17 Elementary Effect Test. 1.2.18 Monte Carlo Filtering. 1.3 Nonindependent Input Factors. 1.4 Possible Pitfalls for a Sensitivity Analysis. 1.5 Concluding Remarks. 1.6 Exercises. 1.7 Answers. 1.8 Additional Exercises. 1.9 Solutions to Additional Exercises. 2. Experimental Designs. 2.1 Introduction. 2.2 Dependency on a Single Parameter. 2.3 Sensitivity Analysis of a Single Parameter. 2.3.1 Random Values. 2.3.2 Stratified Sampling. 2.3.3 Mean and Variance Estimates for Stratified Sampling. 2.4 Sensitivity Analysis of Multiple Parameters. 2.4.1 Linear Models. 2.4.2 One-at-a-time (OAT) Sampling. 2.4.3 Limits on the Number of Influential Parameters. 2.4.4 Fractional Factorial Sampling. 2.4.5 Latin Hypercube Sampling. 2.4.6 Multivariate Stratified Sampling. 2.4.7 Quasi-random Sampling with Low-discrepancy Sequences. 2.5 Group Sampling. 2.6 Exercises. 2.7 Exercise Solutions. 3. Elementary Effects Method. 3.1 Introduction. 3.2 The Elementary Effects Method. 3.3 The Sampling Strategy and its Optimization. 3.4 The Computation of the Sensitivity Measures. 3.5 Working with Groups. 3.6 The EE Method Step by Step. 3.7 Conclusions. 3.8 Exercises. 3.9 Solutions. 4. Variance-based Methods. 4.1 Different Tests for Different Settings. 4.2 Why Variance? 4.3 Variance-based Methods. A Brief History. 4.4 Interaction Effects. 4.5 Total Effects. 4.6 How to Compute the Sensitivity Indices. 4.7 FAST and Random Balance Designs. 4.8 Putting the Method to Work: the Infection Dynamics Model. 4.9 Caveats. 4.10 Exercises. 5. Factor Mapping and Metamodelling. 5.1 Introduction. 5.2 Monte Carlo Filtering (MCF). 5.2.1 Implementation of Monte Carlo Filtering. 5.2.2 Pros and Cons. 5.2.3 Exercises. 5.2.4 Solutions. 5.2.5 Examples. 5.3 Metamodelling and the High-Dimensional Model Representation. 5.3.1 Estimating HDMRs and Metamodels. 5.3.2 A Simple Example. 5.3.3 Another Simple Example. 5.3.4 Exercises. 5.3.5 Solutions to Exercises. 5.4 Conclusions. 6. Sensitivity Analysis: from Theory to Practice. 6.1 Example 1: a Composite Indicator. 6.1.1 Setting the Problem. 6.1.2 A Composite Indicator Measuring Countries' Performance in Environmental Sustainability. 6.1.3 Selecting the Sensitivity Analysis Method. 6.1.4 The Sensitivity Analysis Experiment and its Results. 6.1.5 Conclusions. 6.2 Example 2: Importance of Jumps in Pricing Options. 6.2.1 Setting the Problem. 6.2.2 The Heston Stochastic Volatility Model with Jumps. 6.2.3 Selecting a Suitable Sensitivity Analysis Method. 6.2.4 The Sensitivity Analysis Experiment. 6.2.5 Conclusions. 6.3 Example 3: a Chemical Reactor. 6.3.1 Setting the Problem. 6.3.2 Thermal Runaway Analysis of a Batch Reactor. 6.3.3 Selecting the Sensitivity Analysis Method. 6.3.4 The Sensitivity Analysis Experiment and its Results. 6.3.5 Conclusions. 6.4 Example 4: a Mixed Uncertainty-Sensitivity Plot. 6.4.1 In Brief. 6.5 When to use What? Afterword. Bibliography. Index.

Global Sensitivity Analysis: The Primer

This Handbook aims to provide a guide for constructing and using composite indicators for policy makers, academics, the media and other interested parties. While there are several types of composite indicators, this Handbook is concerned with those which compare and rank country performance in areas such as industrial competitiveness, sustainable development, globalisation and innovation. The Handbook aims to contribute to a better understanding of the complexity of composite indicators and to an improvement of the techniques currently used to build them. In particular, it contains a set of technical guidelines that can help constructors of composite indicators to improve the quality of their outputs. It has been prepared jointly by the OECD (the Statistics Directorate and the Directorate for Science, Technology and Industry) and the Applied Statistics and Econometrics Unit of the Joint Research Centre of the European Commission in Ispra, Italy. Primary authors from the JRC are Michela Nardo, Michaela Saisana, Andrea Saltelli and Stefano Tarantola. Primary authors from the OECD are Anders Hoffmann and Enrico Giovannini. Editorial assistance was provided by Candice Stevens, Gunseli Baygan and Karsten Olsen. The research is partly funded by the European Commission, Research Directorate, under the project KEI (Knowledge Economy Indicators), Contract FP6 No. 502529. In the OECD context, the work has benefitted from a grant from the Danish government. The views expressed are those of the authors and should not be regarded as stating an official position of either the European Commission or the OECD.

/pdf/handbook-on-constructing-composite-indicators-methodology-1a4fw776o7.pdf

Handbook on Constructing Composite Indicators: Methodology and User Guide

PREFACE. 1. A WORKED EXAMPLE. 1.1 A simple model. 1.2 Modulus version of the simple model. 1.3 Six--factor version of the simple model. 1.4 The simple model 'by groups'. 1.5 The (less) simple correlated--input model. 1.6 Conclusions. 2. GLOBAL SENSITIVITY ANALYSIS FOR IMPORTANCE ASSESSMENT. 2.1 Examples at a glance. 2.2 What is sensitivity analysis? 2.3 Properties of an ideal sensitivity analysis method. 2.4 Defensible settings for sensitivity analysis. 2.5 Caveats. 3. TEST CASES. 3.1 The jumping man. Applying variance--based methods. 3.2 Handling the risk of a financial portfolio: the problem of hedging. Applying Monte Carlo filtering and variance--based methods. 3.3 A model of fish population dynamics. Applying the method of Morris. 3.4 The Level E model. Radionuclide migration in the geosphere. Applying variance--based methods and Monte Carlo filtering. 3.5 Two spheres. Applying variance based methods in estimation/calibration problems. 3.6 A chemical experiment. Applying variance based methods in estimation/calibration problems. 3.7 An analytical example. Applying the method of Morris. 4. THE SCREENING EXERCISE. 4.1 Introduction. 4.2 The method of Morris. 4.3 Implementing the method. 4.4 Putting the method to work: an analytical example. 4.5 Putting the method to work: sensitivity analysis of a fish population model. 4.6 Conclusions. 5. METHODS BASED ON DECOMPOSING THE VARIANCE OF THE OUTPUT. 5.1 The settings. 5.2 Factors Prioritisation Setting. 5.3 First--order effects and interactions. 5.4 Application of Si to Setting 'Factors Prioritisation'. 5.5 More on variance decompositions. 5.6 Factors Fixing (FF) Setting. 5.7 Variance Cutting (VC) Setting. 5.8 Properties of the variance based methods. 5.9 How to compute the sensitivity indices: the case of orthogonal input. 5.9.1 A digression on the Fourier Amplitude Sensitivity Test (FAST). 5.10 How to compute the sensitivity indices: the case of non--orthogonal input. 5.11 Putting the method to work: the Level E model. 5.11.1 Case of orthogonal input factors. 5.11.2 Case of correlated input factors. 5.12 Putting the method to work: the bungee jumping model. 5.13 Caveats. 6. SENSITIVITY ANALYSIS IN DIAGNOSTIC MODELLING: MONTE CARLO FILTERING AND REGIONALISED SENSITIVITY ANALYSIS, BAYESIAN UNCERTAINTY ESTIMATION AND GLOBAL SENSITIVITY ANALYSIS. 6.1 Model calibration and Factors Mapping Setting. 6.2 Monte Carlo filtering and regionalised sensitivity analysis. 6.2.1 Caveats. 6.3 Putting MC filtering and RSA to work: the problem of hedging a financial portfolio. 6.4 Putting MC filtering and RSA to work: the Level E test case. 6.5 Bayesian uncertainty estimation and global sensitivity analysis. 6.5.1 Bayesian uncertainty estimation. 6.5.2 The GLUE case. 6.5.3 Using global sensitivity analysis in the Bayesian uncertainty estimation. 6.5.4 Implementation of the method. 6.6 Putting Bayesian analysis and global SA to work: two spheres. 6.7 Putting Bayesian analysis and global SA to work: a chemical experiment. 6.7.1 Bayesian uncertainty analysis (GLUE case). 6.7.2 Global sensitivity analysis. 6.7.3 Correlation analysis. 6.7.4 Further analysis by varying temperature in the data set: fewer interactions in the model. 6.8 Caveats. 7. HOW TO USE SIMLAB. 7.1 Introduction. 7.2 How to obtain and install SIMLAB. 7.3 SIMLAB main panel. 7.4 Sample generation. 7.4.1 FAST. 7.4.2 Fixed sampling. 7.4.3 Latin hypercube sampling (LHS). 7.4.4 The method of Morris. 7.4.5 Quasi--Random LpTau. 7.4.6 Random. 7.4.7 Replicated Latin Hypercube (r--LHS). 7.4.8 The method of Sobol'. 7.4.9 How to induce dependencies in the input factors. 7.5 How to execute models. 7.6 Sensitivity analysis. 8. FAMOUS QUOTES: SENSITIVITY ANALYSIS IN THE SCIENTIFIC DISCOURSE. REFERENCES. INDEX.

Sensitivity Analysis in Practice: A Guide to Assessing Scientific Models

http://www.andreasaltelli.eu/file/repository/PUBLISHED_PAPER.pdf

Variance based sensitivity analysis of model output. Design and estimator for the total sensitivity index

http://www.andreasaltelli.eu/file/repository/Making_best_use.pdf

Andrea Saltelli

Papers

Global Sensitivity Analysis: The Primer

Handbook on Constructing Composite Indicators: Methodology and User Guide

Sensitivity Analysis in Practice: A Guide to Assessing Scientific Models

Variance based sensitivity analysis of model output. Design and estimator for the total sensitivity index

Making best use of model evaluations to compute sensitivity indices