scispace - formally typeset
Search or ask a question
Journal ArticleDOI

Probabilistic sensitivity analysis of complex models: a Bayesian approach

TL;DR: In this article, the authors present a Bayesian framework which unifies the various tools of probabilistic sensitivity analysis, which allows effective sensitivity analysis to be achieved by using far smaller numbers of model runs than standard Monte Carlo methods.
Abstract: Summary. In many areas of science and technology, mathematical models are built to simulate complex real world phenomena. Such models are typically implemented in large computer programs and are also very complex, such that the way that the model responds to changes in its inputs is not transparent. Sensitivity analysis is concerned with understanding how changes in the model inputs influence the outputs. This may be motivated simply by a wish to understand the implications of a complex model but often arises because there is uncertainty about the true values of the inputs that should be used for a particular application. A broad range of measures have been advocated in the literature to quantify and describe the sensitivity of a model's output to variation in its inputs. In practice the most commonly used measures are those that are based on formulating uncertainty in the model inputs by a joint probability distribution and then analysing the induced uncertainty in outputs, an approach which is known as probabilistic sensitivity analysis. We present a Bayesian framework which unifies the various tools of prob- abilistic sensitivity analysis. The Bayesian approach is computationally highly efficient. It allows effective sensitivity analysis to be achieved by using far smaller numbers of model runs than standard Monte Carlo methods. Furthermore, all measures of interest may be computed from a single set of runs.

Summary (4 min read)

1. Introduction

  • The authors suppose that η.·/ is a complex model, such that the way that the model responds to changes in its inputs is not transparent.
  • Sensitivity analysis is concerned with understanding how changes in the inputs x influence the output y.
  • This may be motivated simply by a wish to understand the implications of a complex model but often arises because there is uncertainty about the true values of the inputs that should be used for a particular application.
  • Large process models in engineering, environmental science, chemistry, etc. are often implemented in complex computer codes that require many minutes, hours or even days for a single run.

2.1. Main effects and interactions

  • Note that the definitions of these terms depend on the distribution G of the uncertain inputs.
  • The representation reflects the structure of the model itself, comprising a linear effect of x1 with no x2-effect and no interaction.

2.2. Variance-based methods

  • This approach is reviewed by Saltelli, Chan and Scott (2000).
  • The first is Vi =var{E.Y |Xi/}: Thus, if the authors were to learn xi, then the uncertainty about Y would become var.
  • The second measure, first proposed by Homma and Saltelli (1996), is VTi =var.
  • If it were possible to observe one of the xis, to learn its true value exactly, and the cost of that observation would be the same for each i, then the authors should choose that with the largest Si. Nevertheless, the analysis does suggest where there is the greatest potential for reducing uncertainty through new research.
  • It does not follow that the two inputs with the largest main effect variances will be the best two inputs to observe.

2.3. Variance decomposition

  • When G is such that the elements of X are mutually independent, the authors have already remarked that the definitions of main effects and interactions will directly reflect the model structure.
  • One can also decompose the variance of Y into terms relating to the main effects and various interactions between the input variables.the authors.
  • It is clear that when equation (5) holds the authors can identify V−i = var{E.Y |X−i/} with the sum of all the Wp-terms not including the subscript i.
  • Therefore the total effect index (3) is the proportion of var.Y/ that is accounted for by all the terms in equation (5) with a subscript i, and so STi Si.
  • Without orthogonality, the authors can still define the sum of squares attributable to any set of variables, but sums of squares for different sets of regressors no longer partition the total sum of squares.

2.4. Regression components

  • Thus, if the authors wish to predict Y without gaining any further information about x, then the best prediction (in terms of minimizing the expected squared error) is E.Y/.
  • Then var.Y/ is the expected squared error of this prediction.
  • It should be noted that regression coefficients, correlation coefficients and related sums of squares have been widely used in sensitivity analysis.
  • In practice, it is easy to see that the regression coefficients of Helton and Davis (2000) are estimates of their optimal coefficients γ in the corresponding regression fit.
  • The interpretation is different, the authors allow for non-linear fits and they add the very important step of interpreting the difference between the regression variance component and the corresponding main effect variance as a lack-of-fit variance component.

2.5. Discussion

  • The preceding subsections have presented a very broad perspective on probabilistic sensitivity analysis.
  • The authors formulation unifies a variety of current approaches and offers new measures, to provide a deeper understanding of a model and its dependence on the uncertain model inputs.
  • The authors define new population-based regression measures that provide a link between the sample measures and variance-based sensitivity analysis.
  • The authors proposal to use the difference between Vi and Vxi to measure non-linearity in zi.xi/ is novel and, the authors believe, powerful.
  • The authors end this section by briefly addressing some other issues.

2.5.1. Local sensitivity

  • Local sensitivity analysis is based on partial derivatives of the function η.·/, evaluated at the base-line inputs x0.
  • Baker (2001) suggested approximating η.·/ by a first-order Taylor series and derived the D2i as measures of sensitivity.

2.5.2. Value of information

  • The authors use of squared prediction error as a criterion can be justified formally in decision theoretic terms by using the squared error loss.
  • It may then be shown that Vp is the expected value of gaining perfect information about xp.
  • More generally, wherever the computer model is to be used for decision-making, the authors could again measure sensitivity by the expected value of information, but now defined with respect to the relevant utility or loss function and the available decisions.

2.5.3. Computation

  • For models of sufficient complexity for it not to be obvious how the output would respond to the model inputs, the authors cannot hope for such tractability and must instead seek to obtain the desired measures computationally.
  • If η.·/ is sufficiently cheap to evaluate for many different inputs, simple Monte Carlo methods can be used to estimate var.Y/ or the component of variance Vg.x/ for any regression fit with negligible error.
  • The method of Sobol’ (1993) and the Fourier amplitude sensitivity test, devised by Cukier et al. (1973) and extended by Saltelli et al. (1999), are techniques that have been developed specifically to compute some of these measures.
  • Nevertheless, sensitivity analysis by these techniques demands many thousands of function evaluations.
  • For an expensive function, where the evaluation of η.x/ at a single x might take minutes or even hours, such methods are impractical.

3. Bayesian sensitivity analysis

  • The authors shall develop Bayesian inference tools for estimating all the quantities of interest in sensitivity analysis, for the case of expensive functions.
  • In addition to making it feasible to carry out sensitivity analysis with a much smaller number of model runs, a key benefit of their approach is that it can estimate all the many sensitivity measures that were discussed in Section 2, from a single set of runs.
  • The essence of the Bayesian approach is that the model η.·/ is treated as an unknown function.
  • In an absolute sense, of course, η.·/ is certainly not unknown, since it implements a model that has been specified in precise mathematical form by someone (or some group of people).
  • The authors therefore formulate a prior distribution for the function η.·/.

3.1. Inference about functions using Gaussian processes

  • The authors first develop the prior model for η.·/ in the form of a Gaussian process prior distribution and derive the posterior distribution.
  • The choice of h.·/ is arbitrary, though it should be chosen to incorporate any beliefs that the authors might have about the form of η.·/.
  • This implies an infinite prior variance of η.x/, whereas in practice the authors expect there to be cases when the model developer can provide some proper prior knowledge about the function η.·/.
  • Full details of the prior to posterior analysis can be found in O’Hagan (1994).
  • Monte Carlo methods applied to very cheap functions typically employ many thousands of model runs, so that the estimation error is very small.

3.2. Inference for main effects and interactions

  • First consider inference about E.Y |xp/= ∫ X−p η.x/dG−p|p.x−p|xp/, using obvious notation for the space of possible values for x−p and for its conditional distribution given xp.
  • The authors can derive the posterior mean as follows.
  • From the plot, it is tempting to think of the inputs showing the greatest variation as the most important, but var[EÅ{zi.

3.3. Inference for variances

  • The authors now consider posterior inference for Vi and VTi.
  • Haylock and O’Hagan (1996) derived the posterior mean and variance of var.
  • As before, all the required integrals can be done numerically if necessary but are available analytically for certain common modelling choices.

3.4. Inference for regression fits

  • All the resulting integrals may be computed numerically and may be obtained analytically for common modelling choices.
  • Relevant theory is given for one dimension in O’Hagan (1992) and is easily generalized to higher dimensions.
  • Inference about D2i can then also be derived.

4. Examples

  • The authors present two illustrative examples, which are typical of a variety of models that they have considered.
  • To apply the techniques of Section 3 in practice, it is necessary to identify the functions h.·/ and c.·, ·/ that represent prior beliefs about the function η.·/, and the distribution G.·/ that defines the uncertainty about the model inputs.
  • This implies a belief that the output is an analytic differentiable function of its inputs.

4.1. Synthetic example

  • The authors illustrate their methodology first with a synthetic example.
  • If the new design decreases the value of the integral, the candidate design point is exchanged for the current point.
  • The simulation method involves generating many additional runs of the code η.·/ from its posterior distribution and re-estimating Si each time.
  • The remaining variance after the main effects is estimated as 29% of the total variance (true value 28%).
  • Plotting the posterior expectation (with respect to the unknown function η.·/) of E.Y |xi/ against xi for each variable also allows us to identify the three groups of variables.

4.2. Oil-field simulator

  • This model was used in Craig et al. (1997, 2001) to demonstrate their methodology for calibration and forecasting.
  • The authors choose notional distributions for these inputs; they first take log-transformations of the permeability inputs as in Craig et al. (2001).
  • The authors then suppose that each input has a normal distribution, with the ranges of each input representing six standard deviations.
  • The authors have 101 runs of the code, with the design points chosen to form a Latin hypercube.
  • Though the authors do not show the results here, they have also performed the same sensitivity analysis for different wells in the reservoir, and at different time points.

5. Conclusions

  • The authors method facilitates a deep and thorough analysis of the sensitivity of a model output to variation in its inputs—through decomposition of the output variance into components representing main effects and interactions, through further decomposition of individual terms into components for linear or other regression-based fits, and for non-linearity, and through graphical presentation of main effects and first-order interactions.
  • This is particularly important in the case of expensive models, since Monte Carlo methods become infeasible if each model run takes an appreciable amount of computer time.
  • The Bayesian approach also allows the complete range of sensitivity measures to be computed from a single set of model runs.
  • The authors examples involve 15 and 40 uncertain model inputs and are therefore of realistic, albeit moderate, dimensionality.

Did you find this useful? Give us your feedback

Content maybe subject to copyright    Report

2004 Royal Statistical Society 1369–7412/04/66751
J. R. Statist. Soc. B (2004)
66
, Part 3, pp. 751–769
Probabilistic sensitivity analysis of complex models:
a Bayesian approach
Jeremy E. Oakley and Anthony O’Hagan
University of Sheffield, UK
[Received May 2002. Revised December 2003]
Summary. In many areas of science and technology, mathematical models are built to simu-
late complex real world phenomena. Such models are typically implemented in large computer
programs and are also very complex, such that the way that the model responds to changes in
its inputs is not transparent. Sensitivity analysis is concerned with understanding how changes
in the model inputs influence the outputs.This may be motivated simply by a wish to understand
the implications of a complex model but often arises because there is uncertainty about the
true values of the inputs that should be used for a particular application. A broad range of mea-
sures have been advocated in the literature to quantify and describe the sensitivity of a model’s
output to variation in its inputs. In practice the most commonly used measures are those that
are based on formulating uncertainty in the model inputs by a joint probability distribution and
then analysing the induced uncertainty in outputs, an approach which is known as probabilistic
sensitivity analysis. We present a Bayesian framework which unifies the various tools of prob-
abilistic sensitivity analysis. The Bayesian approach is computationally highly efficient. It allows
effective sensitivity analysis to be achieved by using far smaller numbers of model runs than
standard Monte Carlo methods. Furthermore, all measures of interest may be computed from
a single set of runs.
Keywords: Bayesian inference; Computer model; Gaussian process; Sensitivity analysis;
Uncertainty analysis
1. Introduction
Consider a deterministic model that is represented by y =η.x/, where x is a vector of input
variables and y is the model output. We suppose that η.·/ is a complex model, such that the
way that the model responds to changes in its inputs is not transparent. Sensitivity analysis is
concerned with understanding how changes in the inputs x influence the output y. This may
be motivated simply by a wish to understand the implications of a complex model but often
arises because there is uncertainty about the true values of the inputs that should be used for a
particular application. We have a ‘base-line’ or central estimate x
0
for x but are then interested
in how the true output y =η.x/ might differ from the base-line output y
0
=η.x
0
/.
Local sensitivity analysis is based on derivatives of η.·/ evaluated at x =x
0
and indicates how
y will change if the base-line input values are perturbed slightly. This is clearly of limited value
in understanding the consequences of real uncertainty about x, which would in practice entail
more than infinitesimal changes in the inputs. Global sensitivity analysis considers these more
substantial changes in x. However, there is then the question of how far to perturb individual
inputs. Perturbing each input to the limits that might be considered plausible gives some kind of
limits of plausibility for y, but the resulting range is usually unrealistically wide if many inputs
Addressfor correspondence: Jeremy E. Oakley, Department of Probability and Statistics, University of Sheffield,
Sheffield, S3 7RH, UK.
E-mail: j.oakley@sheffield.ac.uk

752 J. E. Oakley and A. O’Hagan
are perturbed together, or unrealistically narrow if they are only perturbed individually. Such
difficulties are overcome by acknowledging the uncertainty in x and formally treating it as a
random variable with a specified distribution.
Thus, denoting the unknown true inputs by X, it follows that the corresponding output
Y =η.X/ is also unknown. We suppose that our uncertainty about the elements of X is des-
cribed by some probability distribution G. The approach to sensitivity analysis which
exploits this probabilistic setting is known as probabilistic sensitivity analysis. The most imme-
diate question in this case is to characterize the distribution of Y that is induced by giving
X the distribution G. Although this may be regarded as an aspect of probabilistic sensitivity
analysis, it is known by the separate name of uncertainty analysis. Sensitivity analysis proper
is generally seen as going beyond uncertainty analysis by exploring how individual inputs or
groups of inputs contribute to uncertainty in Y. In particular, an important problem is to iden-
tify which elements in X are the most influential, in some sense, in inducing the uncertainty
in Y.
We shall be particularly interested in the case where the model is so complex that simply
computing the output y for any given set of input values is a non-trivial task. For instance, large
process models in engineering, environmental science, chemistry, etc. are often implemented in
complex computer codes that require many minutes, hours or even days for a single run. We call
such models expensive, whereas a model that can be run many thousands of times in a reason-
able time is called cheap. The distinction is important when it comes to calculating any desired
measures of sensitivity, since ‘brute force’ computation that may be the simplest solution for
cheap models is usually impractical for expensive models.
Sensitivity and uncertainty analysis are important techniques for exploring complex models.
Saltelli, Tarantola and Campolongo (2000) and Kleijnen (1997) clearly show the key role of
these tools within the wider context of the building, validation and use of process models. We
present here
(a) a discussion of sensitivity analysis that unifies various other approaches that are consid-
ered in the literature and
(b) a Bayesian method building on the approach of O’Hagan et al. (1999) that is both robust
and highly efficient, allowing sensitivity analysis to be applied to expensive models.
2. Probabilistic sensitivity analysis
We write x ={x
1
,...,x
d
}, and we refer to x
i
as the ith element of x or the ith uncertain model
input. We shall denote the subvector .x
i
, x
j
/ by x
i,j
, and in general if p is a set of indices then
x
p
is the subvector of x whose elements have those indices. Finally, x
i
is the subvector of x
containing all elements except x
i
.
Note, however, that much of the analysis that is discussed here holds if the x
i
s are not simple
scalars, so the notation x ={x
1
,...,x
d
} could denote only a partial decomposition of x into d
subvectors.
2.1. Main effects and interactions
Some widely used methods of sensitivity analysis can be seen in terms of a decomposition of
the function η.·/ into main effects and interactions:
y =η.x/ =E.Y/ +
d
i=1
z
i
.x
i
/ +
i<j
z
i,j
.x
i,j
/ +
i<j<k
z
i,j,k
.x
i,j,k
/ + ... +z
1, 2 , :::,d
.x/, .1/

Probabilistic Sensitivity Analysis 753
where
z
i
.x
i
/ =E.Y |x
i
/ E.Y/,
z
i,j
.x
i,j
/ =E.Y |x
i,j
/ z
i
.x
i
/ z
j
.x
j
/ E.Y/,
z
i,j,k
.x
i,j,k
/ =E.Y |x
i,j,k
/ z
i,j
.x
i,j
/ z
i,k
.x
i,k
/ z
j,k
.x
j,k
/ z
i
.x
i
/ z
j
.x
j
/ z
k
.x
k
/ E.Y/,
and so on. We refer to z
i
.x
i
/ as the main effect of x
i
,toz
i,j
.x
i,j
/ as the first-order interaction
between x
i
and x
j
, and so on.
Note that the definitions of these terms depend on the distribution G of the uncertain inputs.
Consider, for instance, the very simple model η.x
1
, x
2
/ =x
1
.WehaveE.Y/ =E.X
1
/ and z
1
.x
1
/ =
x
1
E.X
1
/.IfG is such that X
1
and X
2
are independent then z
2
.x
2
/ =0 and z
12
.x
1
, x
2
/ =0. In
this case, the representation reflects the structure of the model itself, comprising a linear effect
of x
1
with no x
2
-effect and no interaction. If, however, X
1
and X
2
are not independent, we have
z
2
.x
2
/ =E.X
1
|x
2
/ E.X
1
/ =−z
12
.x
1
, x
2
/, which will not in general be 0.
Computing and plotting the main effects and first-order interactions is a powerful visual tool
for examining how the model output responds to each individual input, and how those inputs
interact in their influence on y.
2.2. Variance-based methods
Variance-based methods of probabilistic sensitivity analysis quantify the sensitivity of the
output Y to the model inputs in terms of a reduction in the variance of Y.
This approach is reviewed by Saltelli, Chan and Scott (2000). Two principal measures of the
sensitivity of Y to an individual x
i
are proposed. The first is
V
i
=var {E.Y |X
i
/}:
The motivation for this measure is that it is the expected amount by which the uncertainty in Y
will be reduced if we learn the true value of x
i
. Thus, if we were to learn x
i
, then the uncertainty
about Y would become var.Y|x
i
/, a difference of var.Y/ var.Y |x
i
/. Since we do not know the
true value of x
i
, the expected difference is var.Y/E {var.Y|x
i
/} =V
i
, by a well-known identity.
Although var.Y/ var.Y|x
i
/ can be negative for some x
i
, its expectation V
i
is always positive, so
this is the expected reduction in uncertainty due to observing x
i
. Note also that V
i
=var{z
i
.X
i
/}
and so is based on the main effect of x
i
.
The second measure, first proposed by Homma and Saltelli (1996), is
V
Ti
=var.Y/ var{E.Y |X
i
/},
which is the remaining uncertainty in Y that is unexplained after we have learnt everything
except x
i
.
Both measures are converted into scale invariant measures by dividing by var.Y/:
S
i
=V
i
=var.Y/, .2/
S
Ti
=V
Ti
=var.Y/ =1 S
i
:.3/
Thus, S
i
may be referred to as the main effect index of x
i
, and S
Ti
is known as the total effect
index of x
i
. The relative importance of each input in driving the uncertainty in Y is then gauged
by comparing their indices.

754 J. E. Oakley and A. O’Hagan
As well as indicating the relative importance of an individual x
i
in driving the uncertainty in
Y, equation (2) can be seen as indicating where to direct effort in future to reduce that uncer-
tainty. If it were possible to observe one of the x
i
s, to learn its true value exactly, and the cost of
that observation would be the same for each i, then we should choose that with the largest S
i
.
In practice, of course, it is rarely possible to learn the true value of any of the uncertain inputs
exactly; nor is the cost of gaining more information likely to be the same for each input. Never-
theless, the analysis does suggest where there is the greatest potential for reducing uncertainty
through new research. We do not believe that there is any comparable interpretation of S
Ti
in
terms of guiding research effort.
It does not follow that the two inputs with the largest main effect variances will be the best
two inputs to observe. We would need to calculate
V
i,j
=var{E.Y|X
i,j
/} =var{z
i
.X
i
/ +z
j
.X
j
/ +z
ij
.X
i,j
/} .4/
for all i and j, since this is the part of var.Y/ that is removed on average when we learn both
x
i
and x
j
. The search for the most informative combinations of inputs is considered further
by Saltelli and Tarantola (2002). In general, V
p
=var{E.Y|X
p
/} is the expected reduction in
variance that is achieved when we learn x
p
.
2.3. Variance decomposition
When G is such that the elements of X are mutually independent, we have already remarked
that the definitions of main effects and interactions will directly reflect the model structure. In
this case, we can also decompose the variance of Y into terms relating to the main effects and
various interactions between the input variables. A decomposition like an analysis of variance
is given in Cox (1982):
var.Y/ =
d
i=1
W
i
+
i<j
W
i,j
+
i<j<k
W
i,j,k
+ ... +W
1, 2 , :::,d
, .5/
where W
p
=var{z
p
.X
p
/}. This result holds because when the X
i
s are independent it is straight-
forward to show that all the terms in equation (1) are uncorrelated. Equation (5) gives us a
partition of the variance into terms that are the variances of the main effects and interaction
terms in equation (1).
We have W
i
=V
i
, i.e. the variance of the main effect is the reduction in var.Y/ that is obtained
by learning the true value of x
i
. W
i,j
is the component of var.Y/ due solely to uncertainty about
the interaction between inputs x
i
and x
j
. Note that equation (4) becomes V
i,j
=W
i
+W
j
+W
i,j
=
V
i
+V
j
+W
i,j
,soW
i,j
is an extra amount of variance removed when we learn both x
i
and x
j
,
over the main effect variances V
i
and V
j
.
It is clear that when equation (5) holds we can identify V
i
=var{E.Y|X
i
/} with the sum
of all the W
p
-terms not including the subscript i. Therefore the total effect index (3) is the pro-
portion of var.Y/ that is accounted for by all the terms in equation (5) with a subscript i, and
so S
Ti
S
i
. It is also clear that Σ
d
i=1
S
i
1 Σ
d
i=1
S
Ti
, with equalities only when all interactions
are 0.
Independence between the input variables, therefore, allows a tidy decomposition of the total
variance into component variances that are directly related to the quantities that were discussed
in Section 2.2. An analogy is that in regression analysis we have a nice partition of the total sum
of squares when the regressors, or groups of regressors, are orthogonal. Without orthogonality,
we can still define the sum of squares attributable to any set of variables, but sums of squares
for different sets of regressors no longer partition the total sum of squares.

Probabilistic Sensitivity Analysis 755
2.4. Regression components
The analogy with regression analysis becomes clearer if we consider the variance of Y as an
expected squared error of prediction. Thus, if we wish to predict Y without gaining any further
information about x, then the best prediction (in terms of minimizing the expected squared
error) is E.Y/. Then var.Y/ is the expected squared error of this prediction. Similarly, if we learn
the true value of the subvector x
p
, then the best predictor of Y becomes E.Y|x
p
/ and results in
an expected squared error of E{var.Y|X
p
/}.
Consider predicting Y =η.x/ by a linear model of the form
ˆη.x/ =α +g.x/
T
γ:.6/
The components of the vector function g.x/ are supposed given. We wish to choose α and γ
to obtain an approximation of the form (6) to minimize the expected squared prediction error
E{Y ˆη.X/}
2
. We then find that the optimal approximation is given by
γ =var{g.X/}
1
cov{g.X/, Y }
and α =E.Y/ E{g.X/}
T
γ. The expected squared error is var.Y/ V
g.x/
, where
V
g.x/
=cov{g.X/, Y }
T
var{g.X/}
1
cov{g.X/, Y }:.7/
The inclusion of the constant term α leads to E{Y ˆη.X/} =0. Furthermore, it then holds that
ˆη.X/ is uncorrelated with η.X/ ˆη.X/, and so we have the variance decomposition
var.Y/ =V
g.x/
+var{η.X/ ˆη.X/}:.8/
The interpretation of equation (8) is that V
g.x/
is the component of var.Y/ that is explained
by this fitted approximation, and that the second term measures its lack of fit. Setting g.x/ =x
i
gives a variance component
V
x
i
=cov.X
i
, Y/
2
=var.X
i
/
for a best prediction of Y by a linear function of x
i
alone. Now, since cov.X
i
, Y/ =
cov{X
i
, E.Y |X
i
/}, this is also the best linear predictor of E.Y |x
i
/, and hence of the main effect
of x
i
. Therefore the difference V
i
V
x
i
measures the lack of linearity of this main effect.
Remembering that one objective of sensitivity analysis is to understand the way that the output
responds to changes to the inputs, these linear variance components and their complementary
lack-of-fit components give further insight into the behaviour of the model. By introducing
quadratic and higher order polynomial terms in g.x/ we can refine this understanding further.
We could equally well look at other regressor variables if the nature of the phenomenon that is
being modelled suggested them, such as harmonic terms for a cyclic input.
It should be noted that regression coefficients, correlation coefficients and related sums of
squares have been widely used in sensitivity analysis. Various sensitivity diagnostics based on a
Monte Carlo sample of runs are presented by Kleijnen and Helton (1999) and the use of regres-
sion coefficients in particular is discussed in Helton and Davis (2000). However, our approach
is different in some important respects.
Their approach is based on a Monte Carlo sample {.x
s
, y
s
/, s =1,2,...,N} that is obtained
by sampling the input vectors x
s
from the distribution G and then running the model at each
sampled x
s
to compute output y
s
=η.x
s
/. They regarded the regression or correlation coefficient
between y and each input variable x
i
as measures of sensitivity of the output to that x
i
. Implic-
itly, they are fitting the statistical model y
s
=α +Σ
d
i=1
β
i
x
is
+"
s
, with random-error term "
s
(although Kleijnen and Helton (1999) also considered rank regression and other analyses of the
Monte Carlo data).

Citations
More filters
Journal ArticleDOI
TL;DR: Existing and new practices for sensitivity analysis of model output are compared and recommendations on which to use are offered to help practitioners choose which techniques to use.

2,265 citations

Journal ArticleDOI
TL;DR: In this article, generalized polynomial chaos expansions (PCE) are used to build surrogate models that allow one to compute the Sobol' indices analytically as a post-processing of the PCE coefficients.

1,934 citations

01 Jan 2011
TL;DR: In this paper, a polynomial dimensional decomposition (PDD) method for global sensitivity analysis of stochastic systems subject to independent random input following arbitrary probability distributions is presented.
Abstract: This paper presents a polynomial dimensional decomposition (PDD) method for global sensitivity analysis of stochastic systems subject to independent random input following arbitrary probability distributions. The method involves Fourier-polynomial expansions of lower-variate component functions of a stochastic response by measure-consistent orthonormal polynomial bases, analytical formulae for calculating the global sensitivity indices in terms of the expansion coefficients, and dimension-reduction integration for estimating the expansion coefficients. Due to identical dimensional structures of PDD and analysis-of-variance decomposition, the proposed method facilitates simple and direct calculation of the global sensitivity indices. Numerical results of the global sensitivity indices computed for smooth systems reveal significantly higher convergence rates of the PDD approximation than those from existing methods, including polynomial chaos expansion, random balance design, state-dependent parameter, improved Sobol’s method, and sampling-based methods. However, for non-smooth functions, the convergence properties of the PDD solution deteriorate to a great extent, warranting further improvements. The computational complexity of the PDD method is polynomial, as opposed to exponential, thereby alleviating the curse of dimensionality to some extent. Mathematical modeling of complex systems often requires sensitivity analysis to determine how an output variable of interest is influenced by individual or subsets of input variables. A traditional local sensitivity analysis entails gradients or derivatives, often invoked in design optimization, describing changes in the model response due to the local variation of input. Depending on the model output, obtaining gradients or derivatives, if they exist, can be simple or difficult. In contrast, a global sensitivity analysis (GSA), increasingly becoming mainstream, characterizes how the global variation of input, due to its uncertainty, impacts the overall uncertain behavior of the model. In other words, GSA constitutes the study of how the output uncertainty from a mathematical model is divvied up, qualitatively or quantitatively, to distinct sources of input variation in the model [1].

1,296 citations

Journal ArticleDOI
TL;DR: This paper presents an overview of SA and its link to uncertainty analysis, model calibration and evaluation, robust decision-making, and provides practical guidelines by developing a workflow for the application of SA.
Abstract: Sensitivity Analysis (SA) investigates how the variation in the output of a numerical model can be attributed to variations of its input factors. SA is increasingly being used in environmental modelling for a variety of purposes, including uncertainty assessment, model calibration and diagnostic evaluation, dominant control analysis and robust decision-making. In this paper we review the SA literature with the goal of providing: (i) a comprehensive view of SA approaches also in relation to other methodologies for model identification and application; (ii) a systematic classification of the most commonly used SA methods; (iii) practical guidelines for the application of SA. The paper aims at delivering an introduction to SA for non-specialist readers, as well as practical advice with best practice examples from the literature; and at stimulating the discussion within the community of SA developers and users regarding the setting of good practices and on defining priorities for future research. We present an overview of SA and its link to uncertainty analysis, model calibration and evaluation, robust decision-making.We provide a systematic review of existing approaches, which can support users in the choice of an SA method.We provide practical guidelines by developing a workflow for the application of SA and discuss critical choices.We give best practice examples from the literature and highlight trends and gaps for future research.

888 citations


Cites background or methods from "Probabilistic sensitivity analysis ..."

  • ...…(FAST (Cukier et al., 1973)) for the approximation of the first-order indices, and the extended FAST (Saltelli et al., 1999) for the total-order indices (for an introduction to these techniques, see Norton (2015)); and (ii) methods using an emulator like the approach by Oakley and O'Hagan (2004)....

    [...]

  • ...…indices is that they are related with the terms in the variance decomposition of the model output (Sobol', 1993), which “reflects the structure of the model itself” (Oakley and O'Hagan, 2004) and holds under relatively broad assumptions, the strongest one being that input factors are independent....

    [...]

  • ...An interesting property of first-order and higher-order indices is that they are related with the terms in the variance decomposition of the model output (Sobol', 1993), which “reflects the structure of the model itself” (Oakley and O'Hagan, 2004) and holds under relatively broad assumptions, the strongest one being that input factors are independent....

    [...]

  • ...In the presence of correlations among the input factors, instead, the tidy correspondence between variancebased indices and model structure is lost (see e.g. discussion in Oakley and O'Hagan (2004)) and counterintuitive results may be obtained....

    [...]

Journal ArticleDOI
TL;DR: A novel geometric proof of the inefficiency of OAT is introduced, with the purpose of providing the modeling community with a convincing and possibly definitive argument against OAT.
Abstract: Mathematical modelers from different disciplines and regulatory agencies worldwide agree on the importance of a careful sensitivity analysis (SA) of model-based inference. The most popular SA practice seen in the literature is that of 'one-factor-at-a-time' (OAT). This consists of analyzing the effect of varying one model input factor at a time while keeping all other fixed. While the shortcomings of OAT are known from the statistical literature, its widespread use among modelers raises concern on the quality of the associated sensitivity analyses. The present paper introduces a novel geometric proof of the inefficiency of OAT, with the purpose of providing the modeling community with a convincing and possibly definitive argument against OAT. Alternatives to OAT are indicated which are based on statistical theory, drawing from experimental design, regression analysis and sensitivity analysis proper.

850 citations


Cites background or methods from "Probabilistic sensitivity analysis ..."

  • ...Good practices for sensitivity analysis are also increasingly seen on this journal based on regression analysis (Manache and Melching, 2008), variance based methods (Confalonieri et al., 2010) and meta-modelling (Ziehn and Tomlin, 2009)....

    [...]

  • ...Existing guidelines and textbooks reviewed here recommend that mathematical modeling of natural or man-made system be accompanied by a ‘sensitivity analysis’ (SA)....

    [...]

References
More filters
Journal ArticleDOI
TL;DR: This paper presents a meta-modelling framework for estimating Output from Computer Experiments-Predicting Output from Training Data and Criteria Based Designs for computer Experiments.
Abstract: Many scientific phenomena are now investigated by complex computer models or codes A computer experiment is a number of runs of the code with various inputs A feature of many computer experiments is that the output is deterministic--rerunning the code with the same inputs gives identical observations Often, the codes are computationally expensive to run, and a common objective of an experiment is to fit a cheaper predictor of the output to the data Our approach is to model the deterministic output as the realization of a stochastic process, thereby providing a statistical basis for designing experiments (choosing the inputs) for efficient prediction With this model, estimates of uncertainty of predictions are also available Recent work in this area is reviewed, a number of applications are discussed, and we demonstrate our methodology with an example

6,583 citations

Journal ArticleDOI
TL;DR: A Bayesian calibration technique which improves on this traditional approach in two respects and attempts to correct for any inadequacy of the model which is revealed by a discrepancy between the observed data and the model predictions from even the best‐fitting parameter values is presented.
Abstract: We consider prediction and uncertainty analysis for systems which are approximated using complex mathematical models. Such models, implemented as computer codes, are often generic in the sense that by a suitable choice of some of the model's input parameters the code can be used to predict the behaviour of the system in a variety of specific applications. However, in any specific application the values of necessary parameters may be unknown. In this case, physical observations of the system in the specific context are used to learn about the unknown parameters. The process of fitting the model to the observed data by adjusting the parameters is known as calibration. Calibration is typically effected by ad hoc fitting, and after calibration the model is used, with the fitted input values, to predict the future behaviour of the system. We present a Bayesian calibration technique which improves on this traditional approach in two respects. First, the predictions allow for all sources of uncertainty, including the remaining uncertainty over the fitted parameters. Second, they attempt to correct for any inadequacy of the model which is revealed by a discrepancy between the observed data and the model predictions from even the best-fitting parameter values. The method is illustrated by using data from a nuclear radiation release at Tomsk, and from a more complex simulated nuclear accident exercise.

3,745 citations

Journal Article
07 Apr 2005

3,470 citations

Journal ArticleDOI
TL;DR: In this paper, a new method of global sensitivity analysis of nonlinear models is proposed based on a measure of importance to calculate the fractional contribution of the input parameters to the variance of the model prediction.

1,662 citations


"Probabilistic sensitivity analysis ..." refers background in this paper

  • ...The second measure, first proposed by Homma and Saltelli (1996), is VTi =var....

    [...]

Journal ArticleDOI
TL;DR: In this paper, the Fourier amplitude sensitivity test (FAST) has been extended to include all the interaction terms involving a factor and the main effect of the factor's main effect.
Abstract: A new method for sensitivity analysis (SA) of model output is introduced. It is based on the Fourier amplitude sensitivity test (FAST) and allows the computation of the total contribution of each input factor to the output's variance. The term “total” here means that the factor's main effect, as well as all the interaction terms involving that factor, are included. Although computationally different, the very same measure of sensitivity is offered by the indices of Sobol'. The main advantages of the extended FAST are its robustness, especially at low sample size, and its computational efficiency. The computational aspects of the extended FAST are described. These include (1) the definition of new sets of parametric equations for the search-curve exploring the input space, (2) the selection of frequencies for the parametric equations, and (3) the procedure adopted to estimate the total contributions. We also address the limitations of other global SA methods and suggest that the total-effect indices are id...

1,652 citations

Frequently Asked Questions (1)
Q1. What are the contributions in "Probabilistic sensitivity analysis of complex models: a bayesian approach" ?

The authors present a Bayesian framework which unifies the various tools of probabilistic sensitivity analysis. Furthermore, all measures of interest may be computed from a single set of runs.