Model uncertainty and sensitivity analysis

© 2004-2005 William Charteris
www.billcharteris.com
www.imperialconsulting.net

Model uncertainty relates to the degree to which a chosen model accurately represents reality. It should ideally be addressed with sensitivity analysis; however, this view is not unanimously shared within the scientific community.

Sensitivity analysis comprises (i) mathematical, (ii) statistical, and (iii) graphical methods. Examples of mathematical sensitivity analysis methods include nominal range sensitivity analysis, breakeven analysis, difference in log odds ratio, and automatic differentiation. They are used to assess sensitivity of a model output to the range of variation of an input. They typically involve calculating the output for a few values of an input that represent the possible range of the input. They do not address the variance in the output due to the variance in the inputs, but they can assess the impact of a range of variation in the input values on the output. Examples of statistical sensitivity analysis methods include regression analysis, analysis of variance, response surface methods, Fourier amplitude sensitivity test, and mutual information index. They involve running simulations, such as Monte Carlo analysis, in which inputs are assigned probability distributions and assessing the effect of variance in inputs on the output distribution. Depending upon the method used, one or more inputs are varied at a time. Statistical methods allow one to identify the effect of interactions among multiple inputs. Examples of graphical sensitivity analysis methods include scatter plots and spider plots. They give representation of sensitivity in the form of graphs, charts, or surfaces. Generally, graphical methods are used to give 2-D and 3-D visual indication of how an output is affected by variation in inputs. They can be used as a screening method before further analysis of a model or to represent complex dependencies between inputs and outputs. In addition, they can be used to complement the results of mathematical and statistical methods for better representation.

There is no single sensitivity analysis method that is clearly superior to all others. Each method has its own strengths and limitations. In this regard, food safety risk models have important features that, taken individually, may favor one method over another. However, when taken together, there is no one obvious best method. A wide range of sensitivity analysis methods are not available in commercial risk analysis software, such as @RISK® and Crystall Ball®. They must be performed separately in dedicated mathematical (e.g. MATLAB®, Mathematica®, etc.) and statistical (e.g. SAS®, MINITAB®, STATISTICA®, etc.) software packages.

This abstract is taken from a paper entitled 'Uncertainty and risk', which was published on December 20, 2004. The paper comprises 3,900 words and 25 references. Individual copies of the paper may be requested by e-mail from the author.