http:// quantitative methods to manage uncertainty in science by andrea saltelli, silvio funtowicz,...
TRANSCRIPT
http://www.jrc.cec.eu.int/uasa 1
QUANTITATIVE METHODSTO MANAGE UNCERTAINTY IN SCIENCEby Andrea Saltelli, Silvio Funtowicz, Stefano Tarantola, Joint Research Centre of the European Communities in Ispra (I), [email protected]
MUST: Managing Uncertainty in science for suSTainability: future research challenges for Europe
FP6 launch event in Brussels, November 11, 2002
http://www.jrc.cec.eu.int/uasa 2
Uncertainty is not an accident of the scientific method, but its substance. Peter Høeg, a Danish novelist, writes in Borderliners (Høeg, 1995):
"That is what we meant by science. That both question and answer are tied up with uncertainty, and that they are painful. But that there is no way around them. And that you hide nothing; instead, everything is brought out into the open".
Uncertainty. Høeg.
http://www.jrc.cec.eu.int/uasa 3
Models mimic systems
N
Naturalsystem
F
Formalsystem
Encoding
Decoding
EntailmentEntailment
Rosen’s formalisation of the modelling process
http://www.jrc.cec.eu.int/uasa 4
“World” (the natural system) and “Model” (the formal system) are internally entailed - driven by a causal structure.
Nothing entails with one another “World” and “Model”; the association is hence the result of a craftsmanship.
But this does not apply to natural systems only: give 10 engineers the blueprint of the same plant and they will return you 10 model based risk assessments for the same plant.
Models mimic systems (Rosen)
http://www.jrc.cec.eu.int/uasa 5
It can help the craftsman that the uncertainty in the information provided by the model (the substance of use for the decoding exercise) is carefully apportioned to the uncertainty associated with the encoding process.
Models mimic systems (Rosen)
http://www.jrc.cec.eu.int/uasa 6
<<[…] most simulation models will be complex, with many parameters, state-variables and non linear relations. Under the best circumstances, such models have many degrees of freedom and, with judicious fiddling, can be made to produce virtually any desired behaviour, often with both plausible structure and parameter values.>>, HORNBERGER G.M., and R. C. Spear (1981) An approach to the preliminary analysis of environmental systems. Journal of Environmental management, 12, 7-18.
<<Cynics say that models can be made to conclude anything provided that suitable assumptions are fed into them.>>, The Economist, 1998.
Models maps assumptions onto inferences ... but often too narrowly
http://www.jrc.cec.eu.int/uasa 7
But yet models are used ...
… and a legitimate question is the following:
“If we had mapped the space of uncertain assumptions honestly and judiciously, would the space of inference still be of use1?”
1Read: do we still have peak around some useful inference (e.g. YES or NO, safe or unsafe, hypothesis accepted or rejected, policy effective or ineffective etc. ) or do we have as many YES as NO etc.?
Use of models in the scientific discourse
http://www.jrc.cec.eu.int/uasa 8
<<I have proposed a form of organised sensitivity analysis that I call “global sensitivity analysis” in which a neighborhood of alternative assumptions is selected and the corresponding interval of inferences is identified. Conclusions are judged to be sturdy only if the neighborhood of assumptions is wide enough to be credible and the corresponding interval of inferences is narrow enough to be useful.>>
Edward E. Leamer, “Sensitivity Analysis would help”, in Modelling Economic Series, Edited by CWJ Granger, 1990, Clarendon Press, Oxford.
Models maps assumptions onto inferences …
http://www.jrc.cec.eu.int/uasa 9
Space of estimated parameters
Space of plausible
models space
Simulation
...
inference
Models maps assumptions onto inferences …
Leamer’s view of global Sensitivity Analysis (SA)
Other assumptions
(Copes with Equifinality)
http://www.jrc.cec.eu.int/uasa 10
Estimated parameters
Input dataModel
Uncertainty and
sensitivity analysis
Models maps assumptions onto inferences …
(Parametric bootstrap version of UA/SA )
Inference
(Parametric bootstrap: we sample from the posterior parameter probability)
(Estimation)
http://www.jrc.cec.eu.int/uasa 11
Estimation of
parameters
Loop on boot-replica of the input data
Model
Bootstrapping-of-the-modelling-process version of UA/SA, after Chatfield, 1995
Inference
(Bootstrap of the modelling process)
(Estimation)
(Model Identification)
http://www.jrc.cec.eu.int/uasa 12
Inference
Data
Prior of Model
Bayesian Uncertainty and Sensitivity Analysis
(Draper 1995, Planas and Depoutot 2000)
Prior of Parameter
s
Prior of Model(s)
Posterior of
Parameters
Posterior of
Model(s)
(Sampling)
http://www.jrc.cec.eu.int/uasa 13
The space of the model induced choices (the inference) swells and shrinks by our swelling and shrinking the space of the input assumptions. How many of the assumptions are relevant at all for the choice? And those that are relevant, how do they act on the outcome; singularly or in more or less complex combinations? (ANOVA-type analysis)
Use of models in the scientific discourse
… and role of uncertainty - sensitivity analysis
http://www.jrc.cec.eu.int/uasa 14
I desire to have a given degree of robustness in the choice, what factor/assumptions should be tested more rigorously? (=> look at how much “fixing” any given f/a can potentially reduce the variance of the output)
E.g. can I confidently “fix” a subset of the input factors/assumptions? The Beck and Ravetz “relevance” issue. How do I find these f/a? (=> total sensitivity indices).
Use of models in the scientific discourse
… and role of uncertainty - sensitivity analysis
http://www.jrc.cec.eu.int/uasa 15
Use of models in the scientific discourse
… and role of uncertainty - sensitivity analysis
))XX|Y(V(E
))XX|Y(E(V
))X|Y(V(E
)XX|Y(V
)Y(V
:toolsSuggested
*iix
*iix
ix
*ii
*i
*i
i*i
i*i
i
x
x
x
x sigma(0,3)1%
sigma(0.2)1%
f(0,4)38%
rest9%
sigma(0,4)1% f(0,1)
6%
f(0,2)14%
f(0,3)30%
Reduced varianceExpected reduced variance
First order effect
Total effect
16
Environmental sustainability Index, From The Economist, Green and growing, The Economist, Jan 25th 2001,
Produced on behalf of the World Economic Forum (WEF), and presented to the annual Davos summit this year.
Robustness ...
17
Mathis Wackernagel, mental father of the “Ecological Footprint” and thus an authoritative source in the Sustainable Development expert community, concludes an argumented critique of the study done presented at Davos by noting:
Robustness ...
http://www.jrc.cec.eu.int/uasa 18
"Overall, the report would gain from a more extensive peer review and a sensitivity analysis. The lacking sensitivity analysis undermines the confidence in the results since small changes in the index architecture or the weighting could dramatically alter the ranking of the nations.”
Robustness - Wackernagel’s critique
http://www.jrc.cec.eu.int/uasa 19
Monte Carlo Analysis: countries’ score for the Technology Development Index, (UN), a composite indicator. Modified as to include variability in weights (e.g. as when using budget allocation or Analytic Hierarchy Process).
Robustness of composite indicator -
a worked example
http://www.jrc.cec.eu.int/uasa 20
-> Monte Carlo of score of country A minus score of country B.
-> Country A generally better off…mostly due to the weight factors a and b
Robustness of composite indicator
a worked example
b
a
http://www.jrc.cec.eu.int/uasa 21
-> Scatter-plot of score of country A minus score of country B (colours) as function of the two most important weights
(red=negative, blue=zero, green=positive)
Robustness of composite indicator
a worked example country A – country B
wei
ght o
f 2nd
indi
cato
rweight of 1st indicator
http://www.jrc.cec.eu.int/uasa 22
How can I identify model structures in the simultaneous presence of several uncertainty sources ?
Use of models in the scientific discourse
… and role of uncertainty - sensitivity analysis
23
?
?
Campolongo et al., 1999, JACSaltelli et al., 1995, JAC
http://www.jrc.cec.eu.int/uasa 24
Is the model-induced choice weak (non robust) because there is an insufficient number of observations, or because the experts cannot agree on an accepted theory?
Use of models in the scientific discourse
… and role of uncertainty - sensitivity analysis
http://www.jrc.cec.eu.int/uasa 25
Useful inference versus falsification of the analysis
Example: imagine the inference is Y = the logarithm of the ratio between the two pressure-on-decision indices (Tarantola et als. 2000).
Y=Log(PI 1/PI 2)
Region where Region whereIncineration Landfillis preferred is preferred
Frequency of occurrence
http://www.jrc.cec.eu.int/uasa 26
Useful inference versus falsification of the analysis
http://www.jrc.cec.eu.int/uasa 27
What happens if I address the space of the policy options?
Use of models in the scientific discourse
… and role of uncertainty - sensitivity analysis
28
Gauging the leverage of the policy options latitude
Simulation Model
parameters
Policy Options
data
errors
resolutionlevels
model structures
uncertainty analysis
sensitivity analysismodel output
feedbacks on input data and model factors
desired target bounds
29
Use of models in the scientific discourse
… and role of uncertainty - sensitivity analysis
A broader context of knowledge production and its quality as input to the policy process. Merging formal with informal tools. Ongoing work at the JRC with Angela and Silvio.
http://www.jrc.cec.eu.int/uasa 30
Simulation Model
parameters
data
errors
resolutionlevels
model structures, assumptions
key indicators
feedbacks on input data and model factors
desired target bounds
Problem definition Problem definition
Policy Options(improving quality of air)
Policy Options(improving quality of air)
Policy Assumptions (10% reduction VOC level)
Policy Assumptions (10% reduction VOC level)
SocietySociety
Institutional LevelInstitutional Level
Institutional & Scientific
Level
Institutional & Scientific
Level
Scientific LevelScientific Level
Issue (health diseases: asthma,… )Issue (health diseases: asthma,… )
Normative frameworkNormative framework
Framing assumptionsFraming assumptions
Assessment: Monitoring changeAssessment: Monitoring change
Stakeholder mapping
Stakeholder mapping
Sensitivity AnalysisSensitivity Analysis
Uncertainty Analysis
Uncertainty Analysis
StakeholdersStakeholders
Scientific Pedigree
Scientific Pedigree
Extended Quality
Assurance
Extended Quality
Assurance© ULYSSES: De Marchi et. al, 1998.
http://www.jrc.cec.eu.int/uasa 31
Conclusions
The output from global Unc. A./Sens. A. can feed back into the extended peer review process via e.g.
- refocusing of the critical issues,
- re-assignment of weights for multiple criteria, or
- of inference falsification (or otherwise)
- identification of policy relevance/ irrelevance
http://www.jrc.cec.eu.int/uasa 32
Further reading on SA
Saltelli et als. Computer Physics Communications 2002, Saltelli et als. Statistical Science, 2000Saltelli et als. JASA, 2000 (available here)
Saltelli et al. Eds., Sensitivity Analysis, John Wiley & Sons publishers, Probability and Statistics series (buy it!)