uncertainty, stochastics & sensitivity analysis · uncertainty, stochastics & sensitivity...
TRANSCRIPT
![Page 1: Uncertainty, Stochastics & Sensitivity Analysis · Uncertainty, Stochastics & Sensitivity Analysis Nathaniel Osgood CMPT 858 March 17, 2011 . ... Smoking Status of Intervention Target](https://reader031.vdocuments.us/reader031/viewer/2022013104/5b9e9bc709d3f2fc778c3763/html5/thumbnails/1.jpg)
Uncertainty, Stochastics & Sensitivity Analysis
Nathaniel Osgood
CMPT 858
March 17, 2011
![Page 2: Uncertainty, Stochastics & Sensitivity Analysis · Uncertainty, Stochastics & Sensitivity Analysis Nathaniel Osgood CMPT 858 March 17, 2011 . ... Smoking Status of Intervention Target](https://reader031.vdocuments.us/reader031/viewer/2022013104/5b9e9bc709d3f2fc778c3763/html5/thumbnails/2.jpg)
Types of Sensitivity Analyses
• Variables involved – One-way
– Multi-way
• Type of component being varied – Parameter sensitivity
analysis: Parameter values
– Structural sensitivity analysis: Examine effects of model structure on results
• Type of variation – Single alternative values
– Monte Carlo analyses: Draws from probability distributions (many types of variations)
• Frequency of variation – Static (parameter retains
value all through simulation)
– Ongoing change: Stochastic process • Accomplished via Monte-Carlo
analyses
• Key for DES & ABM
![Page 3: Uncertainty, Stochastics & Sensitivity Analysis · Uncertainty, Stochastics & Sensitivity Analysis Nathaniel Osgood CMPT 858 March 17, 2011 . ... Smoking Status of Intervention Target](https://reader031.vdocuments.us/reader031/viewer/2022013104/5b9e9bc709d3f2fc778c3763/html5/thumbnails/3.jpg)
Model Uncertainty • Here, we are frequently examining the impact of
changing
– Our assumptions about “how the system works”
– Our decision of how to abstract the system behaviour
• Structural sensitivity analyses
– Vary structure of model & see impact on
• Results
• Tradeoffs between choices
– Frequently recalibrate the model in this process
• Here, we are considering uncertainty about how the current state is mapped to the next state
![Page 4: Uncertainty, Stochastics & Sensitivity Analysis · Uncertainty, Stochastics & Sensitivity Analysis Nathaniel Osgood CMPT 858 March 17, 2011 . ... Smoking Status of Intervention Target](https://reader031.vdocuments.us/reader031/viewer/2022013104/5b9e9bc709d3f2fc778c3763/html5/thumbnails/4.jpg)
Predictor-Corrector Methods: Dealing with an Incomplete Model
• Some approaches (e.g. Kalman filter, Particle Filter) are motivated by awareness that models are incomplete
• Such approaches try to adjust model state estimates on an ongoing basis,
– Given uncertainty about model predictions
– New observations
• Assumption here is that the error in the model is defined by some probability distribution
![Page 5: Uncertainty, Stochastics & Sensitivity Analysis · Uncertainty, Stochastics & Sensitivity Analysis Nathaniel Osgood CMPT 858 March 17, 2011 . ... Smoking Status of Intervention Target](https://reader031.vdocuments.us/reader031/viewer/2022013104/5b9e9bc709d3f2fc778c3763/html5/thumbnails/5.jpg)
Static Uncertainty Sensitivity Analyses
• In variation, one can seek to investigate different – Assumptions – Policies
• Same relative or absolute uncertainty in different parameters may have hugely different effect on outcomes or decisions
• Help identify parameters/initial states that strongly affect – Key model results – Choice between policies
• We place more emphasis in parameter estimation & interventions into parameters exhibiting high sensitivity
![Page 6: Uncertainty, Stochastics & Sensitivity Analysis · Uncertainty, Stochastics & Sensitivity Analysis Nathaniel Osgood CMPT 858 March 17, 2011 . ... Smoking Status of Intervention Target](https://reader031.vdocuments.us/reader031/viewer/2022013104/5b9e9bc709d3f2fc778c3763/html5/thumbnails/6.jpg)
Spider Diagram
Content
redacted
for
copyright
compliance
• Each axis represents a % change in a particular parameter – This proportional change
is identical for the different parameters
• The distance assumed by the curve along that axis represents the magnitude of response to that change – Note that these
sensitivities will depend on the state of system!
http://www.niwotridge.com/images/BLOGImages/SpiderDiagram.jpg
![Page 7: Uncertainty, Stochastics & Sensitivity Analysis · Uncertainty, Stochastics & Sensitivity Analysis Nathaniel Osgood CMPT 858 March 17, 2011 . ... Smoking Status of Intervention Target](https://reader031.vdocuments.us/reader031/viewer/2022013104/5b9e9bc709d3f2fc778c3763/html5/thumbnails/7.jpg)
Systematic Examination of Policies
08-13
Female14-18 19-24 25-34 35-44 45-54 55-64 65+ 08-13
Male14-18
19-24 25-3435-44 45-54
55-6465+
Cessation
Relapse
Uptake
0
50000
100000
150000
200000
250000
300000
350000
Cessation
Relapse
Uptake
Apply Different Uptake Rates for Children of Mothers who Smoked During Pregnancy? No linkage Time 2050
Sum of Total QALYs
InterventionGender Intervention Age Category
Smoking Status of Intervention Target
Tengs, Osgood, Lin
![Page 8: Uncertainty, Stochastics & Sensitivity Analysis · Uncertainty, Stochastics & Sensitivity Analysis Nathaniel Osgood CMPT 858 March 17, 2011 . ... Smoking Status of Intervention Target](https://reader031.vdocuments.us/reader031/viewer/2022013104/5b9e9bc709d3f2fc778c3763/html5/thumbnails/8.jpg)
Sensitivity Analyses in Vensim
![Page 9: Uncertainty, Stochastics & Sensitivity Analysis · Uncertainty, Stochastics & Sensitivity Analysis Nathaniel Osgood CMPT 858 March 17, 2011 . ... Smoking Status of Intervention Target](https://reader031.vdocuments.us/reader031/viewer/2022013104/5b9e9bc709d3f2fc778c3763/html5/thumbnails/9.jpg)
Sensitivity in Initial States
• Frequently we don’t know the exact state of the system at a certain point in time
• A very useful type of sensitivity analysis is to vary the initial model state
• In Vensim, this can be accomplished by – Indicating a parameter name within the “initial value”
area for a stock
– Varying the parameter value
• In an agent-based model, state has far larger dimensionality – Can modify different numbers of people with
characteristic, location of people with characteristic, etc.
![Page 10: Uncertainty, Stochastics & Sensitivity Analysis · Uncertainty, Stochastics & Sensitivity Analysis Nathaniel Osgood CMPT 858 March 17, 2011 . ... Smoking Status of Intervention Target](https://reader031.vdocuments.us/reader031/viewer/2022013104/5b9e9bc709d3f2fc778c3763/html5/thumbnails/10.jpg)
Imposing a Probability Distribution Monte Carlo Analysis
• We feed in probability distributions to reflect our uncertainty about one or more parameters
• The model is run many, many times (realizations)
– For each realization, the model uses a different draw from those probability distribution
• What emerges is resulting probability distribution for model outputs
![Page 11: Uncertainty, Stochastics & Sensitivity Analysis · Uncertainty, Stochastics & Sensitivity Analysis Nathaniel Osgood CMPT 858 March 17, 2011 . ... Smoking Status of Intervention Target](https://reader031.vdocuments.us/reader031/viewer/2022013104/5b9e9bc709d3f2fc778c3763/html5/thumbnails/11.jpg)
Example Resulting Distribution
Empirical Fractiles
![Page 12: Uncertainty, Stochastics & Sensitivity Analysis · Uncertainty, Stochastics & Sensitivity Analysis Nathaniel Osgood CMPT 858 March 17, 2011 . ... Smoking Status of Intervention Target](https://reader031.vdocuments.us/reader031/viewer/2022013104/5b9e9bc709d3f2fc778c3763/html5/thumbnails/12.jpg)
Impact on cost of uncertainty regarding mortality and medical costs
50% 60% 70% 80% 90% 95% 97% 99%
Incremental Costs
6 B
4.5 B
3 B
1.5 B
0 2001 2014 2026 2039 2051
Time (Year)
Static Uncertainty
![Page 13: Uncertainty, Stochastics & Sensitivity Analysis · Uncertainty, Stochastics & Sensitivity Analysis Nathaniel Osgood CMPT 858 March 17, 2011 . ... Smoking Status of Intervention Target](https://reader031.vdocuments.us/reader031/viewer/2022013104/5b9e9bc709d3f2fc778c3763/html5/thumbnails/13.jpg)
Multi-Way Sensitivity Analyses
• When examining the results of changing multiple variables, need to consider how multiple variables vary together
• If this covariation reflects dependence on some underlying factor, may be able to simulate uncertainty in underlying factor
![Page 14: Uncertainty, Stochastics & Sensitivity Analysis · Uncertainty, Stochastics & Sensitivity Analysis Nathaniel Osgood CMPT 858 March 17, 2011 . ... Smoking Status of Intervention Target](https://reader031.vdocuments.us/reader031/viewer/2022013104/5b9e9bc709d3f2fc778c3763/html5/thumbnails/14.jpg)
Performing Monte Carlo Sensitivity Analyses in Vensim
• Need to specify three things
– The parameters to vary
– How to vary those parameters
– Which model variables to save away
![Page 15: Uncertainty, Stochastics & Sensitivity Analysis · Uncertainty, Stochastics & Sensitivity Analysis Nathaniel Osgood CMPT 858 March 17, 2011 . ... Smoking Status of Intervention Target](https://reader031.vdocuments.us/reader031/viewer/2022013104/5b9e9bc709d3f2fc778c3763/html5/thumbnails/15.jpg)
How & What Parameters to Vary
![Page 16: Uncertainty, Stochastics & Sensitivity Analysis · Uncertainty, Stochastics & Sensitivity Analysis Nathaniel Osgood CMPT 858 March 17, 2011 . ... Smoking Status of Intervention Target](https://reader031.vdocuments.us/reader031/viewer/2022013104/5b9e9bc709d3f2fc778c3763/html5/thumbnails/16.jpg)
Model Values to Save Away
![Page 17: Uncertainty, Stochastics & Sensitivity Analysis · Uncertainty, Stochastics & Sensitivity Analysis Nathaniel Osgood CMPT 858 March 17, 2011 . ... Smoking Status of Intervention Target](https://reader031.vdocuments.us/reader031/viewer/2022013104/5b9e9bc709d3f2fc778c3763/html5/thumbnails/17.jpg)
Monte Carlo Analyses
![Page 18: Uncertainty, Stochastics & Sensitivity Analysis · Uncertainty, Stochastics & Sensitivity Analysis Nathaniel Osgood CMPT 858 March 17, 2011 . ... Smoking Status of Intervention Target](https://reader031.vdocuments.us/reader031/viewer/2022013104/5b9e9bc709d3f2fc778c3763/html5/thumbnails/18.jpg)
Sensitivity Results (Prevalence)
![Page 19: Uncertainty, Stochastics & Sensitivity Analysis · Uncertainty, Stochastics & Sensitivity Analysis Nathaniel Osgood CMPT 858 March 17, 2011 . ... Smoking Status of Intervention Target](https://reader031.vdocuments.us/reader031/viewer/2022013104/5b9e9bc709d3f2fc778c3763/html5/thumbnails/19.jpg)
Sensitivity Results (Fraction of Susceptibles)
An observation at this point in time would produce a histogram (approximating a distribution) for fraction of susceptibles
![Page 20: Uncertainty, Stochastics & Sensitivity Analysis · Uncertainty, Stochastics & Sensitivity Analysis Nathaniel Osgood CMPT 858 March 17, 2011 . ... Smoking Status of Intervention Target](https://reader031.vdocuments.us/reader031/viewer/2022013104/5b9e9bc709d3f2fc778c3763/html5/thumbnails/20.jpg)
Stochastic Processes
• System Dynamics models are traditionally determnistic
• As will be discussed in tutorials, models can be made stochastic
– Transitions between states (e.g. duration of infection or immunity)
– Transmission of infection
• As a result, there will be variation in the results from simulation to simulation
![Page 21: Uncertainty, Stochastics & Sensitivity Analysis · Uncertainty, Stochastics & Sensitivity Analysis Nathaniel Osgood CMPT 858 March 17, 2011 . ... Smoking Status of Intervention Target](https://reader031.vdocuments.us/reader031/viewer/2022013104/5b9e9bc709d3f2fc778c3763/html5/thumbnails/21.jpg)
Summarizing Variability • To gain confidence in model results, typically
need to run an ensemble of realizations
– Deal with means, standard deviations, and empirical fractiles
– As is seen here, there are typically still broad regularities between most runs (e.g. rise & fall)
• Need to reason over a population of realizations statistics are very valuable
– Fractile within which historic value falls
– Mean difference in results between interventions
![Page 22: Uncertainty, Stochastics & Sensitivity Analysis · Uncertainty, Stochastics & Sensitivity Analysis Nathaniel Osgood CMPT 858 March 17, 2011 . ... Smoking Status of Intervention Target](https://reader031.vdocuments.us/reader031/viewer/2022013104/5b9e9bc709d3f2fc778c3763/html5/thumbnails/22.jpg)
Closing Question: How can we best adapt our policies to deal with ongoing uncertainty?
• We are dealing here with making decisions in an environment that changes over time
• This uncertainty could come from – Stochastic variability
– Uncertainty regarding parameter value
• There is an incredibly vast # of possible policies
Baseline50% 60% 70% 80% 90% 95% 98% 100%
Average Variable Cost per Cubic Meter
0.6
0.45
0.3
0.15
00 1457 2914 4371 5828
Time (Day)
![Page 23: Uncertainty, Stochastics & Sensitivity Analysis · Uncertainty, Stochastics & Sensitivity Analysis Nathaniel Osgood CMPT 858 March 17, 2011 . ... Smoking Status of Intervention Target](https://reader031.vdocuments.us/reader031/viewer/2022013104/5b9e9bc709d3f2fc778c3763/html5/thumbnails/23.jpg)
Stochastic Processes in Vensim
Baseline50% 60% 70% 80% 90% 95% 98% 100%
Average Variable Cost per Cubic Meter
0.6
0.45
0.3
0.15
00 1457 2914 4371 5828
Time (Day)