from the morning - cnr · 2016. 7. 5. · non-rigorous x14 “better decision with than without ......

15 June 2016 Termites and Penguins Alpine Summer School, Aosta © Leonard Smith

From the MorningModellers are suppose to fiddle and to tune.Simulators must only to tune.

Plausible Stories or Actionable Numbers (probabilities are numbers).


Science advances to provide actionable information. It did not take weather forecasting another 30 years!

“Based on the Laws of Physics” may not be sufficient?is it necessary?


Fitzroy, 1862

Fitzroy aimed to provide decision makers with more information: neither orders nor decisions made.

www.lsecats.ac.uk

From Termites and Penguins to Blocking and the Andes:

Interpreting Models of the Earth System

Leonard Smith

London School of Economics

& Pembroke College, Oxford

This Talk Would Not Be Possible without:

http://www.lsecats.ac.uk/

http://images.google.com/imgres?imgurl=http://www.classicalrecording.co.uk/images/pembroke_crest.gif&imgrefurl=http://www.classicalrecording.co.uk/catalogue/pages/pembroke.htm&h=126&w=107&sz=3&tbnid=F6NKgEvDB9cJ:&tbnh=84&tbnw=72&start=11&prev=/images?q=Pembroke++crest&hl=en&lr=

http://images.google.com/imgres?imgurl=http://www.classicalrecording.co.uk/images/pembroke_crest.gif&imgrefurl=http://www.classicalrecording.co.uk/catalogue/pages/pembroke.htm&h=126&w=107&sz=3&tbnid=F6NKgEvDB9cJ:&tbnh=84&tbnw=72&start=11&prev=/images?q=Pembroke++crest&hl=en&lr=

http://www.ensembles-eu.org/

http://www.ensembles-eu.org/


Hint:

Would you rather be:

Rigorous

Relevant ?

TSA ATS

non-rigorous x14

“Better decision with than without”


Hurricanes, traffic jams,

oil prices, pirate attacks,

epidemics,nuclear stewardship...

bird food consumption

InsightWe cannot simulate many things we can think of!

But what if we have not thought of everything,...Some effects are emergent,...


We cannot simulate many things we can think of!We cannot simulate many things we can think of!didnot

What about things we have not thought about (or cannot afford) that matter?

But what if we have not thought of everything,...Some effects are emergent,...

Hurricanes, traffic jams,

oil prices, pirate attacks,

epidemics,nuclear stewardship...

bird food consumption


If termites are thought to play a major role in the carbon budget of a warmer world:Should we include them in todays models?

If we cannot simulate the relevant environment with sufficient fidelity to drive a perfect-termite-model, what good does it do us to have a subroutine called “termites” and claim they are “included”? Are decisions makers better informed by clearly

knowing the limits of our models or by today’s “best available”?

If we cannot simulate the relevant environment with sufficient fidelity to drive a perfect-termite-model, what good does it do us to have a subroutine called “termites” and claim they are “included”?


Options for Investment in a Forecast System

What to observe regularlyImprecision level

Model ComplexityModel Tuning/Parameter Estimation

User/Operator/Reveiwer Training

Sunk Costs

Operational Costs

Observation (adaptive options)Data Assimilation (Obs into Model-state space)

Ensemble Formation Scheme (selecting monte carlo states at t=0)Ensemble Evolution under “the” Model

Ensemble Distribution to Probability Forecasts (for t ≥ 0)(Forecast Evaluation)

Application to Decision Making Under Uncertainty/Ambiguity


How would you design a forecast system from scratch?Suppose a newly rich nation rang up your earth sciences department and asked for assistance in designing a new “Earth/Economic System” simulation model from scratch. A philosophically sound model for rational decision support:

How complicated/complex a model should you attempt?How will you communicate your results?You would still face some constraints, although money is no object!

You can use the best computer technology of 2016You can use the best scientific understanding of 2016You can provide uncertainty information, even PDFs. (Numerate user)You can provide information as far into the future as you can provide information.Guidance is needed “quickly”, but the exact cost of delay is part of the project!

You are not constrained by:•Legacy code•Legacy domain specialists•Blatant Political Interference

What are you constrained by?(Given a target)


For decision support, the model has to run faster than real time.The larger the lead time, the fewer ensemble members you can run to examine sensitivity.

ForecastLead time

Complex Models

Simple Models

1000

100

10

1

0.1

0.01

0.001

0.0001

Run

Tim

e R

atio

DTC & NUOPC Ensemble Design Workshop 10 Sept 2012 Leonard Smith

We will quantify complexity in terms of a model’s run-time-ratio.A model with run-time-ratio of 10 will run 10x slower than the system being modelled.

(That is, “10” implies it will take ten days to simulate one model-day.Sometimes fine for science, never good for decision support.)

This impacts ensemble size, maximum lead time considered, resolution and which phenomena to “include” in the model.

The cost of Data Assimilation must be counted in addition.

What are you constrained by?How would you design a forecast model?


Complex models may not fit in current hardware, even if you know which model class you would deploy. And the more complex your model, the fewer “simulation hours” available.

ForecastLead time

Complex Models

Simple Models

1000

100

10

1

0.1

0.01

0.001

0.0001

Run

Tim

e R

atio

InaccessibleAccessible

Technological Constraints



Requirements for model fidelity sets a lower bound on the complexity with lead time.Almost always, the model is required to grow more complex at larger lead times.

What are you constrained by?

ForecastLead time

Complex Models

Simple Models

1000

100

10

1

0.1

0.01

0.001

0.0001

Run

Tim

e R

atio


RelevantIrrelevant

Technological Constraints Fidelity Constraints

How would you design a forecast model?


Limits of current scientific/mathematical knowledge mean the model may prove inadequate. In the financial sector, regulators tolerate this as long as the Prob(Big Surprise) < 0.005

ForecastLead time

Complex Models

Simple Models

1000

100

10

1

0.1

0.01

0.001

0.0001

Run

Tim

e R

atio


RelevantIrrelevant

Technological Constraints Fidelity ConstraintsKnowledge Constraints

Prob(Big Surprise) > 1 in 200

be expected to

^



The decision you take will depend on how these three curves lie.

ForecastLead time

Complex Models

Simple Models

1000

100

10

1

0.1

0.01

0.001

0.0001

Run

Tim

e R

atio


RelevantIrrelevant


Prob(Big Surprise) > 1 in 200 (Basel III)



The decision you take will depend on how these three curves lie.

ForecastLead time

Complex Models

Simple Models

1000

100

10

1

0.1

0.01

0.001

0.0001

Run

Tim

e R

atio


RelevantIrrelevant





ForecastLead time

Complex Models

Simple Models

1000

100

10

1

0.1

0.01

0.001

0.0001

Run

Tim

e R

atio


RelevantIrrelevant



Implied Uncertainty(Knightian Risk)

Intractability Ambiguity(Knightian Uncertainty)

What are the challenges we face with interpreting model simulations in different regions of this schematic?



We need to be above the green line, below the red, and to the left of the blue.So we could make one relevant 100 day simulation and have it tomorrow.

ForecastLead time

Complex Models

Simple Models

1000

100

10

1

0.1

0.01

0.001

0.0001

Run

Tim

e R

atio


RelevantIrrelevant



x



But in this case, a valuable “100 day” forecast is out of our reach.We could run a simple model anyway, call it “best available” knowing it is both best and irrelevant; and pass it on (saying clearly that Prob(B.S.)~1)

ForecastLead time

Complex Models

Simple Models

1000

100

10

1

0.1

0.01

0.001

0.0001

Run

Tim

e R

atio


RelevantIrrelevant



x



But in this case, a valuable “100 day” forecast is out of our reach.We could run a simple model anyway, call it “best available” knowing it is both best and irrelevant; and pass it on (saying clearly that Prob(B.S.)~1)

ForecastLead time

Complex Models

Simple Models

1000

100

10

1

0.1

0.01

0.001

0.0001

Run

Tim

e R

atio


RelevantIrrelevant


x


M1 =α1 P1 + (1-α1)Pclim

In weather-like forecast tasks, cases like this are exposed when statistical models outperform complex models (for example: α goes to zero).

What is the best approach in climate-like forecasting tasks?


What is a “Big Surprise”?Big Surprises arise when something our simulation models cannot mimic turns out to have important implications for us.

Often we can identify cases where we are “leaking probability” when a fraction of our model runs explore conditions which we know they cannot simulate realistically. (Science can warn of “known unknowns” even when the magnitude remains unknown)

Big Surprises invalidate (not update) model-based probability forecasts, the I in P(x|I) changes. (Arguably “Bayes’ Theorem” does not apply: this is not a question of probability theory.)

In weather forecasting, we can see when our models become silly, but in climate forecasting we are in the dark.

If our models agreed (in distribution) would we have more confidence in their simulations?


Weighing AlternativesSchematic view of value added for improving initial condition uncertainty.

Increasing Real-time Cost

Return on Investment

100

10

1

Valu

e Add

ed

OBS Coverage (Gaps in Space)OBS Precision (Noise level)Ensemble SizeData Assimilation Complexity

… plus your favourite here …

ENS size

OBS Precision

DA Scheme(s)

OBS Coverage

These curves are not independent. The curves vary with the target.Development costs start from different legacy baselinesHistorically these “optimised” separately (?draw on separate budgets?)

α threshold - - -M1 =α1 P1 + (1-α1)Pclim


What about “the” Multi-model Case?Could there be a general result?

Increasing Real-time Cost

Case Dependent Result

100

10

1Valu

e in

App

licat

ion

Quality Models (each)Careful e-formation (?each?)Complementary Dynamical weaknesses (across)

Focus on # of models for its own sake

Similar modelsUncoordinated e-formation

Optimised single model structure ensemble

DTC & NUOPC Ensemble Design Workshop 10 Sept 2012 Leonard Smith


Ensembles Forecasting as a Resource Allocation Problem

What do ensembles really tell us?

And how much should we invest in them?


One High Class Model Run or Many Cheaper Model Runs

Resource Allocation given finite computer power.Should I invest in:

And what do either tell me??A Probability?!?


P(x | Data,G) G is complete True knowledge

P(x | data, gt) gt is incomplete but True

P(x | data, g) g is useful (?Walsh?) approximations of gt

Laplace's Demon (1814) 1) Perfect Equations of Motion (PMS)2) Perfect noise-free observations3) Unlimited computational power

Demon’s Apprentice (2007) 1) Perfect Equations of Motion (PMS)2) Perfect noise-free observations (Noise Model)

3) Unlimited computational power

Demon’s Novice (2012) 1) Perfect Equations of Motion (PMS)2) Perfect noise-free observations3) Unlimited computational power

P(x | data, I)Laplacian Demons


Smith (2002) Chaos and Predictability in Encyc Atmos Sci

The evolution of this probability distribution for the chaotic Lorenz 1963 system tells us all we can know of the future, given what we know now.

It allows prudent quantitative risk management (by brain-dead risk managers)

And sensible resource allocation.

We can manage uncertainty for chaotic systems (given a perfect model, same code, compiler, machine --- and ghostdog has been retired...).But how well do we manage uncertainty in the real world? For GDP? Weather? Climate?

Probability Forecasts and Chaos(Nonlinearity)

There is a danger of misinterpreting model diversity as if it were physical probability


The NAG Board(Not A Galton Board)

Constructed in 2000 for the 150th Birthday of

Royal Meteorological Society


In the NAG board, probability forecasting corresponds to predicting with a collection (ensemble) of golf balls…

Ensembles inform us of uncertainty growth within our model!(Telling us about the next golf ball.)

And I have observations from 1024 golf balls!

An Ensemble of Golf Balls


Ensembles inform us of uncertainty growth within our model(s)!

But reality is not a golf ball…

Diversity is not Uncertainty

… reality is a red rubber ball.

What exactly does my distribution of 1024 golf balls tell us about the one (and only) red rubber ball?

While we never see similar initial states, we can still learn from our mistakes!(in this weather-like case)


Climate predictions require extrapolating out of the observed archive: into the known-to-be-different ?malleable? unknown.

The best we can hope for is consistency in distribution between our models (indicating that “the details do not matter”).

Climate and the NAG Board

LA Smith, (2002) What Might We Learn fr Climate Forecasts? Proc. National Acad. USA 4(99): 2487-2492.

“No presentation of model-based probabilities is complete without an

expression of model irrelevance.”

The Probability of a Big Surprise

http://www2.lse.ac.uk/CATS/Publications/Publications%20PDFs/LA-Smith,-(2002)-What-Might-We-Learn-from-Climate-Forecasts.pdf


“The advantage of confining attention to a definite group of abstractions, is that you confine your thoughts to clear-cut definite things, with clear-cut definite relations. … The disadvantage of exclusive attention to a group of abstractions, however well-founded, is that, by the nature of the case, you have abstracted from the remainder of things.

... it is of the utmost importance to be vigilant in critically revising your modes of abstraction. Sometimes it happens that the service rendered by philosophy is entirely obscured by the astonishing success of a scheme of abstractions in expressing the dominant interests of an epoch.”

A N Whitehead. Science and the Modern World. Pg 58/9

Fallacy of Misplaced Concreteness

Whitehead was criticising the straightjacket of Newtonian science; today, perhaps, computer simulation may impede more than just the progress of science.

In the real world, mathematics is never rigorously relevant!(beyond the integers!)


Model 1

Model 2

Probability Forecasts: “Simple” “chaotic” Physical System Probabilistic Forecasts: “Simple” “chaotic” Physical System


Model 1

Model 2

Probabilistic Forecasts: “Simple” “chaotic” Physical System


Model 1

Model 2

Big Surprise in the Moore-Spiegel Circuit (by Reason Machette)

One Initial State Another Initial State

Better decision with than without: yes.Probabilistic: yes. Decision relevant Probability: no.

P(x|I) and P(Big Surprise)


So: Do we include Termites in our Model for 2100 or Not?

Lets break this question down a bit:

First assume we can simulate termite populations with high fidelity.And ask:

Are termites a (likely) major player in the carbon cycle this century?IF yes: When are they likely to take on a new role?And how long after that point will they excite other feedbacks?Do model simulate termite environments with sufficient fidelity?

If not: we know we cannot make scientific simulations of the earth system until 2100 for purposes of decision support even with a perfect termite model!What is our new upper bound? 2100 is now known to be intractable.

If termite are a primary driver at this new bound, then build a termite model.


Common, nontrivial, systematic errors

This map shows what is missing in HadCM3 (All 2012-ish GCMs)



This map shows what is missing in HadCM3 (All 2012-ish GCMs)We know how to simulate rock, but the Andes are 2km short.

Can we infer a simulation “fidelity time limit”? Say P(BS) ~ 0.50?


Clarifying Uncertainty and Probability

Smith, L.A. and Stern, N. (2011) 'Uncertainty in science and its role in climate policy', Phil. Trans. R. Soc. A, 369, 1-24.

Imprecision

Ambiguity

Intractability

Indeterminacy

A well defined value that is considered imprecisely known(thermal expansion coefficient of H20, height of the pot as f(Θ),...)on which we put a probabilitydistribution given information I

A well defined value for which we lack sufficient information to pose a quantitative probability distribution.

A quantity that may be precisely defined, but which is beyond our (current) ability to estimate with quantified precision. (GMT2500, T2100(ORD))

A quantity which is in fact not uniquely (precisely) defined.(numerical viscosity, radius of an epicycle of Mars, ...)

This shines a light on the evolution of simulation complex systems.Domain experts might better model as best they can, not for “inclusion”.Can we distinguish “best available” from “adequate for purpose”?And if decision makers ask for (fund) only “adequate” time scale runs?

http://www.lse.ac.uk/CATS/Publications/Publications%20PDFs/86_SmithStern_Uncertainty_2011.pdf


Atmosphere Atmosphere Atmosphere Atmosphere Atmosphere Atmosphere

Land surfaceLand surfaceLand surfaceLand surfaceLand surface

Ocean & sea-ice Ocean & sea-ice Ocean & sea-ice Ocean & sea-ice

Sulphateaerosol

Sulphateaerosol

Sulphateaerosol

Non-sulphateaerosol

Non-sulphateaerosol

Carbon cycle Carbon cycle

Atmosphericchemistry

Ocean & sea-icemodel

Sulphurcycle model

Non-sulphateaerosols

Carboncycle model

Land carboncycle model

Ocean carboncycle model



Off-linemodeldevelopment

Strengthening coloursdenote improvementsin models

1975 1985 1992 1997 Present

Towards Comprehensive Earth System ModelsPast present and future

Rather like Stamp Collecting?Must get one first…

… then find a better one.Exacerbates the Penguin EffectIn 1975 each was known to be potentially important.

Termites


>>

Sexton 2010 http://www.exeter.ac.uk/media/universityofexeter/research/inspiringresearch/sciencestrategy/ccsf/docs/Making_probabilistic_climate_projections_for_the_UK_presentation.pdf

Model diversity is only a lower bound on structural uncertainty, which may well be by far the biggest piece of the pie.Are the UKCP09 probabilities mature? Does anyone believe them?

http://www.exeter.ac.uk/media/universityofexeter/research/inspiringresearch/sciencestrategy/ccsf/docs/Making_probabilistic_climate_projections_for_the_UK_presentation.pdf


End Lecture 1

HE3 in a Changing Climate Bad Honnef 7 Dec 2014 Leonard Smith

Mature Probabilities P(x|I)A mature probability is not expected to change without additional observation or new theoretical insight. (An nontrivial change in I )

If the fidelity of a simulation model is constrained by technology (as when you know exactly what you would do with more compute power, and it is NOT to run massive ensembles/emulators), then probability distributions based on simulations from that model (or family of similar models) are not expected to be mature.

Rational action is constrained only by mature probabilities.

Over confidence (“belief”) leads to the mine shaft gap…

(?How might one use im-mature distributions in decision support?)(Generalised from IJ Good’s “Dynamic Probability”, as when output from a chess program must be used before the algorithm completes. )

IJ Good (~1979) Science


Before Discussing Types of Probability Further,How are Unknown Unknowns Different? So we knew termites might be important.and arguably we know the Andes are important.

“Known Unknowns” we can sample with ensembles (as long as it is only a real value that is unknown!)

Those we choose not to sample (or not to include at all) are “Known Neglecteds”, not really unknowns at all. ?max sim time?

increases P(BS)Before moving on to real climate model simulation (or talking about the Andes in detail) let’s consider “Unknown Unknowns” and how they limit our ability to make decision relevant statements of probability.

We must not have a mind shaft gap!


Mind Shaft Gap

We cannot simulate many things we can think of!didnot

Either no probabilities or two: Prob(x|Imodel) Prob(Big Surprise)

"The cost of solving the Comet mystery must be reckoned neither in money nor in manpower."Winston Churchill, 1954

Neptune Vulcan1846 1859

LeVerrier

https://en.wikipedia.org/wiki/Winston_Churchill


Mind Shaft Gap

We cannot simulate many things we can think of!didnot

How about estimating the time it takes a ball to reach the ground when dropped from a tower?

(thanks to Dave Higdon)


Chasing Model Inadequacy(by dropping balls off towers)

http://www2.nstec.com/Documents/Fact%20Sheets/U1a%20Facility.pdf

1000

ft

Ua1

sha

ft

2 bowling balls3 Basketballs2 golf balls3 Wiffle balls… (no rubber duck)

detectors

Ball(s)


How close was our median time for basketballs?

A. < 0.01 sec

B. <0.1 sec

C. < 1 sec

D. < 1 min

E. > 1 min

Basket ball.Initial velocity zero.1000 ft “tower”.

Laser sheet timing.

Q3.1


How close was our median time for basketballs?

A. < 0.01 sec

B. <0.1 sec

C. < 1 sec

D. < 1 min

E. > 1 min

< 0.01 sec

<0.1 sec

< 1 sec

< 1 min

> 1 min

0%

18%14%

41%

27%

Basket ball.Initial velocity zero.1000 ft “tower”.

Laser sheet timing.

Q3.1

My students and postdocs


So: How accurate were our drop

time estimates?

“The bowling ball was completely destroyed.”“One of the basket balls failed to make it to the bottom.”

A “Big Surprise” is when something your model doesn’t reflect is important: We thought surface roughness was the main Unknown.

Sometimes, scientists can estimate Prob(BS) (but not within the models, of course)

I only have preliminary results, but I think they make my point rather better than any data I could have made up:


An Oxford Bus Stop, Summer 2010:

Addressing Climate-like Questions ScientificallyRequires doing Science in the Dark

Scientists are forced to “violate” traditional best practice guidance if such violations are imposed by the nature of the question being addressed.

One cannot wait 50 years for out-of-sample observations.It is a brute fact that a climate model’s lifetime is less than it’s forecast lead-time!

The physics underlying CO2 induced warming remains as solid as science gets.

Other groups working in the dark sometimes embrace model inadequacy more than we do, and speak much more tentatively.

Geoscientists are not aloneClimate-like Questions are also found in:

Nuclear Stewardship, Novel Engineering DesignNational Intelligence, Reactor Safety, Americas Cup Design, Vehicle Crash-worthiness, Waste Disposal,

(Re)Insurance

Geoscientists are not aloneClimate-like Questions are also found in:

Nuclear Stewardship, Novel Engineering DesignNational Intelligence, Reactor Safety, Americas Cup Design, Vehicle Crash-worthiness, Waste Disposal,

(Re)Insurance


Things are NOT HOPELESS (nor Useless)!

http://www.edfenergy.com/html/index.jsp

http://www.edfenergy.com/html/index.jsp


Things are NOT HOPELESS (Useless)!Weather-like task: Predicting Pirates

In Weather-like tasks one builds up a large archive of forecast-outcome pairs; the life-time of the model is much longer than the lead-time of the forecast.

In Climate-like tasks, the lifetime of a model (sometimes a professional) is much less than the lead-time of the forecast. Knowledge is gained with time, but the problem remains one of extrapolation.


http://www.metoffice.gov.uk/media/pdf/n/3/A3-plots-temp-OND.pdf

Seasonal-like simulation is not quite in the dark.

Weather-like simulations are in the light.

(we use observations to evaluate them as with in traditional science)

What shall we aim for?

And how do we use it?

?Confidence? ?Insight??Numbers? ?P(x|I) & P(BS)?or something different?

With climate-like tasks, science is done in the dark.

http://www.metoffice.gov.uk/media/pdf/n/3/A3-plots-temp-OND.pdf


Page 591

Where should decision makers draw the line?

Clear, plain spoken discussion of what today’s models cannotcapture quantitatively would be of great value.


MUSIC/Quesitons?


Climate Modelling in Practice


Was Stephen Foster ahead of his Time?

It rained all nightThe day I leftThe weather it was dry

The sun so hotI froze to deathSusanna don’t you cry

These words always seemed nonsense to me.And then I learned about “anomalies”.


In this

http://www.globalchange.gov/images/cir/pdf/20page-highlights-brochure.pdf

http://www.ipcc.ch/publications_and_data/ar4/wg1/en/figure-spm-4.html

Statistical post-processing: These are anomalies, not temperatures.How can we ignore an range of anomaly corrections wider than what would cause “dangerous climate change”?

?When are (well labelled) anomalies decision-relevant?

http://www.globalchange.gov/images/cir/pdf/20page-highlights-brochure.pdf

http://www.ipcc.ch/publications_and_data/ar4/wg1/en/figure-spm-4.html

Hanover How to Build Trust 10 June 2015 Leonard Smith

Models agree that a wide range of sorta-Earth-like planets warm about the same amount under the observed forcing.

Anomalies, Systematic Errors, Laws of Physics

IPCC AR4


As we refocus from climate-past to the climate-future, how do we cope with such systematic errors, even as we work to reduce them?

Obs

AR4 Simulations without 1900-1950 anomaly adjustment

Anomalies may be fine for mitigation.They are a nonsense for quantitative adaptation.

Systematic errors are larger than the observed effect


As we refocus from climate-past to the climate-future, how do we cope with such systematic errors, even as we work to reduce them?

Obs

AR4 Simulations without 1900-1950 anomaly adjustment

Moving to anomaly space requires letting go of the “Laws of Physics”.Note model anomalies are notexchangeable even after 100 years!

Anomalies may be fine for mitigation.They are a nonsense for quantitative adaptation.

(and the laws of physics.)(and biology.)

(Ice melts at zero C, plants die at ….)

While systematic errors are larger than the observed effect


AnomaliesThe AR5 is a bit more forthcoming

}


CMIP5


Anomalies

And why was the anomaly period shifted?

The AR5 is a bit more forthcoming

}


Limits to Transparency: Anomalies 2013 X



This map shows what is missing in HadCM3All 2012’s GCMs suffer from related inadequacies.How much does technology limit our foresight of

Health, Energy, & Extreme Events


Model Fidelity

The detail you see above is what is missing in HadCM3: the large squares reflect model grid resolution, the detail reflects the difference between the observed surface height and the model surface height, “constant” “within” a grid point.Insurance Company with a snowfall question…

Climate Model Points(the squares)(What you see is NOT in the model)

A very schematic schematic reflecting phenomena the model “includes”.

“included” vs “realistically simulated”

Karl and Trenberth 2003

“in

clu

ded

” vs

“r

ealis

tica

lly s

imu

late

d”


AR4Implications for quantitative prediction?

The IPCC itself might say which space-time scales are realistic as a function of lead-time?

Smith, L.A. (2002) 'What might we learn from climate forecasts?', Proc. National Acad. Sci. USA, 4 (99): 2487-2492

http://www.lse.ac.uk/CATS/Publications/Publications%20PDFs/LA-Smith,-(2002)-What-Might-We-Learn-from-Climate-Forecasts.pdf


AR5

Real-world GMT is “likely” (~66% chance) to be in “the range” of model-land GMT.

That suggests there is a significant chance the real-word will be outside the range of the models.

If your downscaling model was perfect, there remains a huge chance you could not catch the relevant pathway (as none of today’s models do).

I think it is fair say the IPCC implies that the Probability of a Big Surprise (GMT in 2100) is about one in ~ four to ~ten.

https://www.ipcc.ch/report/ar5/wg1/docs/WGIAR5_SPM_brochure_en.pdf

By law, we require banks and insurance companies hold costly reserves to cover one in 200 year events.

Implications for quantitative prediction?

https://www.ipcc.ch/report/ar5/wg1/docs/WGIAR5_SPM_brochure_en.pdf


>>

Sexton 2010 http://www.exeter.ac.uk/media/universityofexeter/research/inspiringresearch/sciencestrategy/ccsf/docs/Making_probabilistic_climate_projections_for_the_UK_presentation.pdf

For all we know, this piece of the pie is bigger than all the other pieces combined.

A lower bound with 2014 hardware is not an estimated value.

What to do? Say?

Model diversity is only a lower bound on structural uncertainty, which may well be by far the biggest piece of the pie.

We know today our model-based probabilities are not mature.

http://www.exeter.ac.uk/media/universityofexeter/research/inspiringresearch/sciencestrategy/ccsf/docs/Making_probabilistic_climate_projections_for_the_UK_presentation.pdf


http://www.ukcip.org.uk/

Is it plausible to provide a PDF of hottest or stormiest summer day in 2080’s Oxford???

http://www.ukcip.org.uk/


UKCP09 Worked Examples…A “charitable reading” of the UKCP09 documentation is not possible.

An ensemble a “charitable reading” is possible, and leads to vastly different implications for intended interpretation and use, and implied adequacy.

The intention is made rather more clear by the “worked examples” in the main report, and the description of the designers said what would will allow.


I do not have time to discuss the UKCP worked examples…But each of these makes naïve realist assumptions

Simulate and Count

X


Ensembles are not UselessOne needs to be clear which question is being answered.

And whether that question was poised in model-land or real-world.

Whether or not real-world conclusions have both a mathematical and a scientific basis


What do you say when “Best Available” is known not to be “Adequate for Purpose”

Refuse to interpret this to answer questions for which it is inadequate.

Apply state-of-the-art (baffling) statistical fixes (known not to be adequate).

Attempt physical fixes (testing adequacy to the extent possible)Tales of Future Weather (Nature, 2015)

Construct models optimised for adequacy (and thus, say, only addressing lead times on which they add information to “physics-free” empirical [surrogate] models)

(In Adaptation) Change regulatory/design to incorporate real time insight Just Enough Decisive Information (JEDI) Design


If we were to start modelling climate for decision making today, how might we do it differently?

which initial phenomena?require clear empirical evaluation?

enhanced numerical awareness!On what timescales can probabilistic simulation based forecasting outperform benchmark empirical models? (< decade)

Might a minimalist approach:• reduce the “out of range” probability reduce the Prob(BS)• allow principled development of complexity Numerically efficient• yield empirically supported estimates of reliability? At least to 30 years!• suggest development/evaluation pathways for centennial modelling programs.

What is the decision relevant value-added of CMIP6 given CMIP5?How much will the probability of a big surprise decrease?

?What supports “return to skill” arguments in the climate case?

There is value in better probabilistic foresight on weeks to seasons+ in the current climate with which we familiar, and even more in a changed climate

A Modest Proposal: Minimalist Empirically-Adequate Climate Modelling


Moving Forward:Plausible Planets or Implausible Earths?

How can we best develop our models as the available computational power increases?

A) Simulate potentially real planets that get more and more Earth-like while omitting any Earth-relevant process for which the model cannot provide coherent physical drivers on Earth-like scales. (no suggestion of linear superposition intended!)

Does water vapour come after mountains?Does vegetation come after water vapour?Do we avoid the penguin effect? (until it is simulated realistically)

B) Via an hodgepodge of unphysical/unbiological simulations resembling no planet that could possibly exist, but “including” every phenomena we can think of that might be important (including penguins), and hoping the simulated planets will suddenly become Earth-like at some resolution in an ill-defined higgledy-piggledy way.

One might argue physical intuition is more effective in evaluating plausible planets, as there is physics to intuit in that case. (and at least a few examples.)


Thank you

What are Fair Odds?


Mature Probabilities P(x|I)A mature probability is not expected to change without additional observation or new theoretical insight. (An nontrivial change in I )

If the fidelity of a simulation model is constrained by technology (as when you know exactly what you would do with more compute power, and it is NOT to run massive ensembles/emulators), then probability distributions based on simulations from that model (or family of similar models) are not expected to be mature.

Rational action is constrained only by mature probabilities.

Model-based Probability can be used in creative ways (as data).

(?How might one use im-mature distributions in decision support?)(Generalised from IJ Good’s “Dynamic Probability”, as when output from a chess program must be used before the algorithm completes. )

IJ Good (~1979) Science


LA Smith, (2002) What Might We Learn from Climate Forecasts? Proc. National Acad. Sci. USA 4(99): 2487 -2492.Du, H. and Smith, L. A. (2012) Parameter estimation using ignorance Physical Review E 86, 016213Bröcker, J. and Smith, L. A. (2008) From Ensemble Forecasts to Predictive Distribution Functions Tellus A 60(4): 663Smith, LA and Stern, N (2011) Uncertainty in science and its role in climate policy Phil. Trans. R. Soc. A (2011), 369, 1-24

R Hagedorn and LA Smith (2009) Communicating the value of probabilistic forecasts with weather roulette. Meteori App 16 (2): 143K Judd, CA Reynolds, LA Smith & TE Rosmond (2008) The Geometry of Model Error. J of Atmos Sci 65 (6), 1749-1772.DA Stainforth, MR Allen, ER Tredger & LA Smith (2007) Confidence, uncertainty and decision-support relevance in climate predictions, Phil. Trans. R. Soc. A, 365, 2145-2161.

LA Smith (2006) Predictability past predictability present. Chapter 9 of Predictability of Weather and Climate (eds T. Palmer and R Hagedorn). Cambridge, UK. Cambridge University Press.

L A Smith and D A Stainforth (2012) Clarify the limits of climate models in Nature Vol. 489

D Orrell, LA Smith, T Palmer & J Barkmeijer (2001) Model Error in Weather Forecasting, Nonlinear Processes in Geophysics 8: 357

LA Smith (2000) 'Disentangling Uncertainty and Error: On the Predictability of Nonlinear Systems' in Nonlinear Dynamics and Statistics, ed. Alistair I Mees, Boston: Birkhauser, 31-64.

www2.lse.ac.uk/CATS/Publications/Leonard Smith Publications.aspx

http://www2.lse.ac.uk/CATS/Publications/Publications%20PDFs/LA-Smith,-(2002)-What-Might-We-Learn-from-Climate-Forecasts.pdf

http://www.lse.ac.uk/CATS/Publications/Publications%20Abstracts/Du&Smith-PhysicsReviewE-016213-2012.pdf

http://www.lse.ac.uk/CATS/Publications/Publications%20PDFs/74_Broecker_PDFs_tellus_2007.pdf

http://www2.lse.ac.uk/CATS/Publications/Publications%20PDFs/86_SmithStern_Uncertainty_2011.pdf

http://www2.lse.ac.uk/CATS/Publications/Publications%20PDFs/78_Weather_Roulette_t.pdf

http://www2.lse.ac.uk/CATS/Publications/Publications%20PDFs/77_Judd_GeomOfModelError_JAS.pdf

http://www2.lse.ac.uk/CATS/Publications/Publications%20PDFs/75_Stainforth_ConfidenceUncertaintyRelevance_2007.pdf

http://www2.lse.ac.uk/CATS/Publications/Publications%20PDFs/Stainforth&Smith_Nature_correspondence.pdf

http://www2.lse.ac.uk/CATS/Publications/Publications%20PDFs/45_ModelError_2001.pdf

http://www2.lse.ac.uk/CATS/Publications/Publications%20PDFs/40_Disentangling_2000.pdf

http://www2.lse.ac.uk/CATS/Publications/Leonard%20Smith%20Publications.aspx


Placing Consensus before Clarity?http://www.ipcc.ch/pdf/supporting-material/uncertainty-guidance-note.pdf

99 -10090 - 9966 - 9033 - 6610 - 33

1 - 10< 1

We can clarify the impact of alternative presentations.

Is it better to report diversity of opinion, or to obscure it?

http://www.ipcc.ch/pdf/supporting-material/uncertainty-guidance-note.pdf