Bad assumptions or bad luck: Why natural hazard maps (forecasts, warnings, etc)often fail and what to do about itSeth Stein, Northwestern University Robert Geller, University of Tokyo Mian Liu, University of MissouriTohoku, Japan March 11, 2011 M 9.1NY TimesCNN

ChallengesMitigation is insurance against possible disaster, with present costs & potential benefitsUse science, engineering & social science to help society develop strategies that make sense relative to alternative uses of resources (steel in schools versus hiring teachers)Process hampered by limited scientific knowledge about fundamental processes & thus future events (Laquila earthquake)The earth is very complicated and regularly reminds us of the need for humility in the face of the complexities of nature Whenever I hear everybody knows that I wonder how do they know that, and is it true (D. Jackson)

"NASA owes it to the citizens from whom it asks support to be frank, honest, and informative, so these citizens can make the wisest decisions for the use of their limited resources.

Richard Feynman's (1988) report after the loss of the space shuttle Challenger

Tools in preparing for natural disasters include - Long term forecasts: 1-2500 yr (earthquakes), 100 yr (climate change), 1-10 yr (hurricane, volcano)- Short term predictions: days (hurricanes), days to months (volcano), hours (tornado)- Real time warnings: (hours to minutes) tsunami, earthquake shaking, hurricane, tornado, flood

Sometimes these work, sometimes they fail

FAILURESFalse negative - unpredicted hazardLoss of life & property

False positive - overpredicted hazardWasted resources, public loses confidence

Authorities typically ignore, deny, excuse, or minimize failureMore useful to analyze failures to improve future performance

We got a hell of a beating. We got run out of Burma and it is humiliating as hell. I think we ought to find out what caused it, go back, and retake it."General Joseph Stilwell U.S. Army, WWIII am not discouraged, because every wrong attempt discarded is another step forward.Thomas Edison

2008:

Hurricane Ike predicted to hit Miami

Ikes actual track

Ikepredicted to bring certain death

Actual deaths:< 50 of 40,000

Error 800x

If it had been a weekday,Major cost

K. Emanuel CNN 8/26/11NYT 8/28/11

Economic loss ?What if weekday?

Science News6/15/91The local economy collapsed, said Glenn Thompson, Mammoth Lakes' town manager. Housing prices fell 40 percent overnight. In the next few years, dozens of businesses closed, new shopping centers stood empty and townspeople left to seek jobs elsewhere. (NYT 9/11/90)

Predicted disaster probabilities are often very inaccurateP(sinking) = 0P(loss) = 1/100,000

Systematic errors often exceed measurement errors

Uncertainties are hard to assess and generally underestimatedUnderestimated uncertainty and bias (bandwagon effect) in measured speed of light1875-1960

Number of human chromosome pairs

1921-1955: 24 Now: 23

CDC reported "strong possibility" of epidemic. HEW thought "chances seem to be 1 in 2 and virus will kill one million Americans in 1976."

President Ford launched program to vaccinate entire population despite critics reservations

40 million vaccinated at cost of millions of dollars before program suspended due to reactions to vaccine

About 500 people had serious reactions and 25 died, compared to one person who died from swine flu NEGLECTING UNCERTAINTY OVERESTIMATES HAZARD1976 SWINE FLU APORKALPSE

Much ado made that on January 1, 2000 computer systems would fail, because dates used only two digits U.S. & other governments established major programs

Estimated $300 billion spent on preparationsHAZARD OVERESTIMATED: Y2KFew major problems occurred, even among businesses and countries who made little or no preparation

Japan seemed ideal for hazard mapping

Fast moving (80 mm/yr ) & seismically very active plate boundary with good instrumentation & long seismic history

But: 2011 M 9.1 Tohoku, 1995 Kobe M 7.3 & others in areas mapped as low hazard

In contrast: map assumed high hazard in Tokai gapGeller 2011

Hazard maps fail because of - bad physics (incorrect description of earthquake processes)bad assumptions (mapmakers choice of poorly known parameters) bad data (lacking, incomplete, or underappreciated) bad luck (low probability events)and combinations of these

Expected Earthquake Sources 50 to 150 km segments M7.5 to 8.2(Headquarters for Earthquake Research Promotion) Off Sanriku-oki North ~M8 0.2 to 10%Off Sanriku-oki Central ~M7.7 80 to 90%Off Fukushima ~M7.4 7% Off Ibaraki ~M6.7 M7.2 90%Detailed model of segments with 30 year probabilitiesSanriku to Boso M8.2 (plate boundary)20%Sanriku to Boso M8.2 (Intraplate)4-7%

Off Miyagi ~M7.5 > 90%J. MoriAssumption:No M > 8.2

Giant earthquake broke all of the segments2011 Tohoku Earthquake 450 km long fault, M 9.1 (Aftershock map from USGS)J. MoriExpected Earthquake Sources 50 to 150 km segments M7.5 to 8.2(Headquarters for Earthquake Research Promotion)

Tsunami runup approximately twice fault slip (Plafker, Okal & Synolakis 2004) M9 generates much larger tsunamiPlanning assumed maximum magnitude 8 Seawalls 5-10 m highCNNNYTStein & Okal, 2011

http://www.coastal.jp/tsunami2011/index.php?FrontPagehttp://www.geol.tsukuba.ac.jp/~yagi-y/EQ/Tohoku/Tsunami radiates energy perpendicular to faultThus largest landward of highest slip patches

Didnt consider historical record of large tsunamisNYT 4/20/11

Lack of M9s in record seemed consistent with model that M9s only occur where lithosphere younger than 80 Myr subducts faster than 50 mm/yr (Ruff and Kanamori, 1980) Disproved by Sumatra 2004 M9.3 and dataset reanalysis (Stein & Okal, 2007)Short record at most SZs didnt include larger multisegment rupturesStein & Okal, 2011

Most plate motion assumed aseismic, as in Kuriles: Same plate pair, further north - expected only M8

Since 1952 largest thrust earthquakes M8 with average slip 2-3 m.

Previous earthquake sequence ~100 years earlier, so inferred seismic slip rate 2-3 cm/yr

This is 1/3 of plate motion predicted from relative motion models, so the remaining 2/3 was assumed to occur aseismically. Kanamori, 1977

Nature, 2003By 2003, recognize larger (M9?), rarer, multisegment rupturesthat account for much of the proposed aseismic slip

80 mm/yr convergence:

M 8: 2.5 m / 100 yr = 25 mm/yr + M 9: 10 m / 500 yr = 20 mm/yr

NY Times 3/21/11

Scientists will be able to predict earthquakes in five years.Louis Pakiser U.S. Geological Survey, 1971

We have the technology to develop a reliable prediction system already in hand.Alan Cranston, U.S. senator, 1973

The age of earthquake prediction is upon usU.S. Geological Survey, 19751970s optimismSimilar in Japan, China, USSR

Failed prediction: 1975 Palmdale BulgeUSGS director McKelvey stated that a great earthquake would occur in the area ... possibly within the next decade that might cause up to 12,000 deaths, 48,000 serious injuries, 40,000 damaged buildings, and up to $25 billion in damage.

Systematic uncertainty of leveling data underestimated SAF

In 1985, USGS predicted next with 95% confidence by 1993Occurred in 2004 (16 years late)M 5-6 earthquakes about every 22 years: 1857, 1881, 1901, 1922, 1934, and 1966Discounting misfit of 1934 quake predicted higher confidenceFailed prediction: Parkfield

2001 hazard maphttp://www.oas.org/cdmp/document/seismap/haiti_dr.htm2010 M7 earthquake shaking much greater than predicted for next 500 years6 mm/yr fault motion

Accuracy of hazard map prediction depends on accuracy of answers assumed to hierarchy of four basic questions

Where will large earthquakes occur?

When will they occur?

How large will they be?

How strong will their shaking be?Uncertainty & map failure result because these are often poorly known

Where do we expect earthquakes?Can useEarthquake historyPlate motionsGeologyGPSOften, we have to chose which to useDifferent choices lead to different predicted hazards

GSHAP 1999NUVEL-1Argus, Gordon, DeMets & Stein, 1989Swafford & Stein, 2007Slow plate boundary

Africa-Eurasia convergence rate varies smoothly (5 mm/yr)

20042003Swafford & Stein, 2007GSHAP 1999NUVEL-1Argus, Gordon, DeMets & Stein, 1989M 6.4M 6.3Slow plate boundary

Africa-Eurasia convergence rate varies smoothly (5 mm/yr)

USGS2008 Wenchuan earthquake (Mw 7.9) was not expected: map showed low hazard

Hazard map - assumed steady state - relied on lack of recent seismicityDidnt use GPS data showing 1-2 mm/yrM. Liu

Long record needed to see real hazardSwafford & Stein, 20071933 M 7.31929 M 7.2

Our glacial loading model suggests that earthquakes may occur anywhere along the rifted margin which has been glaciated.Stein et al., 19791985Concentrated hazard bull's-eyes at historic earthquake sites2005Diffuse hazard along marginGSCMap depends greatly on assumptions & thus has large uncertainty

Peak Ground Acceleration10% probability of exceedance in 50 years(once in 500 yr)GSHAP (1999)Present StudyHUNGARY: ALTERNATIVE HAZARD MAPSConcentrated hazard inferred from historic seismicity aloneDiffuse hazard inferred incorporating geologyToth et al., 2004

Plate Boundary EarthquakesMajor fault loaded rapidly at constant rate Earthquakes spatially focused & temporally quasi-periodicPast is fair predictorIntraplate Earthquakes

Tectonic loading collectively accommodated by a complex system of interacting faultsLoading rate on a given fault is slow & may not be constantEarthquakes can cluster on a fault for a while then shiftPast can be poor predictorStein, Liu & Wang 2009

New Madrid 1991: because paleoseismology shows large events in 900 & 1450 AD, like those of 1811-12 GPS studies started, expecting to find strain accumulating consistent with large (~M7) events ~500 years apartSurprising result

Science, April 1999Little or no motion!Seismicity migratesRecent cluster transient, possibly endingHazard overestimated

Large continental interior earthquakes reactivate ancient faults geological studies indicate that earthquakes on these faults tend to be temporally clustered and that recurrence intervals are on the order of tens of thousands of years or more. (Crone et al., 2003)Similar behavior in other continental interiors

OrdosPlateauShanxi GrabenBohai BayBeijing1303 HongtongM 8.0Liu, Stein & Wang 2011Weihi rift

OrdosPlateauShanxi GrabenBohai BayBeijing1556 HuaxianM 8.3Weihi riftLiu, Stein & Wang 2011

OrdosPlateauShanxi GrabenBohai BayBeijing1668 TanchengM 8.5Weihi riftLiu, Stein & Wang 2011

OrdosPlateauShanxi GrabenBohai BayBeijing1679 SanheM 8.0Weihi riftLiu, Stein & Wang 2011

OrdosPlateauShanxi GrabenBohai BayBeijing1966 XingtaiM 7.21976 TangshanM 7.81975 HaichengM 7.3Weihi riftLiu, Stein & Wang 2011

No large (M>7) events ruptured the same fault segment twice in past 2000 yearsIn past 200 years, quakes migrated from Shanxi Graben to N. China PlainShanxi GrabenWeihi rift

Maps are like Whack-a-mole - you wait for the mole to come up where it went down, but its likely to pop up somewhere else.

When do we expect earthquakes?

When we have a long history, we can estimate the average recurrence time - but theres a lot of scatterWhen we have a short history, we estimate the recurrence time of large earthquakes from small ones, but this can be biasedIn either case, we have to assume either that the probability of large earthquakes stays constant with time, or that it changes

Different choices lead to different predicted hazards

EARTHQUAKE RECURRENCE IS HIGHLY VARIABLEM>7 mean 132 yr s 105 yr Estimated probability in 30 yrs 7-51%Sieh et al., 1989Extend earthquake history with paleoseismology

When we have a long history, we can estimate the average recurrence time - but theres a lot of scatterMean 132 105Mean 180 72We can describe these using various distributions - Gaussian, log-normal, Poisson but its not clear that one is better than another

Gutenberg-Richter relationshiplog10 N = a -b MN = number of earthquakes occurring M a = activity rate (y-intercept) b = slope M = MagnitudeWhen we have a short history, we estimate the size & recurrence time of large earthquakes from small ones, but this can be biased

GUTENBERG-RICHTER RELATIONSHIP: INDIVIDUAL FAULTSWasatch Basel, Switzerlandpaleoseismic datainstrumental dataYoungs & Coppersmith, 1985Meghraoui et al., 2001paleoseismic datahistorical dataLargest events deviate in either direction, often when different data mismatchWhen more frequent than expected termed characteristic earthquakes. Alternative are uncharacteristic earthquakesThese - at least in some cases - are artifacts of short history that overpredict or underpredict hazardCharacteristicUncharacteristic

POSSIBLE BIASES IN ESTIMATING THE MAGNITUDE AND RECURRENCE TIME OF LARGE EARTHQUAKES FROM THE RATE OF SMALL ONESUndersampling: record comparable to or shorter than mean recurrence - Usually find too-short recurrence time. Can also miss largest events Direct paleoseismic study: Magnitude overestimated, recurrence underestimatedEvents missed, recurrence overestimatedEarthquake RateStein & Newman, 2004CHARACTERISTICUNCHARACTERISTIC

SIMULATIONS Short history: often miss largest earthquake or find a too-short recurrence time

10,000 synthetic earthquake histories for G-R relation with slope b=1Gaussian recurrence times for M> 5, 6, 7Various history lengths given in terms of Tav, mean recurrence for M>7Stein & Newman, 2004

Long history: often still find too-short or too-long recurrence timeStein & Newman, 2004

RESULTS VARY WITH AREA SAMPLEDStein et al., 2005Increasing area around main fault adds more small earthquakesCharacteristic earthquakes on Wasatch fault (Chang and Smith, 2002), but not in entire Wasatch front (data from Pechmann and Arabasz, 1995)

Time dependent predicts lower until ~2/3 mean recurrenceResults depend on both model choice & assumed mean recurrenceAssumed probability of large earthquake & thus hazard depend on recurrence model & position in earthquake cycleHebden & Stein, 2008Not clear which model works best where

CHARLESTON2% in 50 yr (1/2500 yr)Hebden & Stein, 2008At present, time dependent predicts ~50% lower hazard

Still less in 2250

California Time-dependant probabilities

Increased on southern San Andreas

What will happen in large earthquakes?

Major unknowns are magnitude of the earthquake and the ground shaking it will produceTradeoff between these two parametersDifferent choices lead to different predicted hazards

EFFECTS OF ASSUMED GROUND MOTION MODEL

Effect as large as one magnitude unit

Frankel model predicts significantly greater shaking for M >7

Frankel M 7 similar to other models M 8Newman et al., 2001

Newman et al., 2001PREDICTED HAZARD DEPENDS GREATLY ON - Assumed maximum magnitude of largest eventsAssumed ground motion modelNeither are known since large earthquakes rare180%275%

What to doContinue research on fundamental scientific questionsRealistically assess uncertainties stemming from current limited knowledge and present them candidly to allow users to decide how much credence to place in mapsDevelop methods to objectively test hazard maps and thus guide future improvements

Global warming forecasts present uncertainties by showing factor of 3 range of model predictionsThe AOGCMs cannot sample the full range of possible warming, in particular because they do not include uncertainties in the carbon cycle. In addition to the range derived directly from the AR4 multi-model ensemble, Figure 10.29 depicts additional uncertainty estimates obtained from published probabilistic methods using different types of models and observational constraints: the MAGICC SCM and the BERN2.5CC coupled climate-carbon cycle EMIC tuned to different climate sensitivities and carbon cycle settings, and the C4MIP coupled climate-carbon cycle models. Based on these results, the future increase in global mean temperature is likely to fall within 40 to +60% of the multi-model AOGCM mean warming simulated for each scenario. This range results from an expert judgement of the multiple lines of evidence presented in Figure 10.29, and assumes that the models approximately capture the range of uncertainties in the carbon cycle. The range is well constrained at the lower bound since climate sensitivity is better constrained at the low end (see Box 10.2), and carbon cycle uncertainty only weakly affects the lower bound. The upper bound is less certain as there is more variation across the different models and methods, partly because carbon cycle feedback uncertainties are greater with larger warming. IPCC 2007Warming by 2099

In addition to comparing maps, comparing model predictions shows the large uncertainties resulting from different assumptions

Shows contributions to logic tree before subjective weighting

Testing analogy: evidence-based medicine objectively evaluates widely used treatments

Although more than 650,000 arthroscopic knee surgeries at a cost of roughly $5,000 each were being performed each year, a controlled experiment showed that "the outcomes were no better than a placebo procedure."

Bad luck or bad map?

One test is to compare maximum acceleration observed over the years to that predicted by both map and null hypotheses.

A simple null hypothesis is regionally uniformly distributed hazard.

Japanese map seems to be doing worse than this null hypothesis.Geller 2011Need objective criteria to test maps by comparison to what happened after they were published.

Some testing challengesShort time record: can in some cases be worked around. For example, North China record probably has almost or all M7s in 2000 years. Paleoseismology can go back even further, with higher probability of missing some. Subjective nature of hazard mapping, resulting from need to chose faults, maximum magnitude, recurrence model, and ground motion model. This precludes the traditional method of developing a model from the first part of a time series and testing how well it does in the later part. That works if the model is "automatically" generated by some rules (e.g. least squares, etc). In the earthquake case, this can't be done easily because we know what happens in the later part of the series.

3) Biases due to new maps made after a large earthquake that earlier maps missed.

Frankel et al, 2010Before 2010 Haiti M7After 2010 Haiti M74X

A posteriori changes to a model are "Texas sharpshooting: shoot at the barn and then draw circles around the bullet holes.

4) Overparameterized model (overfit data):

Given a trend with scatter, fitting a higher order polynomial can give a better fit to the past data but a worse fit to future data

Analogously, a seismic hazard map fit to details of past earthquakes could be a worse predictor of futureones than a less detailed map

How much detail is useful?Linear fitQuadratic fit

Summary

- Hazard maps depend dramatically on unknown and difficult-to-assess parameters and hence on the mapmakers preconceptions thus have large uncertainties that are generally underestimated and not communicated to public sometimes either underpredict hazard in areas where large earthquakes occur or overpredict hazardWithout objective testing, maps wont improve & seismology will keep having to explain away embarrassing failures

Challenge: Users Want Predictions

Future Nobel Prize winner Kenneth Arrow served as a military weather forecaster. As he described,

my colleagues had the responsibility of preparing long-range weather forecasts, i.e., for the following month. The statisticians among us subjected these forecasts to verification and found they differed in no way from chance. The forecasters themselves were convinced and requested that the forecasts be discontinued.

The reply read approximately: "The commanding general is well aware that the forecasts are no good. However, he needs them for planning purposes."

Gardner, D., Future Babble: Why Expert Predictions Fail - and Why We Believe Them Anyway, 2010

********************************