some science-related strategic challenges - model risk excerpt

8/13/2019 Some Science-related Strategic Challenges - Model Risk Excerpt

1/7

Some science-relatedstrategic challenges

A report for the UKGovernment Office forScience

Ariel Research Services February 2013

Written by Michael Reilly +44 (0)7986599791 [email protected] www.arielresearchservices.com

Prospero and Ariel by Steering for North 2012 All rights reserved

This report has been commissioned by the UK Government Office for Science. The views expressedin this report are not those of the UK Government and do not represent its policies.


2/7

Created by Ariel Research Services February 2013

2

1. Model risk what are the limits to simulation?

Summary

There are perils in using mathematical models to simulate what scientists describe as the

messiness of the real. Yet models of reality have been, and will be in the future, important tomajor policy decisions. Scientists employ models as part of their fundamental aim to betterunderstand the world. But although they are usually aware of the limits of simulation, there isa growing anxiety that models may present reliability, without truth. Decision-makers wantpolitical quality from model-based analysis; but they do not receive risk assessment withrationality. Studies of how decision-makers interact with models are surprisingly rare butsuggest human-computer interaction can produce successful outcomes. When boundariesbetween academia and practice become too porous, models can become engines, notcameras, with alarming results. Scientists and decision-makers use models differently andthere may be a requirement for honest brokers to mediate shared aims. The way forward is

a cultural shift to where decision-makers can observe and experiment, and, as aconsequence, experience that crucial component of science, adaptive learning.

The messiness of the real

The perils of using mathematical models to simulate what some scientists describe as themessiness of the real were exemplified by the collapse of the hedge fund Long TermCapital Management in 1998 and the investment bank Lehmann Brothers in 2007. Bothepisodes were marked by a failure of financial risk models but the ensuing global financialcrisis and the Great Recession that subsequently followed the demise of Lehman werecatastrophic. In both cases, models underestimated the impact and likelihood of extremeevents and their systemic consequences (Danielsson, 2008). In the aftermath of this mostrecent financial crisis, criminal prosecutions have been rare, complicated by differingexplanations of cause and effect, so much so that Andrew Lo, a prominent Professor ofFinance, has drawn an analogy to Akira Kurosawas classic film Rashomon with its multipleyet conflicting viewpoints of the same incident (Lo, 2012). By way of stark contrast, inLAquila in Italy six scientists at the National Commission for the Forecast and Prevention ofMajor Risks were summarily convicted of multiple manslaughters for providing "inaccurate,incomplete and contradictory" information in advance of a devastating earthquake in 2008(Johnston, 2012). In this instance, more than 5000 scientists were unified in agreement thatseismology has limited predictive power and that the case represents a far-reaching

miscarriage of justice (Leshner, 2010).

Decisions, decisions, decisions

Models of reality can be important to major policy decisions. The UK Department forTransport was rocked in 2012 when, under legal challenge by Virgin Trains, mistakes in theevaluation process of bids for the West Coast Mainline Rail Franchise were uncovered. Themodel used to calculate the capital required of bidders to mitigate insolvency risk was weak,its underlying assumptions were miscommunicated to bidders, and, in any event, thecalculation was subsequently delegated to a committee of high-level officials breaching

departmental protocol (Laidlaw, 2012). That bidders, facing uncertain fuel prices and thevagaries of future demand, were also expected to accurately project revenue growth over 15


3/7


3

years, was met with derision (The Economist, 2012). Estimates of global infrastructureinvestment needed over next 20 years range from an incredible $40-$50t (Doshi, et al.,2007). If such estimates can be trusted, then the magnitude of model risk could increasesubstantially.

How do scientists use models?

Scientists employ models as part of their fundamental aim to better understand the world. Itis easy to forget in an age of supercomputing that early models were physical rather thanabstract for example orreries constructed to simulate the orbits of the planets. Nowadays,mathematical models continue to be used to synthesise data, stimulate observation andexperimentation, and simulate the future (Oresekes, 2007). Many scientific experts,however, are aware of the limitations of simulation; and predicting the future can actually bedamaging to the fundamental aim of science. In rare circumstances where the system isdescribed by a small number of measurable parameters or is highly repetitive and therefore

conducive to adaptive learning prediction may be possible (Oresekes, 2007). Even in theseinstances an erroneous conceptualisation of the system can produce a false positive. Aphilosopher of science Eric Winsberg, who argues that modelling possesses a differentepistemology to science, calls this reliability without truth (Winsberg, 2006).

Reliability without truth is a growing anxiety in science. Professor Sherry Turkle, apsychologist, observed the impact of computing on science at MIT. In a study of the 1980sand the 2000s, she documents evidence of deterioration in the scientific identity of youngscientists who are drunk with code and indifferent to the inherent limitations of thecomputer as a scientific instrument (Turkle, 2009). But she finds that simulation has alsofreed students and researchers from repetitive processes, stimulating observation andexperimentation, and supporting adaptive learning. The price to be paid for this immersion isdiminishing scepticism. For one MIT physicist:

My students know more and more about computer reality, but less and less about the realworld. And they no longer even really know about computer reality because the simulationshave become so complex that people dont build them any more. They just buy them andcant get beneath the surface. If the assumptions behind some simulation were flawed, mystudents wouldnt even know where or how to look for the problem. (Turkle, 2009)

Andrew Lo, thus, did not err in referencing Rashomon. But the ubiquity of computersimulation and visualisation may be inevitable. Science is increasingly grappling withcomplex phenomena for which explanatory models are considerably more advanced thanrigorous theories (Jogalekar, 2012). Professor Jerry Ravetz at Oxford University calls thispost-normal science and it has serious implications for the legitimacy of decision-making(Funtowicz & Ravetz, 1994).

How do decision-makers use models?

Professor Colin Thirtle, an agricultural scientist at Imperial College in London, is fond ofquoting an anecdote about President Lyndon Johnson. When confronted with a range ofoutcomes, Johnson remarked give me a number, ranges are for cattle. Evidence suggests

that societies may be pre-disposed to leaders who exhibit over-confident determinism(Menand, 2005). It would be nave to underestimate the political quality required of model-


4/7


4

based analysis or, correspondingly, to lower ones guard to the dismal record of policy-drivenprediction. The tragic aftermath of the earthquake in LAquila is one of many examplesillustrating the pitfalls of simulation for decision-making. In the Spring of 1997, the town ofGrand Forks in North Dakota received two flood outlooks from the National WeatherService for the peak of the Red River. These scenarios, based on different assumptions,were of 47.5 and 49 feet. Although the outlooks were qualified for uncertainty, when thetown was inundated at river levels of 54 feet, causing $1-2bn in property damage, theNational Weather Service was blamed by many for the disaster. The mayor of Grand Forksclaimed they blew it big (Pielke, 1999). A further question is, then, how do decision-makersconsume model-based analysis? Behavioural psychologists are discovering that riskassessments are not received with rationality (Kahneman, 2011).

Studies of how decision-makers interact with models are surprisingly rare. Professor EarlHunt at the University of Washington observed naval weather forecasters working with avisual interface that presented multiple simulations in order to mitigate model risk (Hunt,

2003). He found that forecasters tend to rely on a favourite model regardless of its historyand then adjust its outputs based on observations and satellite patterns. The results of thisadaptive human-computer interaction were generally good and superior to that of the modelalone. This outcome demonstrates a lesson learned from a famous interaction between GaryKlein and Daniel Kahneman in 2009 (Kahneman & Klein, 2009). Although previously atloggerheads over the merits or otherwise of intuitive expert judgement, they concluded thatexpert judgement is valid in an environment 1) sufficiently regular to be predictable, and, 2)where an opportunity exists to learn these regularities through prolonged practice. Incontrast, in low validity environments characterised by significant uncertainty andunpredictability, simple algorithms may be superior to human judgement. The committee ofhigh-level officials that breached Department for Transport protocol by over-ruling theadmittedly flawed capital calculation process for the West Coast Mainline Rail Franchisewere also dismissing the opinion of arguably two of the worlds best psychologists.

The merit of a simple algorithm for decision-making in low-validity environments is notmerely the whim of a psychologist. There is a trade-off in model construction between detailand error (Oresekes, 2007). Whilst scientists are eager to observe the interplay of importantvariables, too much detail will result - through the accumulation of errors - in a loss ofstatistical power. The judgement on how open a model is and how to close it usingassumptions is difficult and depends on the aims of the exercise. Separating out thebehaviour of important variables in a controlled environment is what makes randomised

controlled trials so alluring - and elusive - to social scientists.

Engines, not cameras

Professor Donald MacKenzie of Edinburgh University, an observer of the sociology offinance, published in 2006 a seminal analysis of the interweaving of innovation in financialeconomics and developments in financial markets, which in many ways anticipated thefinancial crisis that occurred soon afterwards (MacKenzie, 2006). He uncovered anepistemic culture in academia where models were considered a source of knowledge butwith deep ambivalence towards their realism. Nevertheless, the boundary betweenacademia and practice was highly porous. Again, scepticism was lost in immersion.MacKenzie states that finance theory became thoroughly embedded in financial markets in


5/7


5

technical, linguistic, and legitimatory ways. In a powerful metaphor he concludes thatfinancial models were engines, not cameras. It is not a huge leap to suppose that modelsused to determine important policy decisions like the West Coast Rail Franchise havebecome engines too. In neuroscience, knowledge-poor fMRI visualisations allegedly biasopinion more favourably towards research articles (McCabe & Castel, 2008). And the stakesmay just be getting higher. If, as Tim Berners-Lee contends, data is the new raw material ofthe 21st century then this performativity of models could become more even moresignificant. Smart cities, for example, could prove to be remarkably dumb (Rooney, 2012).

Is there a way forward?

For some experts worried about too much immersion and not enough scepticism, theallegory of Platos Cave is instructive. In the cave, a group of prisoners can only observe theoutside world through shadows cast on a blank wall. The shadows become their reality. ForPlato, it was the role of philosophers to free the world from this illusion, but those

discontented by simulation feel a similar responsibility (Turkle, 2009). Scientists anddecision-makers use models in different ways. They also want different things from models predominantly analytical and methodological quality for the former and political quality for thelatter. These differences are not irreconcilable but there is an obvious need for an honestbroker to help free stakeholders from misconceptions and mediate shared aims.

An interesting case study of human-computer interaction might be the Threshold 21 model ofthe Millennium Institute in Washington DC (Millennium Institute, 2012). This model waspurposely designed for human-computer interaction in the service of long-term nationaldevelopment planning. Model detail is eschewed in favour of a transparent process whereby- with the aid of mediation - the decision-maker can actually understand how inputs andassumptions affect the interplay of important variables. The black box environment isunlocked.

What is clear is that model-based analysis should not be the sole evidence underpinning adecision. The validity of the environment in which the decision is being made may call for themental models of intuitive expert judgement (Kahneman & Klein, 2009). For shorter-termdecisions improved monitoring of the policy environment may be a better investment than animprovement in model detail (Oresekes, 2007). For longer-term decisions such asinfrastructure investment, the decision process may have to be designed to fit an uncertainand unpredictable policy environment (Harford, 2012). Above all else, decision-makers

should experience models in ways that are based on good science. The way forward is acultural shift to where decision-makers can observe and experiment, and, as aconsequence, experience that crucial component of science, adaptive learning.


6/7


6

ReferencesDanielsson, J., 2008. Blame the models. Journal of Financial Stability, Volume 4, pp. 321-328.

Doshi, V., Schulman, G. & Gabaldon, D., 2007. Lights! Water! Motion!. Booz & Co strategy +

business, Volume 46, pp. 1-16.

Funtowicz, S. & Ravetz, J., 1994. Uncertainty, complexity and post-normal science.Environmental toxicology and chemistry, Volume 13, pp. 1881-1885.

Harford, T., 2012. Adapt: why success always starts with failure. London: Little Brown.

Hunt, E., 2003. Human-computer decision making: the view from psychology. Washington:University of Washington.

Jogalekar, A., 2012. Theories, models and the future of science. [Online]Available at: http://blogs.scientificamerican.com/the-curious-wavefunction/2012/09/05/theories-models-and-the-future-of-science/

Johnston, A., 2012. L'Aquila quake: Italian scientists guilty of manslaughter. [Online]Available at: http://www.bbc.co.uk/news/world-europe-20025626

Kahneman, D., 2011. Thinking Fast and Slow. New York: Fararr, Straus and Giroux.

Kahneman, D. & Klein, G., 2009. Conditions for intuitive expertise: a failure to disagree.American Psychologist, Volume 64, pp. 515-526.

Laidlaw, S., 2012. Report of the Laidlaw Inquiry: Inquiry into the lessons learned for the

Department for Transport from the InterCity West Coast Competition, 2012: HMSO.

Leshner, A., 2010. AAAS Protests Charges Against Scientists Who Failed to PredictEarthquake. [Online]Available at: http://www.aaas.org/news/releases/2010/0630italy_letter.shtml

Lo, A., 2012. Reading about the financial crisis. Journal of Economic Literature, Volume 50,pp. 151-178.

MacKenzie, D., 2006. An engine not a camera: how financial models shape markets.London: The MIT Press.

McCabe, D. & Castel, A., 2008. Seeing is believing: the effect of brain images on judgements of scientific reasoning. Cognition, Volume 107, pp. 343-352.

Menand, L., 2005. Everybody's an expert. [Online]Available at: http://www.newyorker.com/archive/2005/12/05/051205crbo_books1?pri

Millennium Institute, 2012. Threshold 21 Model. [Online]Available at: http://www.millennium-institute.org/integrated_planning/tools/T21/

Oresekes, N., 2007. The role of quantitative models in science. In: s.l.:s.n.

Pielke, R., 1999. Who decides? Forecasts and responsibilities in the 1997 Red River Flood.Applied Behavioural Science Review , Volume 7, pp. 83-101.


7/7


7

Rooney, B., 2012. 'Smart city' planning needs the right balance. [Online]Available at:http://online.wsj.com/article/SB10000872396390443916104578020411910063242.html

The Economist, 2012. Train franchises: Wrong Track. [Online]

Available at: http://www.economist.com/node/21564272

Turkle, S., 2009. Simulation and its discontents. London: The MIT Press.

Winsberg, E., 2006. Models of success versus the success of models: reliability withouttruth. Synthese, Volume 152, pp. 1-19.

some science-related strategic challenges - model risk excerpt

Documents