20161007 - raveemismailarpcpresentation
TRANSCRIPT
FORECASTING POLITICALVIOLENCE FREQUENCY
AFIRST STRUCTURED EXPERT JUDGEMENTELICITATION IN (RE)INSURANCE
DR RAVEEM ISMAILSPECIALTY TREATY UNDERWRITER,ARIEL RE
[email protected] [email protected]@oxon.org
Overview:
Opinions&Judgements
There are no hard facts, just endless opinions. Every day, the news media deliver forecasts withoutreporting, or even asking, how good the forecasters really are. Every day, corporations andgovernments pay for forecasts that may be prescient or worthless or something in between. Andevery day, all of us - leaders of nations, corporate executives, investors, and voters - make criticaldecisions on the basis of forecasts whose quality is unknown.
Superforecasting: The Art & Science Of Prediction.
Tetlock & Gardner, 2015, Crown Publishers.
We put investor capital at risk in exchange for premium.
Decision-making should ideally arise from objective criteria: exhaustive data / sound physical principles.
Yet we often have to act in data-poor environments, where we rely heavily on Expert Judgement.
3
• A single expert’s opinion could be an outlier …… but consulting 10 experts will yield 10 answers.
Each answer is an unknowable function of that expert.• No method of selecting between judgements…
… decision makers often stick to what they know best.No indicators demanded of forecasting capability.
• How to makes sense of multiple opinions?Simple averaging: only limited gains.
Each expert weighted equally without regard for capability.Final answer may be worse than individual answers.
4
StructuredExpertJudgement(SEJ)
SEJ differs from, and extends, previous opinion pooling methods.1. Experts must be rated on previous performance.
• Ask set of seed questions, to which answer is already known.• Performance on seed questions ascertains expert’s weighting.
2. Ask the experts the target question.3. Weightings drawn from seed questions then used to combine experts’
judgement on target.This produces one outcome which truly combines different judgements in a
performance-based way, and is potentially better than any individual answer.• Seed question design is critical!
6
SEJ differs from, and extends, previous opinion pooling methods.1. Experts must be rated on previous performance.
• Ask set of seed questions, to which answer is already known.• Performance on seed questions ascertains expert’s weighting.
2. Ask the experts the target question.3. Weightings drawn from seed questions then used to combine experts’
judgement on target.This produces one outcome which truly combines different judgements in a
performance-based way, and is potentially better than any individual answer.• Seed question design is critical!
7
Seed Question A
Seed Question B
Seed Question C
Expert 1 Weighting 1
Expert 1 Weighting 1
Expert 2 Weighting 2
Expert 3 Weighting 3
Expert 4
Expert 5
Seed Questions
Weighting 4
Weighting 5
SEJ differs from, and extends, previous opinion pooling methods.1. Experts must be rated on previous performance.
• Ask set of seed questions, to which answer is already known.• Performance on seed questions ascertains expert’s weighting.
2. Ask experts the target question (the actual judgement being sought).3. Weightings drawn from seed questions then used to combine experts’
judgement on target question.This produces one outcome which truly combines different judgements in a
performance-based way, and is potentially better than any individual answer.• Seed question design is critical!
9
Target question
Weighted Judgement
Weighted Experts 1
2 34
5
AFirst(Re)insuranceElicitation:
FuturePoliticalViolenceFrequency
SEJ method: Cooke’s Classical Model. Two metrics requested from experts, within the range of possible values:• A 5% to 95% confidence interval.• A central median value.
11
Minimumvalue
Maximumvalue
LowerUncertaintyBound(5%)
UpperUncertaintyBound(95%)
CentralValue(50%)
These provide a measure of:
“Calibration” How accurate the expert is.
“Information” How tightly the expert gauges his/her own uncertainty.
12
Expert Calibration InformationEqualweights
Performanceweights
1 0.2295 1.864 0.1111 0.428
2 <0.0001 1.783 0.1111 <0.0001
3 0.002 2.04 0.1111 0.0041
4 0.2274 1.665 0.1111 0.3785
5 0.0002 2.153 0.1111 0.0006
6 <0.0001 3.01 0.1111 0.0002
7 <0.0001 1.505 0.1111 <0.0001
8 <0.0001 2.495 0.1111 <0.0001
9 0.0001 0.734 0.1111 <0.0001Equal -weightedcombination
0.6286 0.869
Performance-weightedcombination
0.5173 1.701
0.4242
0.7561
14
15
Conclusion
• Structured Expert Judgement is still judgement.
• But it is not pure guess work, it is:• a transparent method of pooling multiple opinions,• which are weighted according to comprehensible performance criteria,• those performance criteria are directly aligned to the actual judgements being sought.
• Where data and models are lacking, provides objective and auditable method of producing decision-making judgements and inputs to models.
• We have described a first SEJ in our area of interest, where the method identifies true expertise and outperforms uncalibrated methods.
• So not a magic bullet, but where applicable, it enhances decision-making and risk appraisal:Stochastic pricing, Reinsurance purchase, Capital Modelling, ILS, etc.
18
Questions?
19
Notes:• The full study, Structured Expert Judgement (SEJ) For Political
Violence In (Re)insurance, will be a forthcoming publication in a scientific journal, permanent URL until then: http://1drv.ms/1VZkuGh .
• Cooke Classical Method for elicitation of Structured Expert Judgement: Experts In Uncertainty, 1991, Oxford University Press.
• The ISCH EU COST Action IS1304 for Structured Expert Judgement aims to bridge the gap between scientific uncertainty and evidence-based decision making. The political violence elicitation referenced here took place in London in January 2016, kindly hosted by Dickie Whitaker and the Lighthill Risk Network, run by the COST Action’s Reinsurance Special Interest Group, chaired by Dr Raveem Ismail, with other principal investigators being Christoph Werner (Strathclyde University), and Professor Willy Aspinall (Bristol University).
