challenges in validating catastrophe models
DESCRIPTION
Challenges in Validating Catastrophe Models. Dr. Paul Rockett . Casualty Actuaries in Europe, 31 May 2013. Agenda. Introduction Validation tools Validating catastrophe models The bigger picture Summary. Introduction. What are catastrophe models?. - PowerPoint PPT PresentationTRANSCRIPT
Challenges in Validating Catastrophe ModelsDr. Paul Rockett
Casualty Actuaries in Europe, 31 May 2013
31 May 2013Page 2
Agenda
► Introduction► Validation tools► Validating catastrophe models► The bigger picture► Summary
Challenges in Validating Catastrophe Models
31 May 2013Page 3
Introduction
Challenges in Validating Catastrophe Models
► Tools designed to evaluate the risk profile of a risk or set of risks exposed to catastrophe events
31 May 2013Page 4
What are catastrophe models?
Challenges in Validating Catastrophe Models
Event Set Module
Hazard Module
Vulnerability Module
Financial Analysis Module
Exposure Module
IntensityLo
ss R
atio
Event Set Module
Hazard Module
Exposure Module
Vulnerability Module
Financial Analysis Module0
0.01
0.02
0.03
0.04
0.05
0.06
0 0.2 0.4 0.6 0.8 1
Prob
abili
ty D
ensit
y
Damage Ratio
Damage Distribution
Insurance Cover
Over Limit
Deductible
► They combine the frequency of catastrophe events with their severity to produce annual exceedance probability (EP) curves
► EP Curves can be thought of like VAR. SII SCR is a VAR measure corresponding to the 99.5% conf level of non-exceedance in 1-year
Convolution
31 May 2013Page 5
What are catastrophe models?
Challenges in Validating Catastrophe Models
31 May 2013Page 6
What do they cover?
► There are catastrophe models covering► Earthquakes► Windstorms, Tropical Cyclones► Storm Surge, River Floods, ...► Severe Convective Storm (Tornado, Hail, ...)► Bushfire, wild fire, ...
► And man-made catastrophes► Terrorism, Explosion, Conflagration► Pandemic
► Most important models for the insurance industry are US Hurricane, US Earthquake and Europe Windstorm
► But it varies by company
Challenges in Validating Catastrophe Models
31 May 2013Page 7
What are catastrophe models used for?
Challenges in Validating Catastrophe Models
Catastrophe Models
Capital model inputs
Reinsurance decision
making
Pricing risksAccumulation management
Estimate post event loss
size
AIR
EQECAT
RMS
Brokers
Consultants
Marketplace – e.g.
OASIS LMF
Who builds catastrophe models?
InsurersCat
Modelling Companies
Reinsurers Others
31 May 2013Page 8 Challenges in Validating Catastrophe Models
Why validate catastrophe models?
► Extreme events and subsequent losses e.g.► Hurricane Katrina► Hurricane Ike► Christchurch Earthquakes
Solvency II directive:► Article 124
(Validation Standards) ► Article 229 TSIM18
(Validation Process)
► Catastrophe models can drive the Solvency Capital Requirement
► Limited data► Many assumptions
UncertaintyMaterial
Why Validate?
Regulatory pressureHistory
31 May 2013Page 9 Challenges in Validating Catastrophe Models
Validation Tools
31 May 2013Page 10 Challenges in Validating Catastrophe Models
Solvency II Validation tools Tools for use by all firms (Article 230 TSIM19)
Purpose: Modelling and
estimation errors, Any model
weaknesses
Applications Back-testing Goodness of fit Dependency
assumptions Supplement with
qualitative analysis
Purpose: Impact of single or
multiple events
Applications Challenge results
and parameters Dependencies and
tails of distributions Understanding of
the risk profile Supports capital
allocation decisions Exceptional but
plausible large-loss events
Reverse stress testing
Limitations of the model
Purpose: Are capital
requirements produced robust?
Applications Sensitivity of
results to changes in the key underlying assumptions
Stability of the model
Results against experience Model robustness Stress & scenario
testing
Purpose: Are risks covered
by the model complete?
Explain Profits and Losses
Applications Explains actual
results using the model.
Assess level of unexplained profits and ability to reflect risk profile of the business
Profit & loss attribution
31 May 2013Page 11 Challenges in Validating Catastrophe Models
Solvency II Validation tools: Further (optional) tools set out by EIOPA
Purpose: Reinforce the
appropriateness of Pillar I and II results
ApplicationsConsider results of peers to investigate
whether alternative modelling approaches could improve quality
Note differences in company risk profiles
Purpose: Improve
comparisons between different models or versions
ApplicationsHypothetical portfolios of assets and/or liabilities chosen by the
company to analyse outcomes from different versions of internal models
Care needs to be taken to avoid creating systemic risk
Purpose: Analyse how the
results of the model have changed
Applications Reconciles results Validates that the
model is working as expected
Benchmarking Analysis of change Hypothetical portfolio
Purpose: Additional
qualitative validation tools
ApplicationsUse test Using the model in
BAU demonstrates sufficient level of comfort that the results are appropriate for use
Qualitative reviewWith dialogue and review, gain comfort: Theoretical basis of
the internal model Any part of the
internal model
Qualitative review
31 May 2013Page 12 Challenges in Validating Catastrophe Models
Validating Catastrophe Models
31 May 2013Page 13 Challenges in Validating Catastrophe Models
► Each component needs validation
Validating catastrophe models
► Main focus is on model output
Event Set
Hazard
Vulnerability
Financial Analysis
Exposure DataEP Curves
ELTsExposure
Input Cat Modules Output
31 May 2013Page 14 Challenges in Validating Catastrophe Models
► Key Aspects
► Typical ApproachQualitative review inc. data audit
► Validation ChallengeEnsuring processes in place track all the exposure information, and that it passes quality control checks before modelling
Exposure data
Geospatial Resolution
Risk Attributes
Valuation Data Heuristics
Event Set
Hazard
Vulnerability
Financial
Exposure Data
EP Curves
ELTs
Exposure
O
utpu
t
C
at M
odul
es
Inpu
t
31 May 2013Page 15 Challenges in Validating Catastrophe Models
► Key Aspects
► Typical ApproachBack-testing, Sensitivity Testing, Benchmarking
► Validation ChallengesNeed an understanding of the physical mechanism to define suitable tests
Event set module
Event Frequency
Clustering of Events
Event Characteristics
Event Set
Hazard
Vulnerability
Financial
Exposure Data
EP Curves
ELTs
Exposure
O
utpu
t
C
at M
odul
es
Inpu
t
31 May 2013Page 16 Challenges in Validating Catastrophe Models
► Key Aspects
► Typical ApproachBack-testing, Qualitative Review
► Validation ChallengeSensitivity testing very difficult
Hazard module
Event Set
Hazard
Vulnerability
Financial
Exposure Data
EP Curves
ELTs
Exposure
O
utpu
t
C
at M
odul
es
Inpu
t
Event Footprints
Hazard Distribution
Spatial Correlation
31 May 2013Page 17 Challenges in Validating Catastrophe Models
► Key Aspects
► Typical ApproachBenchmarking, Audit
► Validation ChallengeVery involved to properly check
Exposure module
Geocoding Data Recognition
Event Set
Hazard
Vulnerability
Financial
Exposure data
EP Curves
ELTs
Exposure
O
utpu
t
C
at M
odul
es
Inpu
t
31 May 2013Page 18 Challenges in Validating Catastrophe Models
► Key Aspects
► Typical ApproachQualitative Review, Sensitivity Testing
► Validation ChallengeLimited data
Vulnerability module
Event Set
Hazard
Vulnerability
Financial
Exposure Data
EP Curves
ELTs
Exposure
O
utpu
t
C
at M
odul
es
Inpu
t
Vulnerability Functions
Secondary Modifiers
31 May 2013Page 19 Challenges in Validating Catastrophe Models
► Key Aspects
► Typical ApproachQualitative Review, Audit
► Validation ChallengeUnderstanding the real level of uncertainty
Financial module
Event Set
Hazard
Vulnerability
Financial
Exposure Data
EP Curves
ELTs
Exposure
O
utpu
t
C
at M
odul
es
Inpu
t
Implementation Uncertainty Aggregation
31 May 2013Page 20 Challenges in Validating Catastrophe Models
► Key Aspects
► Typical ApproachBack-testing, Sensitivity Testing, Qualitative Reviews, Benchmarking
► Validation ChallengeLimited data
Output
Event Set
Hazard
Vulnerability
Financial
Exposure Data
EP Curves
ELTs
Exposure
O
utpu
t
C
at M
odul
es
Inpu
t
EP Curves Event Loss Tables
Historical Event Losses
31 May 2013Page 21 Challenges in Validating Catastrophe Models
Bigger Picture
31 May 2013Page 22 Challenges in Validating Catastrophe Models
► Validating catastrophe models should be seen within the context of internal model validation
► Each aspect needs validation
Cat Model
Data Quality
Accumulation
Pricing
Capital Modelling
Corporate Governance
Model Risk Management
Understanding
Non-modelled
Risks
Scope of validation
31 May 2013Page 23 Challenges in Validating Catastrophe Models
How to validate non-modelled perils?
► Key Aspects
► Typical ApproachQualitative Review
► Validation ChallengeSubjective
Limited Geographic Coverage
Missing Lines of Business
Missing Sub-Perils
Non-modelled Risks
Model Risk Management
Corporate Governance
Understanding
31 May 2013Page 24 Challenges in Validating Catastrophe Models
Corporate governance
► Key Aspects
► Typical ApproachQualitative Review, Audit
► Validation ChallengeChecking the right structures, processes and controls are in place to ensure the IM is robust
Materiality (SCR, Pricing, Accumulation)
Processes and Controls
Documentation Change Management
31 May 2013Page 25 Challenges in Validating Catastrophe Models
Non-modelled Risks
Model Risk Management
Corporate Governance
Understanding
31 May 2013Page 26
Understanding cat models
Challenges in Validating Catastrophe Models
Cat Team
Capital Modelling and Accumulation
Management Teams
Audit
Board
Knowledge of Catastrophe Models
► ChallengeImproving knowledge flow
Non-modelled Risks
Model Risk Management
Corporate Governance
Understanding
31 May 2013Page 27
Model Risk Management. Is the cat model really reasonable?
► Scenario: After a validation exercise, model C is selected for the IM.
► Model C is revised. The company doesn’t approve.
► Validation ChallengeRobustness of the IM
Challenges in Validating Catastrophe Models
0 50 100 150 200 250
Model AModel BModel C
Return Period
Loss
(€)
Non-modelled Risks
Model Risk Management
Corporate Governance
Understanding
Summary
Validating catastrophe models is challenging ►Catastrophe models are complex, and comprise a number of sub-modules
►Each sub-module is subject to significant uncertainties
►Data to validate them is generally limited, but there are tools to help
Validation needs to be seen in a broad context ► Internal Model perspective includes adequate understanding by stakeholders,
corporate governance, model risk management and non-modelled risks
► It also considers the interface between cat models and capital models, pricing and accumulation management tools
►Effort needs to be proportional to materiality
Page 28 Challenges in Validating Catastrophe Models31 May 2013
Paul RockettSenior ManagerT: +44 (0)20 7951 1098 M: +44 (0)78 2408 4806E: [email protected]
Contact
31 May 2013Page 29 Challenges in Validating Catastrophe Models
Challenges in Validating Catastrophe Models
Reliance►This document is provided on the basis that you agree that Ernst & Young LLP (including its partners and staff) accepts no responsibility and shall have no liability in contract, tort or otherwise in relation to the contents of this document and that any use you make of this document is entirely at your own risk.►The information in this presentation pack is confidential and contains proprietary information of Ernst & Young LLP. It should not be provided to anyone other than the intended recipients without our written consent.►The information in this pack is intended to provide only a general outline of the subjects covered. It should not be regarded as comprehensive or sufficient for making decisions, nor should it be used in place of professional advice.►Accordingly, Ernst & Young LLP accepts no responsibility for loss arising from any action taken or not taken by anyone using this pack.►The information in this pack will have been supplemented by matters arising from any oral presentation by us, and should be considered in the light of this additional information.
31 May 2013Page 30
Thank you