how model risk can impact the financial markets & model risk at banks

Model Risk:How Model Risk Can Impact the Financial Markets&Model Risk at Banks

Bart Rikkert, Lead Model Validator, AEGON

Dr. Ebbe Negenman, Head Regulatory Risk, ABN AMRO Bank

Global Association of Risk Professionals

November 2016

2

The views expressed in the following material are the

author’s and do not necessarily represent the views of

the Global Association of Risk Professionals (GARP),

its Membership or its Management.

3 | © 2014 Global Association of Risk Professionals. All rights reserved.Helping people achieve a lifetime of financial security

Model Risk

‘How model risk can impact the

financial markets’

Amsterdam, 7 November 2016

Bart Rikkert

4 | © 2014 Global Association of Risk Professionals. All rights reserved.

What & Why

What exactly is a model and why do

we need it?

3

Model types

What are the key model types and

their sources of model risk?

7

Model Risk

Some recent examples of model risk

11

Model Risk - Agenda

Content

Discussion

Will model risk be controlled or grow

worse?

1

5


Quantitative method, system, or approach that applies statistical,

economic, financial, or mathematical theories, techniques, and

assumptions to process input data into quantitative estimates.

A model consists of three components:

- Information input component – delivers assumptions and data to the model

- Processing component – transforms inputs into estimates

- Reporting component – translates the estimates into useful business information

Source: Supervisory Guidance on Model Risk Management, Board of Governors of the

Federal Reserve System, April 4, 2011

What is a model - Theory

The what and why behind models

Definition for a model


Reporting

and Use of

resultsApplication/

Calculation

kernel

Selected

assumptions

Selected model

version

Selected model

points

Selected data

Per model run

Reports

User

validation

Reconciliation

processes

Reconciliation

processes

Data

processing

OutputOutputMethodology

Assumptions &

Expert Judgments

Data

Assumption input

process / interface

Data

input

process /

interface

Reporting

and Use of

results

Model

development

and Testing

Preparing for and Validating model runs

What does a model look like

Calculation

parameters

Calculation

parametersRunlist



What is a models added value

What is the added value of models?

Defining a model

Make calculations and aggregations

Show impact of underlying Assumptions and Expert Judgments

Support reporting and control

Help to quickly assess the impact of changes/choices

Increase understanding of modelled products/risks


What is a good Model Use

Key goal of a model is to consider all possible realities using:

► Market data

► Portfolio specific/client data

► Product characteristics & regulations

► Expert judgments & assumptions

► Advanced modelling skills

A model can predict based

on what we know and

can improve predictions:

While models can be further

improved, predictions stay

based on the past.



What & Why


we need it?

3

Model types



7

Model Risk


11

Model Risk - Agenda

Content

Discussion


worse?

1

5


Capital Models

Market Models

- Interest rate (volatility), Fixed income, Default, currency and hedges

Actuarial/Liability models:

- Longevity, Mortality, Policy holder behaviour, Cost and Operational Risk

Scenario and aggregation models

- Scenario engines, correlations, proxy models

and assessment/monitoring models to understand movements

Key Solvency II Internal Model components

Examples


Valuation/Best Estimate models

Models are aligned with capital models

Use same key risk drivers as capital models.

Market Models for capturing market values

Valuation and DCF models for non-publically traded asset-valuation

Actuarial models to value insurance commitments for the coming 30-

50 years.

Cash flow and valuation models

Examples


Other models

Key model types where we run model risk

Examples

Pricing / MCVNB models

- To determine value of new business and set pricing

- Biggest difference: Includes first cost

Strategic/Planning models

- For future plans and hedging

Supporting models

- Scenario models, (big) data


What & Why


we need it?

3

Model types



7

Model Risk


11

Model Risk - Agenda

Content

Discussion


worse?

1

5


Model Risk - examples

Liquidity completely dried up for RMBS market in 2008

Negative rates for Government Bonds (current)

Mortgage prepayment rates fluctuate due to government policy

Actuarial longevity tables changed considerably

Solvency II ratio difficult to predict by market/report on by companies

Methodology / Expert Judgment

Model Risk


Residual Model Risk

• (Insurance) companies spend significant effort and resources on

containing model risk.

• Many controls are focussed on day to day use of the models and

focus on the day to day (small) movements.

• Concern is that most models can be quite accurate for small

movements, but become much less accurate for big movements.

• Senior Management and Regulators however often require

models to be able to handle all possible futures.

Model Risk

Controls around Model Risk


Accept that the model has limitations

- Seems like a weak approach; ‘just accepting’

- Does not improve the technical model itself or the current output of the numbers

Focus on creating clarity on these model limitations

- Improves model understanding

- Gives clear boundaries to model use

- Add value to monitoring and links to management actions

Model risk and Model limitations

Model Risk

Managing residual Model Risk


What & Why


we need it?

3

Model types



7

Model Risk


11

Model Risk - Agenda

Content

Discussion


worse?

1

5


For discussion

Statement 1:

Technology and computing power is still growing strong, allowing millions of

scenarios to be run and hugely complicated models to be build and used in a

reasonably timeframe. A few ‘mathemagicians’ still understand the math, the

rest considers them magicians creating huge black boxes.

Statement 2:

For valuation and capital models the are becoming more and more similar with

spreading risk as a key assumption. This creates a huge systemic risk,

especially in combination with ever increasing capital requirements and the

quantitative easing of the central banks.

Developments and potential impact on model risk

For discussion


19The end

http://www.facebook.com/aegon

http://www.facebook.com/aegon


Model Risk: The Definitions In Use At ABN AMRO

A model:

‘a quantitative method that applies statistical, economic, financial, or

mathematical theories, techniques, and assumptions to process

input information into quantitative estimates.’

The use of models exposes the bank to model risk:

‘the potential loss the bank may incur, as a consequence of

decisions that could be principally based on the output of internal

models, due to errors in the development, implementation or use of

such models.’

A good read:

SR 11-7 Guidance on Model Risk Management

– https://www.federalreserve.gov/bankinforeg/srletters/sr1107.htm


Models Are Widely Used And Grow In Importance

Currently e.g. used for:

identifying and measuring risks,

valuing exposures, instruments or positions,

pricing, liquidity and capital allocation,

conducting stress testing,

measuring compliance with internal limits,

meeting financial or regulatory reporting requirements

other

Growing importance, e.g., for

Monitoring customer accounts

Anticipate credit deterioration using automatic alert models

Fraud and money laundering detection

Customer on boarding, engagement and marketing campaign

models

Automated decisions making


Model Risk: One Of The Largest Risks Of Banks

McKinsey on Risk Number 1, Summer 2016

“ Model risk. Banks’ increasing dependence on business modelling requires that risk managers

understand and manage model risk better. Although losses often go unreported, the consequences of

errors in the model can be extreme. For instance, a large Asia–Pacific bank lost $4 billion when it

applied interest-rate models that contained incorrect assumptions and data-entry errors. Risk

mitigation will entail rigorous guidelines and processes for developing and validating models, as well

as the constant monitoring and improvement of them.”

– http://www.mckinsey.com/business-functions/risk/our-insights/mckinsey-on-risk


Increasing Regulatory Focus On Model Risk

Financial Times 16 Aug. 2015

“ECB doubles the time needed to review banks' risk models - Having originally hoped to

complete the review of banks' risk models within a year or two, the ECB has set a

deadline of four years for work on the project, according to a tender document seen by

the Financial Times.”

Likely regulatory questions to banks:

Does management body understand the degree of model risk

in credit, market & operational risk?

To what extend are models used to support significant

business decisions?

How significant is model risk, is there sensitivity, scenario’s &

stress testing?

How sound are model validation & review processes?

What are model risk control mechanism & how are these

tested?


Q4 2008: Fortis Bank Posts a Net Loss of About €6 Billion due to Model Risk


Capital Calculation At ING Group As It Used To Be

€3

€1 €1 €2

€5 €10 €8

€17 €13

€6 €8 €4

€25


2007: ING Group Claims To Have €5bn Capital Excess


October 2008: ING takes a €10bn Capital Injection from Dutch State


In the Netherlands we have > € 700bn of mortgages. But client behaviour cannot be estimated

on historical data due to current fundamental macro economic disruptions.

Current Model Risk In Mortgages: Disruptive Client Behavior

3% mortgage

(after tax)4% on savings 1.25% mortgage

( after tax)0.4% on savings

Clients will maximize loan,

keep all savings Clients will invest excess savings

to reduce mortgage costs

World of 2006 World of 2016


Model Risk Effect of Miscomputing An LTV

Den Helder -13%:

LTV =115.2%

Amsterdam: + 17.3%

LTV = 85%

+5%:

Den Helder : LTV = 95.2%

Amsterdam : LTV = 95.2%


Effects Of Miscomputing An LTV Is Large

Effect on Risk can be large

• LTV is primary driver of LGD model, and expected loss computation.

Effect on Earnings of the bank can be large

• Clients with low LTV get a discount (or can get it during the lifetime)


Model Risk Life Cycle at ABN AMRO

Effective Challenge is key: Model Validation is

involved in the Model Life Cycle at two stages:

after initial model development or re-development

of an existing model.

Model review is carried out risk based and in line

with regulatory requirements. Each risk model is

to be reviewed internally with a minimum

frequency of once a year, or additionally, when an

event has taken place which significantly impacts

the model performance

For market risk and ABN AMRO Clearing models,

the valuation models which serve as the input to

the risk models must be reviewed and revalidated

at least once every three years, but on an annual

basis an assessment is made if model reviews

should be moved forwards.

Model

development

Model

validation

Model

approval

Model

Implementation

Model

Use

• Inappropriate input

• Inappropriate methodology

• bad performance

• Poor model documentation

• Incomplete validation

• Poor documentation

• Misunderstanding

• Wrong decision

• Incorrect implementation/

deployment of

methodology

• Misuse of the model

• Incorrect run of model

• Inappropriate testing


Model Risk Scorecards: Independent From Model Origination

Review Element

Risk Score

Raw data set(s) are defined and appropriate 1

Data checks are in place &

Outliers are recognized and treated appropriatelyN.A.

Data assumptions are identified and supportable 1

Input models are identified and supportable 3

Modeling methodology and parameters are sound 2

Model assumptions are identified and supportable 1

Relevant model backtests are performed 1

Documentation is of good quality 2

Model performance backtesting results are adequate 1

Regulatory and Policy requirements are satisfied 1

There are no concerns regarding Market outlook 2

There are no concerns regarding model implementation 1

0

MoC MoC level used in the model 0

Severity What is the severity of the detected "high risks"? 2

1.92

Medium

Model Risk Rating

(including MOC)Medium

Size of Portfolio Size of the portfolio 176 148

Model category Model category ALM(IR BB-ILAAP)

Overdue MAG date 6/6/2016

Number of action points Low Risk 1

Medium Risk 3

High Risk

Additional factors

Number of Action Points

Client Rate Model (ANMLWW08)

Component Risk Score

Data & Model inputs 3.00

Model Methodology 1.50

Model Assessment 1.25

Model Risk Rating

Enter the name of the model

(model code in the parentheses)

& the name of the validator

Enter the Value in million

For credit risk, use Outstanding

Enter the MAG date when the model is/will be APPROVED

(keep the format the same as the example: DD-MM-YYYY)

(The cell will assign:

White = MAG date is in the future

Green=Model is not overdue.

Amber=Model is overdue less than 6 month.

Red=Model is overdue more than 6 month.)

Enter the MoC in percentage

(without the percentage

sign).

Enter scores for the

review elements:

1: Good

2: Moderate

3: Poor

Determine the severity of the

detected high risk (=red)

elements. The final model risk

score will be adjusted

accordingly.


Integrated Model Risk Reporting At ABN AMRO

Direct resource costs (economic and human) and development and implementation time


Model Risk – Herd Behavioral Risk

Since it is costly (both in time and money) to develop a model that is eventually rejected by

the regulator, banks will have a tendency to choose risk modelling techniques that have been

proven to be acceptable by the regulator. In other words banks are more identical in their

models than you would expect from real competitors.


There is Always Hope

Essentially, all models are wrong,

…but some are useful.”

George Edward Pelham Box

(18 October 1919 –

28 March 2013)

C r e a t i n g a c u l t u r e o f

r i s k a w a r e n e s s ®

Global Association of

Risk Professionals

111 Town Square Place

14th Floor

Jersey City, New Jersey 07310

U.S.A.

+ 1 201.719.7210

2nd Floor

Bengal Wing

9A Devonshire Square

London, EC2M 4YN

U.K.

+ 44 (0) 20 7397 9630

www.garp.org

About GARP | The Global Association of Risk Professionals (GARP) is a not-for-profit global membership organization dedicated to preparing professionals and organizations to make

better informed risk decisions. Membership represents over 150,000 risk management practitioners and researchers from banks, investment management firms, government agencies,

academic institutions, and corporations from more than 195 countries and territories. GARP administers the Financial Risk Manager (FRM®) and the Energy Risk Professional (ERP®)

Exams; certifications recognized by risk professionals worldwide. GARP also helps advance the role of risk management via comprehensive professional education and training for

professionals of all levels. www.garp.org.


how model risk can impact the financial markets & model risk at banks

Economy & Finance