kpmg model risk management...

November 8, 2013

KPMG Model Risk

Management Survey

Discussion and Summary

IACPM

Background Model Risk Governance and

Validation:

A (Very) Brief History

©2013 KPMG LLP, a Delaware limited liability partnership and the U.S. member firm of the KPMG network of independent member firms affiliated with KPMG International Cooperative, a Swiss entity. All rights reserved.

Financial Models

3

A long, long time ago

A few basic models were used

Interest Rate Risk (Balance Sheet Value measure and Net Interest Income/Margin)

VaR for Trading Book

Simple valuation models/tools

Risk Ratings and Transition Matrices

Portfolio Construction and Asset Allocation

Loss Forecasting / Reserving

Actuarial Models and Tables (Insurance)


Financial Models

4

Over time, models have grown in number and increased their importance

Expanded modeling use includes areas such as:

Derivative and Financial Instrument Pricing, Risk, and Valuation (and much more complex derivatives)

Securitization (cashflow and waterfall; highly structured products, synthetics)

Credit Decisioning/Underwriting

Credit Portfolio Management

Credit Loss Modeling (PD/LGD/EAD; ALLL; OTTI)

Operational Risk

Capital (Economic Capital, CCAR/Capital Adequacy and Forecasting, Stress Testing)

Risks associated with trading (Duration, Convexity; VaR, cVAR)

Counterparty Risk and Exposure (EE, PFE, EAD; CVA)

Financial Reporting (CVA/DVA, Fair Value Balance Sheet)


Financial Models

5

Largest Institutions

Goldman Sachs, Morgan Stanley, JPMorgan Chase, Bank of America Merrill Lynch, Citigroup, Wells Fargo,

Thousands of models, including some highly complex and interrelated, others very specific in their use

Large, Complex Institutions

Fannie Mae, Freddie Mac, FHLBs

GE, Coca-Cola, AIG, MetLife, Prudential, State Street

Banks > $50 Billion (SunTrust, Regions, Capital One, BBVA Compass)

100+ models/systems/tools

Other Large Companies

Financial Reporting Models, especially around derivatives or investments

Financial Forecasting Models

Insurance – actuarial models and economic capital

Commodity and FX hedging (exposure, hedge accounting, derivative valuation)


Model Governance – Issues & Concerns

6

Frequently-Seen Critical Regulatory Concerns on Model Risk Management:

1. An incomplete model inventory;

2. Lack of a robust method to update the model inventory on a regular basis;

3. Lack of independence of the model validation department;

4. Model validation failed to demonstrate “effective challenge”;

5. Lack of data used in the model development process to support review/validation;

6. Lack of developmental evidence to substantiate model assumptions;

7. Lack of an explanation to support the application of expert judgment and model overrides;

8. Lack of a transparent process to generate important inputs for a model; (e.g. a bank fails to

demonstrate how it determines the magnitude of PD shocks in a macroeconomic stress

testing model.)

9. Fail to include relevant risk drivers in the model; this is particular common in macroeconomic

stress testing.

10. Models have not been validated.

11.Failure to maintain comprehensive and up-to-date model documentation.

KPMG Survey:

Objectives and methodology

Introduction

KPMG undertook a survey to confirm where industry practices were established and where there were

areas of divergence.

Areas of practical application remain in the interpretation of the guidance, and will require further

consistent clarity from regulators. Until such time practices will vary across institutions, for example:

• Practical questions around which systems, processes, tools, calculations, and applications should be

considered “models” in the sense of the interagency regulatory guidance, and which should not?

• How should model risk be assessed, both for individual models and in the aggregate given that

practices and regulatory expectations are evolving, creating various points of uncertainty?

• Practical model validation issues including: At what point does a model update or a recalibration

require re-validation? Is full validation required if no significant or material changes have been made

to a model since its previous validation?

• Institutions also have governance standards concerns including:

• What are the implicit independence requirements on model risk management functions, and what exactly

does independence mean;

• What classification and materiality criteria are standard requirements of all institutions and which criteria

are optional depending on an individual profile;

• What actions would satisfactorily demonstrate the required level of executive management and Board

engagement; and,

• How and under what circumstances might models not passing validation be approved for use?

Model Risk Governance Survey

Introduction

8 ©2013 KPMG LLP, a Delaware limited liability partnership and the U.S. member firm of the KPMG network of independent member firms affiliated with KPMG International Cooperative, a Swiss entity. All rights reserved.

9


Objectives and methodology

Objectives

To identify and understand the following

industry-applied model risk management

practices in relation to regulatory model risk

management guidance contained in OCC 2011-

12 / Fed SR 11-7:

- Model definition scope

- Model classification

- Model risk management governance

- Validation process practices

Model development and

implementation issues

Key attributes of model validation

program and issues

- Resources and staffing

- Risk assessment of models

- Regulatory issues and future trends

Methodology

Web-based survey conducted during mid 2013 among

financial services executives responsible for model risk

management.

Total respondents: 60+

The respondents included a wide representation of

financial institutions with varying asset sizes (less

than $100B to over $500B):

Global banks

International - Non-U.S. banks**

Global systemically important financial

institutions “G-SIFI”

Regional U.S banks

Insurance companies

* Global bank operating in the U.S under regulatory guidance and not necessarily

defined as G-SIFI.

** Bank headquartered outside the U.S., with US Operations


Executive summary

A. Model definition

The regulatory model definition prescribed in the guidance is consistently applied by respondents

irrespective of size, complexity, and geographical footprint:

Majority of respondents report that regulators have provided feedback on the definition (e.g., not broad enough)

and/or have emphasized certain characteristics expanding scope.

In-scope inventory includes 95-100 percent of the following traditional models: risk management, regulatory

and economic capital impact, stress testing and scenario analysis, and valuation.

Divergence exists across institutions on including / excluding the following: product development,

underwriting and pricing, compliance and marketing/targeting, client asset management, decision

support/monitoring and, strategic / other business planning / forecasting models:

Only 58 percent include rule based decision models (e.g., fraud) and 70 percent exclude maintaining formal control

structure models

79 percent believe there are clear criteria/definitions with the institution to distinguish in/out of scope models.

Institutions typically maintain a centralized model inventory, however this may vary based on the nature of

institution, numerous distributed lines of business, model types, geographical footprint etc.

Most utilize in-house/proprietary tools, databases, or spreadsheets to facilitate the model inventory

management:

Vendor provided and other tools are used to a lesser degree.

Report an ongoing effort to further rationalize model inventory over the next 12 months

For example include regulatory models and anti-money laundering (AML)/fraud models which have traditionally

while excluding budget and forecasting, and economic scenario generation models.


Executive summary – Model definition



Executive summary – Model classification

B. Model classification

Model classification methods commonly applied include one or more of the following:

Materiality based: Financial or regulatory materiality / threshold defined by institution (e.g., Tier 1 model

represents material financial statement impact or High model represents significant regulatory impact)

Risk or factor based: Risk/factors generally translated into a tier or high, medium or low classification (e.g., high

represents a new model or with significant model changes, or complex with high operational/process risk)

Materiality/risk based combination: Combination of risk based factor, materiality and judgmental overlay.

Model reliance, complexity, regulatory and financial impact are most common criteria used to determine

model validation priority classification.

The financial impact is the most important criteria followed by regulatory impact especially for large banks.

The model tiering/classification methods are consistently applied across regional banks given the benefits of

central control and a limited geographical footprint.

Model classifications are mostly reviewed by institutions on a periodic basis (e.g., annually most common)

or upon a significant model change/update or in reaction to other internally defined trigger event.

Validation practices and standards vary based on the model classification (e.g., low rated model may only be

subject to periodic monitoring):

Respondents evenly divided on work effort associated with models not subject to a full independent review or

outcome analysis i.e., may result in an ultra conservative approach or disproportionate attention on low risk models

Varying practices may include higher documentation, back testing standards and level of staff assigned.



Executive summary – Governance

C. Model risk management governance (MRMG)

MRMG responsibility predominantly rests with Enterprise Risk Management (ERM) or Chief Risk Officer

(CRO). Some respondents report this function falling under internal audit, credit or the business unit.

Responsibility for assigning model classification generally rests with the independent model risk

management (MRM) group or a combination of functions (e.g., business unit and MRM).

MRMG policies clearly address the appropriate roles and responsibilities assigned to each function (e.g.,

model owners, developers, users, validation, compliance, etc)

The primary roles of the independent MRM team are as follows:

Establish model governance and validation and compliance program including policy and procedures

Approve the models after development and validation, and control model use (i.e., sign-off on model)

Review the model from a governance perspective

Most significant actions taken by MRMG resulting from the changing regulatory environment include:

Increased number of model validation staff (e.g., greater quantitative experience and education)

Updated model inventory (e.g., fresh review of the inventory constituents vs. model definition / regulator input)

Review of model risk governance infrastructure (e.g., policy, procedures, and use of technology)

Increased scrutiny from senior management

Increased urgency to validate regulatory models.

The role of internal audit (i.e., 3rd line) has changed for most respondents in particular regional banks.

Significant work still remains to enhance key less mature program attributes (e.g., model change

management policy / procedures, and ongoing model monitoring / tracking program.


D. Model development practices

Most common model development and implementation issues include:

Failure of model documentation to meet internal policy standard

Insufficient model development life cycle documentation

Limited explanation to support the application of judgment and model overrides

Transparency of the models and how their output impacts business decisions

Insufficient data used in the model development process to support review/validation especially in global banks

Scheduling model validations with model users in a timely manner particularly for regional banks.

Regulator expectations/emphasis regarding model development and implementation practices:

Reduced implementations without validations, outcomes analysis, and/or documentation

Developers gaining a better understanding of model use, data quality, justifying model overlays, and metrics

Improved enterprise-wide model governance (e.g., clear lines between developer, validator and model

approver)

Enhanced documentation with emphasis on developmental evidence and sensitivity analysis

Increased focus on pre-implementation validation (i.e., rigorous with thorough testing and approval process)

First line of defense to clearly explain the data applicability and cleaning process

More effective challenge by the MRM group during development.


Executive summary – Model development practices


E. Model validation practices

Common model validation issues include:

Input data and assumptions (e.g., data integrity and reliability from certain data sources, heavy reliance on

management’s subjective assumptions, etc.)

Challenge scheduling validations in a timely manner

Models used for purposes other than what was originally intended when designed and implemented

Shortage of experienced and qualified staff to build and validate the models

Incomplete model inventory documentation.

Model finding and severity rating definitions (e.g., Satisfactory, needs improvement, unsatisfactory) are

consistently applied by respondents across models.

Half the respondents indicate that the validation scope and approach differs for internally developed vs.

external 3rd party vendor models (e.g., depending on access to documentation and level of transparency to code,

assumptions etc).

Model performance is monitored on a regular basis (i.e., to meet the “ongoing model performance

monitoring and tracking” requirements).

MRMG staffing needs have increased for all respondents over the last 12 to18 months:

Primarily for regulatory compliance purposes

Resources (e.g., validators and developers) increased depending on maturity of program

In many cases a new MRMG group, sub-function recently created.


Executive summary – Model validation practices


F. Model risk assessment

The majority of respondents have not established a model risk appetite and associated limits.

Less than half of respondents measure model risk and establish a reserve/buffer or provision in some form

(e.g., set aside a model risk reserve as a component of regulatory or economic capital).

Of those respondents that reserve, a 2-3 percent buffer on average is established using the following approaches:

Subjective qualitative calculation included in regulatory or economic capital

Component included in operational risk using a qualitative based metric (e.g., number and severity of issues)

Judgmental qualitative add-on on top of regulatory or economic capital

Include model uncertainty/imprecision incorporated in capital as part of CCAR/DFAST stress testing process

May be based on model attributes (i.e., materiality, number of validations completed, findings from validations,

severity of issues, level of model risk assessment/quality grade, or whether systemic models).

Model risk aggregation is more common at larger institutions.

Models assigned a model risk buffer/reserve include:

Regulatory capital (including allowance for loan losses), economic capital and valuation models

A low percentage of respondents reserve for all models.

Respondents expect more regulatory pressure to quantify risk on an aggregated level.

Banks are challenged to develop a consistent quantitative approach across model risk measures and types.

Most respondents consider this area to be a work in progress.


Executive summary – Model risk assessment


G. Staffing and resources

MRM staffing resources (e.g., developers/validators) has increased for all respondents the last 12-18 months.

Key drivers for resource changes include:

Enhancement of overall MRM program

Expanded model definition scope (e.g., increased types and number of models)

Increased demand on ongoing monitoring and outcomes analysis.

Model validation staff are segregated by majority of institutions by business unit, specialty, process, or

model type (e.g., valuation, credit, Basel, risk models).

Respondents report that model developers are generally not independent from users (i.e. this may limit the

degree of effective challenge prior to validation).

Most respondents don’t have a formal dedicated model validation staff/ model user training program.

Common challenges faced with scheduling and completing validations in a timely manner include:

Pressure from model user to undertake and complete validation

Insufficient number of staff

Insufficient documentation by model owners

Increased volume of models requiring validation

Unresolved open issues and exceptions


Executive summary – Staffing


© 2013 KPMG LLP, a Delaware limited liability partnership and

the U.S. member firm of the KPMG network of independent

member firms affiliated with KPMG International Cooperative

(“KPMG International”), a Swiss entity. All rights reserved.

NDPPS 179790

The KPMG name, logo and “cutting through complexity” are

registered trademarks or trademarks of KPMG International.

Contact Details:

Mark J. Nowakowski Principal, Financial Risk Management

KPMG LLP

(404) 222-3192

[email protected]

www.kpmg.com

kpmg model risk management...

Documents