![Page 1: Risk Management Model Management Model Governance · 6 Model Risk Management Principles (SS 3/18) 1. Banks have an established definition of a model and maintain a model inventory](https://reader033.vdocuments.us/reader033/viewer/2022042200/5e9fc2b139ecb90f7010e3f9/html5/thumbnails/1.jpg)
1© 2019 The MathWorks, Inc.
Risk Management
Model Management
Model Governance
MATLAB Computational Finance Conference
Paul Peeling, MathWorks
![Page 2: Risk Management Model Management Model Governance · 6 Model Risk Management Principles (SS 3/18) 1. Banks have an established definition of a model and maintain a model inventory](https://reader033.vdocuments.us/reader033/viewer/2022042200/5e9fc2b139ecb90f7010e3f9/html5/thumbnails/2.jpg)
2
Agenda
▪ Guidance from the Regulators
▪ Realising Model Risk Management with MATLAB
– Model Inventory
– Model Development
– Model Documentation and Review
– Model Monitoring
▪ Interpretability of Machine Learning Models
![Page 3: Risk Management Model Management Model Governance · 6 Model Risk Management Principles (SS 3/18) 1. Banks have an established definition of a model and maintain a model inventory](https://reader033.vdocuments.us/reader033/viewer/2022042200/5e9fc2b139ecb90f7010e3f9/html5/thumbnails/3.jpg)
3
Model Risk Management with MATLAB
![Page 4: Risk Management Model Management Model Governance · 6 Model Risk Management Principles (SS 3/18) 1. Banks have an established definition of a model and maintain a model inventory](https://reader033.vdocuments.us/reader033/viewer/2022042200/5e9fc2b139ecb90f7010e3f9/html5/thumbnails/4.jpg)
4
Model Risk Guidance 2018
![Page 5: Risk Management Model Management Model Governance · 6 Model Risk Management Principles (SS 3/18) 1. Banks have an established definition of a model and maintain a model inventory](https://reader033.vdocuments.us/reader033/viewer/2022042200/5e9fc2b139ecb90f7010e3f9/html5/thumbnails/5.jpg)
5
Mitigating Model Risk (ECB guide to internal models)
▪ Unified inventory (“registry”) of models
▪ Consistency of modelling approaches
▪ Documentation standards – such that a 3rd party can implement
▪ Usage of models monitored on an ongoing basis
![Page 6: Risk Management Model Management Model Governance · 6 Model Risk Management Principles (SS 3/18) 1. Banks have an established definition of a model and maintain a model inventory](https://reader033.vdocuments.us/reader033/viewer/2022042200/5e9fc2b139ecb90f7010e3f9/html5/thumbnails/6.jpg)
6
Model Risk Management Principles (SS 3/18)
1. Banks have an established definition of a model and maintain a model
inventory
2. Banks have implemented an effective governance framework, policies,
procedures and controls to manage their model risk.
3. Banks have implemented a robust model development and
implementation process, and ensure appropriate use of models.
4. Banks undertake appropriate model validation and independent review
activities to ensure sound model performance and greater understanding
of model uncertainties.
![Page 7: Risk Management Model Management Model Governance · 6 Model Risk Management Principles (SS 3/18) 1. Banks have an established definition of a model and maintain a model inventory](https://reader033.vdocuments.us/reader033/viewer/2022042200/5e9fc2b139ecb90f7010e3f9/html5/thumbnails/7.jpg)
7
Model Risk Management Frameworks
![Page 8: Risk Management Model Management Model Governance · 6 Model Risk Management Principles (SS 3/18) 1. Banks have an established definition of a model and maintain a model inventory](https://reader033.vdocuments.us/reader033/viewer/2022042200/5e9fc2b139ecb90f7010e3f9/html5/thumbnails/8.jpg)
8
Reality and Vision
Data Federated at the Centre
Model Objects
Model Execution
getData
GRA as ServiceDiscovery
External Cloudwith access to tooling
Black-box technology
ExpensiveVendors
Data Madness
FROM THIS TO THISPoor Quality ModelsRegulatory ScrutinyHigh CostInconsistencyFrustrated Users
Reduced Cycle TimeAccess to Tooling
Freedom to AnalyseConsistency
Lineage & TractabilityLow Cost
Agile/DevOpsFocused
SDLC Vendor Driven
![Page 9: Risk Management Model Management Model Governance · 6 Model Risk Management Principles (SS 3/18) 1. Banks have an established definition of a model and maintain a model inventory](https://reader033.vdocuments.us/reader033/viewer/2022042200/5e9fc2b139ecb90f7010e3f9/html5/thumbnails/9.jpg)
10
Definition and Development
Review and Approval
Quality Assurance,
Pre-implementation
validation
Implementation and Deployment
Monitoring, Reporting and Performance Assessment
Model
Inventory
Model developers,
quants, analysts.
Business lines
Independent model
review and audit
Regulator
Model Validation, IT
IT, front-office
End-users
Risk management
Board and stakeholders
Regulator
Model Owners
Model Lineage throughout the Model Risk Management cycle
![Page 10: Risk Management Model Management Model Governance · 6 Model Risk Management Principles (SS 3/18) 1. Banks have an established definition of a model and maintain a model inventory](https://reader033.vdocuments.us/reader033/viewer/2022042200/5e9fc2b139ecb90f7010e3f9/html5/thumbnails/10.jpg)
11
Opportunity for Cost Savings with Model Risk Management
▪ Banks have 1,000s of models used in decision making
▪ One FTE can manage approximately 10 models
▪ One model per month can be validated
▪ Number of models increasing by 10-25% annually
▪ Model risk management can reduce costs by 30%
▪ 20% of institutions have fully adopted model risk management
Source: McKinsey 2017
Evolution of Model Risk
Management
![Page 11: Risk Management Model Management Model Governance · 6 Model Risk Management Principles (SS 3/18) 1. Banks have an established definition of a model and maintain a model inventory](https://reader033.vdocuments.us/reader033/viewer/2022042200/5e9fc2b139ecb90f7010e3f9/html5/thumbnails/11.jpg)
12
Definition and Development
Review and Approval
Quality Assurance,
Pre-implementation
validation
Implementation and Deployment
Monitoring, Reporting and Performance Assessment
Model Inventory
✓ Manage model validation projects
✓ Workflows and approval
✓ Action and report on model issues
✓ History and lineage of models
Model
Inventory
![Page 12: Risk Management Model Management Model Governance · 6 Model Risk Management Principles (SS 3/18) 1. Banks have an established definition of a model and maintain a model inventory](https://reader033.vdocuments.us/reader033/viewer/2022042200/5e9fc2b139ecb90f7010e3f9/html5/thumbnails/12.jpg)
13
Data Governance
Sourcing data from
multiple platforms and
processes
Vetting data quality
No unified data model
or interpretation
Inconsistent handling of
data by location and
over time
Historical data cannot
be reproduced
![Page 13: Risk Management Model Management Model Governance · 6 Model Risk Management Principles (SS 3/18) 1. Banks have an established definition of a model and maintain a model inventory](https://reader033.vdocuments.us/reader033/viewer/2022042200/5e9fc2b139ecb90f7010e3f9/html5/thumbnails/13.jpg)
14
Definition and Development
Review and Approval
Quality Assurance,
Pre-implementation
validation
Implementation and Deployment
Monitoring, Reporting and Performance Assessment
Model Development Environment (MDE)
✓ Build models in steps
✓ Document as-you-go
✓ Trusted and reproducible
✓ Credit and Market Risk templates
✓ Explore, research and experiment
![Page 14: Risk Management Model Management Model Governance · 6 Model Risk Management Principles (SS 3/18) 1. Banks have an established definition of a model and maintain a model inventory](https://reader033.vdocuments.us/reader033/viewer/2022042200/5e9fc2b139ecb90f7010e3f9/html5/thumbnails/14.jpg)
15
Model Development Environment (MDE)
Mission: Improve the pace, transparency and reproducibility of the model development and
review processes through user-friendly tools that encourage a consistent approach.
What are the pain points?
➢ Pace of building and reviewing models
➢ Ability to reproduce results
➢ Consistency of modeling approaches
What is the solution?
➢ MATLAB toolbox for risk modelling at HSBC
➢ Functions, apps, demos, and documentation
➢ Supports all stages of the workflow
➢ Leverages MATLAB toolboxes
➢ Target users: risk modellers and analysts
➢ Aims: improve pace, transparency, accuracy, reproducibility, consistency
Model
Executable
Model
MDE
Toolbox
![Page 15: Risk Management Model Management Model Governance · 6 Model Risk Management Principles (SS 3/18) 1. Banks have an established definition of a model and maintain a model inventory](https://reader033.vdocuments.us/reader033/viewer/2022042200/5e9fc2b139ecb90f7010e3f9/html5/thumbnails/15.jpg)
16
Building Models as a Sequence of Steps
1. Data loading and pre-processing
2. Exploratory Data Analysis
3. Sampling and Segmentation
4. Feature Engineering
5. Train Models
6. Model Validation
7. Documentation
8. Deployment
Build and Validate ModelsAccess and Explore DataProcess Data and Create
Feature
Deploy Model
Review Model
![Page 16: Risk Management Model Management Model Governance · 6 Model Risk Management Principles (SS 3/18) 1. Banks have an established definition of a model and maintain a model inventory](https://reader033.vdocuments.us/reader033/viewer/2022042200/5e9fc2b139ecb90f7010e3f9/html5/thumbnails/16.jpg)
17
Building a credit scorecard
![Page 17: Risk Management Model Management Model Governance · 6 Model Risk Management Principles (SS 3/18) 1. Banks have an established definition of a model and maintain a model inventory](https://reader033.vdocuments.us/reader033/viewer/2022042200/5e9fc2b139ecb90f7010e3f9/html5/thumbnails/17.jpg)
18
Definition and Development
Review and Approval
Quality Assurance,
Pre-implementation
validation
Implementation and Deployment
Monitoring, Reporting and Performance Assessment
Model Review Environment (MRE)
✓ Validate models in any language
✓ What-if analysis
✓ Explain results in prose
✓ Benchmark and back-test
✓ Collaborate with developers
✓ Regulator reports
![Page 18: Risk Management Model Management Model Governance · 6 Model Risk Management Principles (SS 3/18) 1. Banks have an established definition of a model and maintain a model inventory](https://reader033.vdocuments.us/reader033/viewer/2022042200/5e9fc2b139ecb90f7010e3f9/html5/thumbnails/18.jpg)
19
Collaborative Development and Review Process
Data
Pre-processing
Exploratory Data
Analysis
Exploratory Data
Analysis
Data Review Data Review
Feature
Engineering
Feature
Selection
Candidate
Models
Segmentation
Model Selection
Calibration
Performance
Assessment
Segmentation
Analysis
Model
Validation
Deployment
Development activity
Review activity
![Page 19: Risk Management Model Management Model Governance · 6 Model Risk Management Principles (SS 3/18) 1. Banks have an established definition of a model and maintain a model inventory](https://reader033.vdocuments.us/reader033/viewer/2022042200/5e9fc2b139ecb90f7010e3f9/html5/thumbnails/19.jpg)
20
Regulatory Documentation Authoring
Automatic generation of supporting analysis
relieves the burden on model development and
validation teams, by:
▪ Keeping visualizations and tables in sync with
model developments (no copy-and-paste)
▪ Adhering to corporate styles, templates and
quality output
A model development document typically ranges
between 200 and 500 pages, and consumes
30% of the effort.
Our approach allows developers and reviews to
focus effort on insight, assumptions and
limitations.
Example: IRB Application Modules
1. Scoping
2. Technical model reviews
3. IT and Data
4. Use test and experience test
5. Permanent partial use and roll out plans
6. Financial reporting and stress testing
7. Internal audit and independent validation
8. Governance
Authoring of highlighted modules are supported
by the Model Review environment.
![Page 20: Risk Management Model Management Model Governance · 6 Model Risk Management Principles (SS 3/18) 1. Banks have an established definition of a model and maintain a model inventory](https://reader033.vdocuments.us/reader033/viewer/2022042200/5e9fc2b139ecb90f7010e3f9/html5/thumbnails/20.jpg)
21
Documentation Authoring Workflow
![Page 21: Risk Management Model Management Model Governance · 6 Model Risk Management Principles (SS 3/18) 1. Banks have an established definition of a model and maintain a model inventory](https://reader033.vdocuments.us/reader033/viewer/2022042200/5e9fc2b139ecb90f7010e3f9/html5/thumbnails/21.jpg)
22
Definition and Development
Review and Approval
Quality Assurance,
Pre-implementation
validation
Implementation and Deployment
Monitoring, Reporting and Performance Assessment
Model Execution Environment (MEE)
✓ Secure and controlled
✓ Scales horizontally and vertically
✓ Immediate deployment
✓ Hosted in-house or on-cloud
✓ Audit trail of model usage
![Page 22: Risk Management Model Management Model Governance · 6 Model Risk Management Principles (SS 3/18) 1. Banks have an established definition of a model and maintain a model inventory](https://reader033.vdocuments.us/reader033/viewer/2022042200/5e9fc2b139ecb90f7010e3f9/html5/thumbnails/22.jpg)
23
Packaging, Production Deployment and Monitoring of Models
▪ Automated deployment of models into
production without translation
▪ Integrate with existing front-end and back-
end tech, or self-service platforms
▪ Performance of models monitored for
operational and regulatory requirements
![Page 23: Risk Management Model Management Model Governance · 6 Model Risk Management Principles (SS 3/18) 1. Banks have an established definition of a model and maintain a model inventory](https://reader033.vdocuments.us/reader033/viewer/2022042200/5e9fc2b139ecb90f7010e3f9/html5/thumbnails/23.jpg)
24
Definition and Development
Review and Approval
Quality Assurance,
Pre-implementation
validation
Implementation and Deployment
Monitoring, Reporting and Performance Assessment
Model Monitoring Dashboard (MMD)
✓ Visuals and metrics
✓ Multiple views
✓ Real-time monitoring
✓ Configurable alerts
✓ Configurable layout
![Page 24: Risk Management Model Management Model Governance · 6 Model Risk Management Principles (SS 3/18) 1. Banks have an established definition of a model and maintain a model inventory](https://reader033.vdocuments.us/reader033/viewer/2022042200/5e9fc2b139ecb90f7010e3f9/html5/thumbnails/24.jpg)
25
Model Monitoring Workflow
Publish
Analyse
Deploy
Monitor
MDE
App Designer
MEE
MMD
![Page 25: Risk Management Model Management Model Governance · 6 Model Risk Management Principles (SS 3/18) 1. Banks have an established definition of a model and maintain a model inventory](https://reader033.vdocuments.us/reader033/viewer/2022042200/5e9fc2b139ecb90f7010e3f9/html5/thumbnails/25.jpg)
26
Model Monitoring
![Page 26: Risk Management Model Management Model Governance · 6 Model Risk Management Principles (SS 3/18) 1. Banks have an established definition of a model and maintain a model inventory](https://reader033.vdocuments.us/reader033/viewer/2022042200/5e9fc2b139ecb90f7010e3f9/html5/thumbnails/26.jpg)
27
Opportunity for Artificial Intelligence in Model Risk Management
Interpretable, explainable
Data-driven, agile
Pillar 1 Capital Models
Credit scoring
Anti money laundering
Customer churn
![Page 27: Risk Management Model Management Model Governance · 6 Model Risk Management Principles (SS 3/18) 1. Banks have an established definition of a model and maintain a model inventory](https://reader033.vdocuments.us/reader033/viewer/2022042200/5e9fc2b139ecb90f7010e3f9/html5/thumbnails/27.jpg)
28
Explaining Machine Learning
Explanation Methods
Model-specific
Logistic Regression
Decision Tree
xNN
Model-agnostic
Global
Predictor Importance
Partial Dependency
Local
LIME
SHAPley Values
![Page 28: Risk Management Model Management Model Governance · 6 Model Risk Management Principles (SS 3/18) 1. Banks have an established definition of a model and maintain a model inventory](https://reader033.vdocuments.us/reader033/viewer/2022042200/5e9fc2b139ecb90f7010e3f9/html5/thumbnails/28.jpg)
29
Explaining Machine Learning
Partial Dependency Plots
Marginal effect of a feature on the
prediction
SHAPley Values
How far is the instance away from
the mean prediction?
PD of instance
mean PD
Total
![Page 29: Risk Management Model Management Model Governance · 6 Model Risk Management Principles (SS 3/18) 1. Banks have an established definition of a model and maintain a model inventory](https://reader033.vdocuments.us/reader033/viewer/2022042200/5e9fc2b139ecb90f7010e3f9/html5/thumbnails/29.jpg)
30
Machine Learning Ease-of-Use
Classification Learner app
Protect Against
Overfitting
Learn About
Model Types
Compare
Models with a
Variety of
Evaluation
Metrics
![Page 30: Risk Management Model Management Model Governance · 6 Model Risk Management Principles (SS 3/18) 1. Banks have an established definition of a model and maintain a model inventory](https://reader033.vdocuments.us/reader033/viewer/2022042200/5e9fc2b139ecb90f7010e3f9/html5/thumbnails/30.jpg)
31
Policies
Software
Processes
Environments
Data Governance
& Provenance
Model Lifecycle & Standards Deployment
Procedures
Risk Operations
Monitoring
Data Lineage and
Sourcing
Development
API
Review and
Validation API
Execution
Runtime
Monitoring API
Data Identification
and Validation
Model
Development
Model Review Model
Execution
Model
Monitoring
Data Governance MATLAB MDE MATLAB MRE MathWorks
MEE
MathWorks
MMD
![Page 31: Risk Management Model Management Model Governance · 6 Model Risk Management Principles (SS 3/18) 1. Banks have an established definition of a model and maintain a model inventory](https://reader033.vdocuments.us/reader033/viewer/2022042200/5e9fc2b139ecb90f7010e3f9/html5/thumbnails/31.jpg)
32
Reducing the approval, deployment and refresh timeline
Model
Definition
Model
Development
Independent
Model Review
Regulatory
Approval
Model
Deployment
Model
Definition
Model
Development
Independent
Model Review
Ongoing
Regulatory
Approval
Model
Refresh
![Page 32: Risk Management Model Management Model Governance · 6 Model Risk Management Principles (SS 3/18) 1. Banks have an established definition of a model and maintain a model inventory](https://reader033.vdocuments.us/reader033/viewer/2022042200/5e9fc2b139ecb90f7010e3f9/html5/thumbnails/32.jpg)
33
Model Risk Maturity Assessment
Legacy Model Transcription
Rebuild and Recalibration
Development and Review
Inventory Risk Management
Implementation Options for Existing Models
▪ Co-execution
– Directly execute Python and R models in MATLAB
– Supported for validation, execution and monitoring
▪ Transcription to MATLAB
– Automated for formats such as PMML
– Guidance for SAS models
▪ Videos
▪ Cheat-sheets
▪ 1-1 sessions with MathWorks
![Page 33: Risk Management Model Management Model Governance · 6 Model Risk Management Principles (SS 3/18) 1. Banks have an established definition of a model and maintain a model inventory](https://reader033.vdocuments.us/reader033/viewer/2022042200/5e9fc2b139ecb90f7010e3f9/html5/thumbnails/33.jpg)
34
Implementation Challenges and Data Considerations
▪ Best-in-class tools embrace an Agile/DevOps approach
– Version and configuration control is mandatory for traceability
– Reviews, workflow, project management for complex software
▪ Support innovation in modelling
– Reproducibility and performance across different platforms
– Permit scrutiny and independent implementation
– Reusing innovations in modelling and methodology in different tools
▪ Data considerations
– Cleansing not always possible in source systems
– Data modelling is not independent of risk modelling