model practice survey 2009

Practice makes perfect Model Practice Survey 2009

2 3 Model Practice Survey 2009 Practice makes perfect

1. Preface 3

2. Executive summary 5

3. Introduction 63.1 Objective of the survey 63.2 Methodology 7

4. Survey results 84.1 Respondents 84.2 General observations 94.3 Governance 114.4 Risk models 124.5 Reporting and calculation systems 144.6 Data 154.7 Time and importance of the model practice elements 16

5. Contacts 18

Contents

We are pleased to present the results of our survey on model practice at financial institutions. The survey is designed by Deloitte Capital Markets in the Netherlands and conducted early 2009. It elaborates on the challenges that financial institutions face with risk models in an environment in which prevailing financial risk management is being questioned. Moreover, financial institutions are increasingly under scrutiny by regulatory authorities, while stakeholders express growing concerns regarding the reliability of model practice. Therefore we believe the results of this survey are important, as they not only provide insight on the dynamics of existing risk models, but also show us ways in which financial risk management today can be enhanced to benefit from opportunities and cope with challenges.

The survey focuses on models used in the context of Basel II and Solvency II. It helps to gain a deeper insight in model governance and model standards at financial institutions. Furthermore, the survey provides greater understanding of the reliability of reporting and calculation systems and the availability of high quality data.

This is the first edition of the Model Practice Survey. Our intention is to publish a second survey early 2010. This will make it possible to compare results over time and take different market circumstances into account, particularly the advancements of the regulatory environment of banks and insurers. For now we hope you will consider the current survey to be useful and that it provokes new ideas.

Hans van LeeuwenPartner, Head of Deloitte Capital Markets

1. Preface


As models provide a powerful tool for risk managers to quantify and control risks, models have an increasing prominent role in risk management at financial institutions. Both internal as external stakeholders rely on information provided by models. Therefore, models should be transparent, reliable and integrated into the business. Such elements should be safeguarded by a proper model practice. This survey investigates current model practice for the elements ‘Governance’, ‘Risk models’, ‘Reporting and calculation systems’ and ‘Data’. The survey focuses on models used by banks and insurers in the context of Basel II and Solvency II.

On the whole, many improvements are possible with respect to each of the model practice elements. Although considerable differences exist between the level of maturity per element, a significant part of the respondents rate their implementation levels as hardly mature. The governance element of model practice is the most mature. Accountability and transparency to stakeholders can be regarded as quite mature. Similarly, approval levels of authorities and the role of the risk appetite are to a large extent embedded within the organisation.

The ‘risk models’ element has reached a basic maturity. Model development, implementation, validation and monitoring procedures are well executed. Models are regularly challenged and standards for acceptance are available. However, independence between model development and validation does not seem to be safeguarded properly and a training program for the use of models is mostly absent.

Reporting and calculations systems have almost achieved a basic maturity. They are well aligned within the business and are mostly able to cope with changes in the business environment. More efforts should be put into trend, scenario and stress analysis as external stakeholders consider these to be important. Furthermore, reporting should play a more prominent part in pricing strategies and portfolio management.

Data is the least mature model practice element. Although in general data history is sufficiently long, the source databases are not always consistent and the audit trails can be improved. Data quality and its link to senior management scores quite low and procedures to cope with data quality issues seem largely absent.

Banks score better than insurers with respect to all model practice items. This result is in line with the advancements of Basel II as compared with Solvency II. Whereas banks are already expected to fully comply with the requirements of Basel II, the Solvency II regulatory framework is still being developed. The difference between banks and insurers is most apparent for the model practice element data. Improvement can be achieved with respect to data. As data is at the base of nearly every model within the scope of this survey, it is recommended to increase the maturity of this model practice element. In particular, increasing training of model usage is important as incorrect model usage can contribute to low data quality.

2. Executive summary

“Practice makes perfect”


The role of models in financial risk management has become increasingly important over the years. Models provide a powerful tool for risk managers to quantify and control risks. Both internal as external stakeholders rely on information provided by models. As a result of years of research, model reliability and credibility has grown. Financial institutions, academia and supervisors have all endorsed the use of risk management models. This has manifested in a prominent role for models in supervisory frameworks such as Basel II and Solvency II. These frameworks require such models to be transparent and of high quality. The outcome of the models should be such that they can be interpreted unambiguously and applied consistently. Elements like these should be safeguarded by designing and embedding a proper model practice.

Experience shows that model practice is open to improvement. This study will show on which topics the maturity can be improved and to what extent.This is the first edition of the Model Practice Survey. Given the results of the survey and the anticipated enhancements of the future model practice, a second survey will be initiated early 2010 to investigate developments in model practice.

3.1 Objective of the surveyThe objective of the survey is twofold. First of all, interest is focused on gaining insight into the level of maturity with regard to the four elements that constitute the foundation of model practice1:• Clear governance that addresses appropriate levels of

approval;• Appropriate risk models and standards for model

acceptance;• Reporting and calculation systems that are reliable,

flexible, transparent and easy to operate; and• Data that is easily accessible, consistent and of

sufficient quality.

Additionally, interest resides in identifying whether differences exist within and between peer groups.

1. Pre-implementation: Few efforts have yet been made to implement the required level of model practice.

2. Scattered implementation:

Some required model practice elements have already been implemented.

3. Basic implementation: The organisation has achieved the minimum level of the required model practice.

4. Aligned implementation: Model practice of various models is mutually aligned and is aligned with model practice requirements.

5. Optimised implementation:

Model practice is fully developed. It is an integral part of risk management and is aligned with strategic business decisions.

3. Introduction

3.2 MethodologyThis survey was executed during the first quarter of 2009 and was set up by means of a web based questionnaire.

To be able to investigate differences between peer groups the survey was submitted to a wide variety of managers in financial institutions. A total of approximately 180 managers were approached across Europe and the Middle East, spread across 150 different organisations. Given the focus on Basel II and Solvency II, mainly banks and insurers were targeted. Additional organisations such as investment managers and energy companies were contacted as well, albeit to a lesser extent. By means of the questionnaire information was collected about the peer group of a respondent followed by more specific questions addressing each of the four key elements of model practice: • The ‘governance’ element surrounding models was

assessed by enquiring to what extent the governance structure creates accountability and transparency to stakeholders and to what extent the governance structure defines scope, approval levels of authorities, risk appetite and knowledge standards for model users.

• The ‘risk models and standards for model acceptance’ element was assessed by enquiring to what extent risk models are consistent, compliant with external requirements and challenged. Furthermore, the survey enquired to what extent relevant procedures are in place and whether standards for acceptance are complete and adequate for development, monitoring, validation, implementation and change management purposes.

• The ‘reporting and calculation systems’ element was assessed by enquiring to what extent these systems are flexible and used for forecasting, trend and sensitivity analysis. Furthermore, it was assessed to what extent reporting is integrated into the business and aligned with business practice.

• The ‘data’ element was assessed by enquiring to what extent the data history is sufficiently long and auditable. Furthermore, it was assessed to what extent documented procedures guarantee the consistency of source databases and handle missing and/or polluted data.

Each of the specific questions could be answered on a scale with scores from 1 to 5. Table 1 gives the definition of each score. The survey concluded with two questions inquiring about the time spent on the four elements up to now as well as their importance.

1 In this report the four

elements have been

abbreviated to ‘Governance’,

‘Risk models’, ‘Reporting

and calculation systems’ and

‘Data’ respectively.

Table 1: Definition of the scores.


A total of 31 respondents completed the questionnaire. Paragraph 4.1 elaborates on the characteristics of these respondents. Paragraph 4.2 discusses general observations. These observations are not restricted to one of the four elements of model practice. The four elements are discussed separately in paragraph 4.3 to 4.6. Finally, paragraph 4.7 discusses inferences on the relative attention that is paid to each of the four elements compared to the desired situation.To be able to make proper inferences it was decided to merge the scores 1 and 2. These scores are assigned to the category ‘hardly mature’. The maturity score 3 remains, but is renamed ‘mature’. Finally, maturity scores 4 and 5 are combined and are assigned to the category ‘advanced’. Only the answers to the questions for each of the four model practice elements will be presented using the original scores.

4.1 RespondentsThe population of respondents is distributed across the following categories:

Geography: 42% of the respondents operate in organisations that reside in the Netherlands. Belgium, Scandinavia and Germany are represented to a lesser extent: 19%, 16%, and 16%, respectively. Other countries or regions have a single respondent. The Dutch majority is due to the survey being led by Deloitte’s Dutch member firm.

Industry: 65% of the respondents are active in banking. Insurers account for 27% of the respondents. 8% of respondents represent investment managers and energy companies. This distribution resembles the composition of the population of all respondents initially approached.

Asset size: The asset size of the organisations involved are widespread, ranging from asset sizes under € 10 billion to asset sizes over € 250 billion. Organisations are categorised in four groups, which enables inferences on differences in the maturity level of model practice between smaller and larger organisations.

Function respondents: The respondents operate in functions that are closely involved with risk models; i.e. model development, model validation and senior management. Because of this close involvement the quality of the responses of the questionnaire can be considered as high.

Finally, the large majority of the respondents indicated to be able to evaluate the model practice with regard to the various types of risk areas in their organisation; at least 80% of the respondents indicated to oversee all applicable risk areas. Therefore, it is not necessary to distinguish between risk areas during the remainder of the study.

Banks and insurers have a lot in common, but important differences exist as well. The nature of the risk and the risk models used differ between the industries. However, the questions posed are sufficiently generic to allow for a comparison between industries.

4.2 General observationsThis paragraph discusses the most striking observations that are inferred by comparing the distinct model practice elements. The elements are elaborated on separately in the next 4 paragraphs.In general, the level of maturity of the four model practice elements seems to vary considerably. This is best shown by Figure 5 (page 10) in which the scores given by all respondents to all questions within a particular element are averaged. The following patterns are worthwhile noticing:• For each model practice element a considerable part

of the respondents indicate that it is not yet maturely implemented in the organisation;

• The governance element is implemented most maturely;

• The data element is implemented least maturely.These observations will be discussed in more detail in paragraph 4.7, which discusses the relation with the time spent on each element.

4. Survey results

Basel II risk area Respondents able to evaluate:

Credit Risk 92%

Market Risk 88%

Operational Risk 80%

Pillar 2 83%

Solvency II risk area Respondents able to evaluate:

Credit Risk 100%

Market Risk 100%

Operational Risk 90%

Non-life underwriting risk 80%

Life underwriting risk 90%

Health underwriting risk 80%

Pillar 2 90%

Figure 1:

Figure 2:

Figure 3:

Figure 4:

Country

42%

19%

16%

16%

6%

Industry

65%

27%

8%

BankingInsuranceOther

Asset size organisation ( in billion of € )

23%

19%

26%

32%

Function respondents operate in

26%

26%32%

13%

3%


4.3 GovernanceThe governance structure creates clear accountability and transparency to all stakeholders at most organisations, with more than half of the respondents (55%) indicating they have an advanced implementation. According to 70% of respondents, the approval level of authorities is consistent for all models and in line with strategic business decisions. Remarkably, senior management scores 15% lower than respondents with other functions. Based on this result, it can be inferred that senior management would prefer to be more involved than they are at the moment.It appears that the governance structure defines the risk appetite and its connection to the organisations

strategy and capital plan. The majority of the respondents (77%) indicate to have implemented a basic level or better. Compared to banks, insurers score half a notch lower (on a scale of 1 to 5).Although the governance structure apparently defines a risk appetite, almost 40% indicates that the communication process of the risk appetite is hardly mature. This mismatch undermines the added value of the well defined risk appetite.Of all parts of the governance structure, the standards for the general level of knowledge of the models for users and management are least mature with more than 40%, indicating that such standards hardly exist. Also, only 7% believes no further improvements are necessary.

The implemented model practice seems generally more advanced at banks as compared with insurers. This is shown in Figure 6 in which all scores across all model practice elements are averaged. This result is in line with expectations: banks are expected to already adhere to the requirements of Basel II whereas insurers are still preparing for Solvency II compliance.

Particular interest was aimed at differences in model practice maturity when comparing smaller with larger organisations. It is not hard to imagine that a difference in the maturity of model practice would exist: larger organisations are more likely to operate in a more complex environment requiring a larger variety of risk models. In order to safeguard that such an environment remains controllable, proper model practice is required. However, for the smaller organisations, presumably with a smaller modelling environment, it is relatively straightforward to set up and maintain a proper model practice. On the whole, no significant differences in maturity between smaller and larger organisation are observed, as Figure 7 shows. Apparently, the above differences in the model environment offset each other. As a final note, respondents give consistent scores, i.e. either the majority of their scores occurs in the ‘advanced’ category or the majority of their scores occur in the hardly mature category. From this it can be inferred that when organisations pay attention to model practice they generally do not ignore a single model practice element.

60%

50%

40%

30%

20%

10%

0%

Hardly mature

Mature

Advanced

Governance Risk models Reporting &

calculation

systems

Data

50%

40%

30%

20%

10%

0%

Hardly mature

Mature

Advanced

Banks Insurers All respondents

5

4

3

2

1

Less than 10

10-50

50-250

More than 250

The governance structure ...

Pre-implementation

Scattered implementation

Basic implementation

Aligned implementation

Optimised implementation

0% 20% 40% 60% 80% 100%

...creates clear accountability and transparency to internal stakeholders.

...creates clear accountability and transparency to external stakeholders.

...defines a clear scope.

...defines appropriate approval levels of authorities.

...addresses conflicts of interest.

...defines the risk appetite and its connection to the overall strategy andcapital plan.

...defines a clear communication process throughout the organisation of the organisations risk appetite.

...defines standards for the general level of knowledge of the models for users and management.

Figure 8: Answers to questions with respect to governance.

Figure 5: Average maturity per model practice element.

Figure 6: Average maturity per industry.

Figure 7: Average score per model practice element and per asset category (in billions of €).


4.4 Risk modelsSimilar to the governance structure, the risk models and standards for model acceptance are also implemented relatively maturely. For example, 97% indicates to be compliant with supervisory and accounting requirements. Moreover, respondents have indicated that the risk models for different risk types are consistent with respect to assumptions, time horizon, double counting and conservatism. This is also one of the few topics in which insurers have a more mature implementation than banks.On the other hand, independence between model development and validation is not generally safeguarded, with 34% of respondents indicating this is not yet mature. Surprisingly, the score that respondents working in model validation gave this topic is not much higher. Apparently, installing a separate model validation department is not sufficient to ensure independence between model development and validation. It is unclear what causes the lack of independence.

Pre-implementation





0% 20% 40% 60% 80% 100%

Risk models for different risk types are consistent, e.g. similar assumptions, same time horizon, no double counting and aligned conservatism level.

Risk models are compliant with regulatory and accounting requirements.

Inter risk and inter business diversification effects are addressed and allocated.

Models are periodically challenged in a quantitative and qualitative manner.

Procedures for development, validation, implementation, monitoring and use of models are in place.

Independence between development and validation of models is safeguarded.

Development, validation, implementation and monitoring are continuously documented and version management is in place.

Internal and external standards for acceptance for model development purposes are complete and adequate.

Standards for acceptance are complete and adequate for monitoring and validation purposes.

Standards for acceptance are complete and adequate for implementation and change management purposes.

A periodic training program on model usage is in place.

5

4

3

2

1

Less than 10

10-50

50-250

More than 250

Risk models

A training program on model usage is largely absent in 60% of the companies. This is more than anticipated, as correct model usage is of fundamental importance for a correct use of model results. Misuse of models is likely to result in low quality of data. Specifically, 85% of respondents with a limited training program also indicate to have limited data quality (see paragraph 4.6). Therefore, organisations should increase training on model usage to increase the data quality.It is also interesting to notice that risk models is the only model practice element for which a clear positive relation exists between the level of maturity and the asset size of the organisation (see Figure 9). This may be a result of the advanced approaches larger organisations prefer to adopt. In general, standardised approaches require less procedures and standards than advanced approaches do.

Figure 9: Average score per asset size category for the element ‘risk models’ (in billions of €).

Figure 10: Answers to questions with respect to risk models.


4.6 Data The fourth element of model practice shows to be least advanced. For insurers this is remarkably worse than for banks, as can be seen in Figure 14. For almost half (45%) of the respondents the data element is hardly mature. Especially the topics on reporting of data quality to senior management and the responsibility and accountability of senior management in respect of the quality of data score very low. For questions relating to data quality approximately 57% indicated to be hardly mature.Insurers score remarkably low on the documented procedures with respect to data source consistency and the audit trails of the data. For both these items 60% indicated these items to be hardly mature, and none indicated these items to be advanced. For banks these numbers are better, as can be seen in Figure 13. On average, 62% of insurers have a limited implementation and only 7% score an advanced implementation.

Reporting and calculationsystems are used for forecasting and scenario analysis

Reporting and calculationsystems are used for trend and sensitivity analysis against key risk indicators

Reporting and calculationsystems are used for trend and sensitivity analysis against key performance indicators

Stress testing is adequately designed and implemented.

50%

40%

30%

20%

10%

0%

Hardly mature Mature Advanced

Pre-implementation





Reporting of risk figures and risk-adjusted performance measures is aligned with business practice.Reporting and calculation systems are flexible, e.g. in coping with organisational change, new insight, new products and regulatory changes.

Reporting and calculation systems are used for forecasting and scenario analysis.

Reporting and calculation systems are used for trend and sensitivity analysis against key risk indicators.

Reporting and calculation systems are used for trend and sensitivity analysis against key performance indicators.Stress testing is adequately designed and implemented.

Reporting is integrated into the business and guides pricing strategies.

Reporting is integrated into the business and guides portfolio management.

0% 20% 40% 60% 80% 100%

Banks Insurers Banks Insurers

Source database consistency Audit trails of data history

70%

60%

50%

40%

30%

20%

10%

0%


Banks Insurers All respondents

70%

60%

50%

40%

30%

20%

10%

0%


Figure 13: Maturity of source database consistency and audit trails of data history.

Figure 14: Maturity of the model practice element data per industry.

Figure 12: Answers to the questions with respect to reporting and calculation systems.

4.5 Reporting and calculation systemsThe element ‘Reporting and calculation systems’ is less mature than the two model practice elements previously referred to. This model practice element has an average maturity below the level of a basic implementation. Moreover, only 8% indicate the reporting and calculation systems are optimised and integrated into strategic business decisions. Regarding the reporting of risk figures and risk adjusted performance measures the majority of organisations have aligned these figures and measures to business practice. However, the reporting and calculation systems are not very flexible. Most organisations seem to possess reporting and calculation systems that only partially adapt to organisational changes or new supervisory regimes.Recent emphasis has been put on trend, scenario and stress analysis. However, in Figure 11 a dispersed picture emerges when looking at whether reporting and calculation systems are used for forecasting and scenario analysis, with a minor tendency to an advanced implementation. A bit less optimistic picture emerges when looking at whether reporting and calculation systems are used for trend and sensitivity analysis against key risk and key performance indicators. Here, almost 40% is hardly mature. Furthermore, although on average stress testing is adequately designed and implemented, 40% indicates the design and implementation of stress testing is hardly mature.

The integration of reporting into the business to guide portfolio management and pricing strategies is quite disappointing. 40% of the organisations have integrated reporting only to a very limited level, and only 27% have an advanced implementation. These reports could provide valuable information for the business to improve their risk and reward profile, but apparently most organisations fail to benefit from the information.

Figure 11: Maturity of the role of trend, scenario and stress analysis.

Pre-implementation





The consistency of source databases is controlled by documented procedures.

Data history is sufficiently long.

Data history is captured by audit trails.

The quality of data is regularly reported to senior management.

Data quality management involves clear responsibilities and accountability at senior management level.

Policies and procedures exist for handling missing and/or polluted data.

0% 20% 40% 60% 80% 100%

Figure 15: Answers to the questions with respect to data.


4.7 Time and importance of the model practice elements

The respondents were asked to indicate how the time spent up to now was distributed over the four elements. Similarly, the respondents were requested to indicate the relative importance of each of the four elements. Figure 16 exhibits the response.

This figure illustrates that organisations are mostly satisfied with the degree of emphasis on the different elements: the stacks hardly differ. With 6% difference between time spent and importance, risk models is the element with the largest difference. Furthermore, banks indicated that time spent and the importance of risk models are 8% and 6% lower than insurers have indicated, as can be seen in Figure 16.Although the differences seem to be quite small, digging a little deeper leads to interesting observations. For each model practice element,

19

23

25

39

14

26

32

31

11

24

28

35

14

27

22

31

19 19

28

28

25

27

25

29

100%

80%

60%

40%

20%

0%

Data

Reporting and

calculation

systems

Risk models

Governance

Time spent Importance

Banks Insurers All Banks Insurers All

Figure 17 compares the importance given to the elements to the time spent on the element. Respondents indicate that up to now 35% of their time was spent on risk models. Interestingly, 50% indicate that the importance of this element is less than the time spent suggests. Similarly, with respect to the data element, 50% of the respondents indicate that the importance is larger than the time spent suggest. On the other hand, 65% of the respondents indicate that the importance of the governance element is larger than the time that is spent up to now.This may be caused by the large amount of time needed to develop new models, which has been

70%

60%

50%

40%

30%

20%

10%

0%

Less Equal More

Governance Risk models DataReporting andcalculation systems

Figure 17: Importance of model practice element compared to time spent.

the major focus of the recent past. Once models are developed, time necessary for this model practice element can decrease and time can be spend on the other model practice elements. Apparently, clear governance is high on the list of priorities, closely followed by data. One of the most remarkable observations throughout this survey is that the data element, which generally scored to be hardly mature (see paragraph 4.6), deserves less attention according to 31% of the respondents. Since models are calibrated on internally available data, it is recommended to put more emphasis on this data element.

A note on methodology:

The information presented in this report is intended to provide general information on model

practice at financial institutions. The study solely reports on the surveyed population as expounded

in Paragraph 4.1. Deloitte has made every effort to ensure that the information provided is accurate.

However, before making any decisions you should consult a qualified professional adviser.

Figure 16: Time spent and importance of the model practice elements.

18

Should you have any questions regarding this report, please contact:

Hans van LeeuwenPartner, Head of Capital Marketst +31 88 288 3293 e [email protected]

Diederik FokkemaDirector, Capital Marketst +31 88 288 4043e [email protected]

Tjeerd DegenaarSenior Manager, Capital Marketst +31 88 288 1344 e [email protected]

5. Contacts “Regulation is constantly evolving, it’s a challenge to keep up”

Deloitte refers to one or more of Deloitte Touche Tohmatsu, a Swiss Verein, and its network of member firms, each of which is a legally separate and independent entity. Please see www.deloitte.com/about for a detailed description of the legal structure of Deloitte Touche Tohmatsu and its member firms.

Deloitte provides audit, tax, consulting, and financial advisory services to public and private clients spanning multiple industries. With a globally connected network of member firms in 140 countries, Deloitte brings world class capabilities and deep local expertise to help clients succeed wherever they operate. Deloitte’s 165,000 professionals are committed to becoming the standard of excellence.

Deloitte’s professionals are unified by a collaborative culture that fosters integrity, outstanding value to markets and clients, commitment to each other, and strength from cultural diversity. They enjoy an environment of continuous learning, challenging experiences, and enriching career opportunities. Deloitte’s professionals are dedicated to strengthening corporate responsibility, building public trust, and making a positive impact in their communities.

© 2009 Deloitte, Member of Deloitte Touche Tohmatsu

Designed and produced by MCBD at Deloitte, Rotterdam.

Deloitte Financial Advisory ServicesCapital MarketsOrlyplein 10PO Box 59237 1040KE Amsterdam The Netherlands www.deloitte.nl

model practice survey 2009

Documents