a new approach to backtesting and risk model selection

Theoretical framework Model validation Model selection

A new approach to backtesting and risk modelselection

Jacopo Corbetta (Ecole des Ponts - ParisTech)

Joint work with: Ilaria Peri (University of Greenwich)

June 18, 2016

Jacopo Corbetta Backtesting & Selection June 18, 2016 1 / 22

Outline

1. Theoretical framework

2. Model validation

3. Model selection

An internal point of view

VaR is only as good as its backtest. When someone shows me aVaR number, I don’t ask how it is computed, I ask to see thebacktest.

Aaron Brown - Risk manager of the year 2012

Many banks that have adopted an internal model-based approachto market risk measurement routinely compare daily profits andlosses with model-generated risk measures to gauge the quality andaccuracy of their risk measurement systems. This process, knownas “backtesting”, has been found useful by many institutions asthey have developed and introduced their risk measurementmodels.

Basel Committee (1996)

An internal point of view

VaR is only as good as its backtest. When someone shows me aVaR number, I don’t ask how it is computed, I ask to see thebacktest.

Aaron Brown - Risk manager of the year 2012

Many banks that have adopted an internal model-based approachto market risk measurement routinely compare daily profits andlosses with model-generated risk measures to gauge the quality andaccuracy of their risk measurement systems. This process, knownas “backtesting”, has been found useful by many institutions asthey have developed and introduced their risk measurementmodels.

Open Problems

1. Methodologies for the backtesting of numerous alternativerisk measures are less straight-forward (approximations orMonte-Carlo simulations).

2. Rigorous definition of backtesting and backtestability is stillmissing.

3. Model selection among risk models is meaningful only forelicitable estimator and only different forecasts of the samerisk estimator can be compared.

Outline

2. Model validation

3. Model selection

Some notation

(Ω,F ,P), probability space.

I M1 set of all probability measures on the real line.

I L0 set of all almost surely finite real valued random measures.

I R set of law invariant risk measure.

I X ∈ L0 represent the returns, its distribution is given byP(x) = P(X < x).

I % : P% → R+ is a law invariant risk measure.

I (P, %) will be called a risk procedure.

Some notation

(Ω,F ,P), probability space.

I M1 set of all probability measures on the real line.

I L0 set of all almost surely finite real valued random measures.

I R set of law invariant risk measure.

I X ∈ L0 represent the returns, its distribution is given byP(x) = P(X < x).

I % : P% → R+ is a law invariant risk measure.

I (P, %) will be called a risk procedure.

Idea behind our framework

The fraction actually covered can then be compared with theintended level of coverage to gauge the performance of the bank’srisk model

Main question: What is the actual level of coverage provided bythe risk model?

I For VaR the level of coverage can be naturally associated withits confidence level λ. For other risk measures alternative thisis not straightforward;

I The choice of the confidence level is a critical issue pointedout by both practitioners and academics (see Kerkhof andMelenberg (2004) for detail explanation).

Idea behind our framework

The fraction actually covered can then be compared with theintended level of coverage to gauge the performance of the bank’srisk model

Main question: What is the actual level of coverage provided bythe risk model?

I For VaR the level of coverage can be naturally associated withits confidence level λ. For other risk measures alternative thisis not straightforward;

I The choice of the confidence level is a critical issue pointedout by both practitioners and academics (see Kerkhof andMelenberg (2004) for detail explanation).

Level of coverage

Definition: Level of coverage

Given a % : P% ⊂M1 → R and a probability measure P, the levelof coverage associated to the risk measure % applied to P isdefined by:

λ%P := P(−%(P)) = P(X < −%(−P)) ∈ [0, 1]

The level of coverage λ%P offered by the risk model is theprobability of having a violation, where:

I the probability is provided by the model P for forecastingreturns;

I the violation occurs in respect of the capital requirement−%(P).

Basic example:λVarεP = ε.

Level of coverage

Definition: Level of coverage

Given a % : P% ⊂M1 → R and a probability measure P, the levelof coverage associated to the risk measure % applied to P isdefined by:

λ%P := P(−%(P)) = P(X < −%(−P)) ∈ [0, 1]

The level of coverage λ%P offered by the risk model is theprobability of having a violation, where:

I the probability is provided by the model P for forecastingreturns;

I the violation occurs in respect of the capital requirement−%(P).

Basic example:λVarεP = ε.

A new definition of Backtestability

Definition

A risk measure % : P% ⊂M1 7→ R is backtestable over a setP ⊆M1 if there exists a map P% that associates to every P ∈ Pthe coverage level given by % applied to P.

I the backtestability of a risk measure does not depend by anytheoretical property, such as the elicitability or consistency;

I the backtestability only depends by the identification of thelevel of coverage provided by the risk model.

A new definition of Backtestability

Definition

A risk measure % : P% ⊂M1 7→ R is backtestable over a setP ⊆M1 if there exists a map P% that associates to every P ∈ Pthe coverage level given by % applied to P.

I the backtestability of a risk measure does not depend by anytheoretical property, such as the elicitability or consistency;

I the backtestability only depends by the identification of thelevel of coverage provided by the risk model.

Universal Backtesting

Theorem

Every law invariant risk measure % is backtestable over its domainof definition P% in the sense of Definition (9)

Observations:

I determining the capital requirement by a risk measure and itsbacktesting are two separate issues;

I the objective of the backtesting should be to verify if thecoverage has been adequate and this prescinds from theprocess of generation of the capital requirement.

Universal Backtesting

Theorem

Every law invariant risk measure % is backtestable over its domainof definition P% in the sense of Definition (9)

Observations:

I determining the capital requirement by a risk measure and itsbacktesting are two separate issues;

I the objective of the backtesting should be to verify if thecoverage has been adequate and this prescinds from theprocess of generation of the capital requirement.

Outline

2. Model validation

3. Model selection

The setting

(Ω, Ftt=1,...,T ,P) filtered space. Xt returns at time t.

I Ft(x) = P(Xt < x) real unknown probability distributions ofthe returns.

I Pt(x) = P(Xt < x |Ft−1) forecast probability distributions ofthe returns.

The setting

(Ω, Ftt=1,...,T ,P) filtered space. Xt returns at time t.

I Ft(x) = P(Xt < x) real unknown probability distributions ofthe returns.

I Pt(x) = P(Xt < x |Ft−1) forecast probability distributions ofthe returns.

The violations

Let % be a risk measure. We define the violations as

1 if Xt < −%(Pt)

0 otherwise .

Remark

The violations follow a Bernulli distribution with parameterλ0t = λ%Pt

under the model probability, and with parameterλt = Ft(Xt < −%(Pt)) under the real probability

Key Assumption

The violations are independent.

The violations

1 if Xt < −%(Pt)

0 otherwise .

Remark

Key Assumption

The violations

1 if Xt < −%(Pt)

0 otherwise .

Remark

Key Assumption

Test 1: unilateral coverage test

We set the null and alternative hypothesis as follows:

I H0: λ0t = Pt(−%(Pt)) = λt = Ft(−%(Pt)) for every t

I H1: λt ≥ λ0t for every t, with strict inequality for some t.

Test statisticWe define the test statistic

Z1 :=T∑t=1

I Under H0, Z1 follows a Binomial Poisson distribution with parametersλ0

tt .I For the significance level α, the rejection region is

CZ1 = z1 : PZ1(z1) > 1− α.I Generalizes the traffic light approach (Basel, 1996) but only two regions.

Z1 :=T∑t=1

Test 2: asymptotic bilateral coverage test

I H1: λt 6= λ0t for some t.

Test statisticWe define the test statistic:

∑Tt=1(It − λ0

t )√∑Tt=1 λ

0t (1− λ0

I Under H0, by Lyapunov Central Limit Theorem Z2 converges to a

Standard Normal: Z2d−→ N(0, 1) .

I For the significance level α, the rejection region isCZ2 :=

z2 : z2(x) < qN

)∪z2 : z2(x) > qN

(1− α

∑Tt=1(It − λ0

t )√∑Tt=1 λ

0t (1− λ0

z2 : z2(x) < qN

)∪z2 : z2(x) > qN

(1− α

∑Tt=1(It − λ0

t )√∑Tt=1 λ

0t (1− λ0

z2 : z2(x) < qN

)∪z2 : z2(x) > qN

(1− α

Outline

2. Model validation

3. Model selection

We validated, and now?

I The sole study of the number of exception does not tell if themodel is good or not. One should study also the magnitudeof violations

I Model selection can be conducted by minimizing the averagebacktesting error (loss) (Lopez 1998)

I However, the common practice of using loss functions thatare not consistent with the forecasting estimator leads tomeaningless inferences (Gneiting 2011).

I As a consequence, model selection seems to be possible onlyfor elicitable estimator.

Our proposal: the magnitude of exceptions

We propose a method for comparing forecasting outcomes of everyrisk estimator.

Definition

Let (P, %) be a risk measure procedure with coverage levelλ0t ∈ (0, 1). The magnitude of the exceptions over the backtestingperiod T is:

M((P, %)) =T∑t=1

(xt − (−%(Pt)))+g(Pt(−%(Pt)))

+ (xt − (−%(Pt)))−g(1− Pt(−%(Pt)))

A new order

Consider the family (P, %)ii∈I = (Pitt , %i )i∈I .

We say that (P, %)i is preferable to (P, %)j for the magnitude order,and we write

(P, %)i M (P, %)j

if and only ifM((P, %)i ) ≤ M((P, %)j)

I M is a total preorder.

A new order

Consider the family (P, %)ii∈I = (Pitt , %i )i∈I .

We say that (P, %)i is preferable to (P, %)j for the magnitude order,and we write

(P, %)i M (P, %)j

if and only ifM((P, %)i ) ≤ M((P, %)j)

I M is a total preorder.

Best performing risk measure procedure

We can choose the best performing procedure as

(P, %) = arg mini∈I

M((P, %)i )

I This method can be used for selecting the best estimatoramong all the risk models previously validated.

I Our aim is not to propose an alternative property foridentifying ”superior” statistical functionals (as done by theelicitability or consistency)

Best performing risk measure procedure

We can choose the best performing procedure as

(P, %) = arg mini∈I

M((P, %)i )

I This method can be used for selecting the best estimatoramong all the risk models previously validated.

I Our aim is not to propose an alternative property foridentifying ”superior” statistical functionals (as done by theelicitability or consistency)

Conclusions

I We propose a practical approach to backtesting.

I Our framework as a natural interpretation: we provide 2straightforward test.

I We propose a simple way to compare performance of differentrisk procedures.

Thank you for your attention

Conclusions

I We propose a practical approach to backtesting.

I Our framework as a natural interpretation: we provide 2straightforward test.

I We propose a simple way to compare performance of differentrisk procedures.

Thank you for your attention

References

I Acerbi, C., Szekely, B., 2015. Backtesting Expected Shortfall.

I Campbell, S., 2005. A review of backtesting and backtesting procedures.

I Christoffersen, P., 2010. Encyclopedia of Quantitative Finance -Backtesting.

I Embrechts, P., Hofert, M., 2014. Statistics and Quantitative RiskManagement for Banking and Insurance.

I Gneiting, T., 2011. Making and evaluating point forecasts.

I Kerkhof, J., Melenberg, B., 2004. Backtesting for Risk-Based RegulatoryCapital.

I Lopez, J.A., 1998. Methods for Evaluating Value-at-Risk Estimates.

a new approach to backtesting and risk model selection

Documents