robust covariances common risk versus specific risk outliers
DESCRIPTION
ROBUST COVARIANCES Common Risk versus Specific Risk Outliers. R. Douglas Martin Professor of Applied Mathematics Director of Computational Finance Program University of Washington [email protected] R-Finance Conference 2013 Chicago, May 17-18. 5/30/2012. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: ROBUST COVARIANCES Common Risk versus Specific Risk Outliers](https://reader034.vdocuments.us/reader034/viewer/2022051700/5681613d550346895dd0a058/html5/thumbnails/1.jpg)
1
ROBUST COVARIANCESCommon Risk versus Specific Risk Outliers
5/30/2012
R. Douglas Martin
Professor of Applied MathematicsDirector of Computational Finance Program
University of [email protected]
R-Finance Conference 2013Chicago, May 17-18
![Page 2: ROBUST COVARIANCES Common Risk versus Specific Risk Outliers](https://reader034.vdocuments.us/reader034/viewer/2022051700/5681613d550346895dd0a058/html5/thumbnails/2.jpg)
R Packages and Code Used R robust package
PerformanceAnalytics package
Global minimum variance portfolios with constraints
– GmvPortfolios.r: gmv, gmv.mcd, gmv.qc, etc.
Backtesting
– btShell.portopt.r: btTimes, backtet.weight – gmvlo & gmvlo.robust.r
2
![Page 3: ROBUST COVARIANCES Common Risk versus Specific Risk Outliers](https://reader034.vdocuments.us/reader034/viewer/2022051700/5681613d550346895dd0a058/html5/thumbnails/3.jpg)
Robust Covariance Uses in Finance Asset returns EDA, multi-D outlier detection and portfolio
unusual movement alerts– SM (2005), MGC (2010), Martin (2012)
Data cleaning pre-processing– BPC (2008)
Reverse stress testing– Example to follow
Robust mean-variance portfolio optimization– Is it usefull ???? If so, which method ????
3
![Page 4: ROBUST COVARIANCES Common Risk versus Specific Risk Outliers](https://reader034.vdocuments.us/reader034/viewer/2022051700/5681613d550346895dd0a058/html5/thumbnails/4.jpg)
4
Robust vs. Classical Correlations(Two assets in a larger fund-of-funds portfolio)
-0.0
40.
000.
04
Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3
1995 1996 1997 1998 1999 2000 2001
USHYHinDM
-0.0
40.
000.
04
Pfand
![Page 5: ROBUST COVARIANCES Common Risk versus Specific Risk Outliers](https://reader034.vdocuments.us/reader034/viewer/2022051700/5681613d550346895dd0a058/html5/thumbnails/5.jpg)
5
ROBUST CORR. = .65
A more realistic view of a lower diversification benefit!
CLASSIC CORR. = .30
What you get from every stats package. Gives anoverly optimistic view ofdiversification benefit!
Tolerance Ellipses (95%)
USHYHinDM
Pfa
nd
-0.04 -0.02 0.0 0.02 0.04
-0.0
4-0
.02
0.0
0.02
0.04
0.06
ROBUSTCLASSICAL
![Page 6: ROBUST COVARIANCES Common Risk versus Specific Risk Outliers](https://reader034.vdocuments.us/reader034/viewer/2022051700/5681613d550346895dd0a058/html5/thumbnails/6.jpg)
6
Hedge Fund Returns Example
-0
.05
0.00
0.05
F1
0.00
0.05
2000 2001 2002 2003 2004
F2
0.00
0.05
F3
0.0
0.1
F4
0.00
0.05
F5
-0.1
0.0
0.1
2000 2001 2002 2003 2004
F6
0.0
0.1
2000 2001 2002 2003 2004
F7
-0.0
50.
000.
05
F8
![Page 7: ROBUST COVARIANCES Common Risk versus Specific Risk Outliers](https://reader034.vdocuments.us/reader034/viewer/2022051700/5681613d550346895dd0a058/html5/thumbnails/7.jpg)
7
0.26
0.3
0.34
0.05
0.36
0.37
0.14
0.3
0.28
-0.13
0.32
0.35
-0.03
0.45
0.06
0.5
0.53
0.44
0.01
0.3
0.24
0.09
-0.3
0.16
-0.01
0.4
0.51 0.38
0.32
0.29
0.44
0.26
0.34
0.16
0.2
0.23
0.45
0.2
-0.15
-0.04
0.06
0.31
0.36
0.36
0.24
0.16
0.21
0.1
-0.01
0.13
0.02
-0.31
-0.19
0.26
0.28 0.33
F1 F2 F3 F4 F5 F6 F7 F8
F1
F2
F3
F4
F5
F6
F7
F8
ROBUSTCLASSICAL
Hedge Fund Returns Example
![Page 8: ROBUST COVARIANCES Common Risk versus Specific Risk Outliers](https://reader034.vdocuments.us/reader034/viewer/2022051700/5681613d550346895dd0a058/html5/thumbnails/8.jpg)
Portfolio Unusual Movement Alerts
Retrospective analysis
Dynamic alerts
8
2 1ˆˆ ˆt t td r μ Σ r μ
Mahalanobis Squared Distance (MSD)
Crucial to use a robust covariance matrix estimate ! Σ
![Page 9: ROBUST COVARIANCES Common Risk versus Specific Risk Outliers](https://reader034.vdocuments.us/reader034/viewer/2022051700/5681613d550346895dd0a058/html5/thumbnails/9.jpg)
9
Commodities Example(see Appendix A of Martin, Clark and Green, 2009)
-0.2
0.0
0.2
CATTLE
2004 2005 2006 2007 2008 2009
HOGS COPPER
COFFEE SILVER
-0.2
0.0
0.2
SUGAR
-0.2
0.0
0.2
2004 2005 2006 2007 2008 2009
OJ PLANTINUM
2004 2005 2006 2007 2008 2009
OILC
![Page 10: ROBUST COVARIANCES Common Risk versus Specific Risk Outliers](https://reader034.vdocuments.us/reader034/viewer/2022051700/5681613d550346895dd0a058/html5/thumbnails/10.jpg)
10
Robust Alerts
Unreliable alerts!
Classical Alerts
1
2
3
4
5
6
7
0 10 20 30 40 50 60
CLASSICAL
57
0 10 20 30 40 50 60
ROBUST
9
11
28 50 52
57
58
59 60
Index
CO
MM
OD
ITY
RE
TUR
NS
DIS
TAN
CE
S
![Page 11: ROBUST COVARIANCES Common Risk versus Specific Risk Outliers](https://reader034.vdocuments.us/reader034/viewer/2022051700/5681613d550346895dd0a058/html5/thumbnails/11.jpg)
11
library(xts)library(robust)library(lattice)
ret = read.zoo("commodities9.csv",sep=",",header = T,format = "%m/%d/%Y")ret = as.xts(ret)ret = ret['2004-01-31/2008-12-31',]xyplot(ret,layout = c(3,3)) # Not the same as slidedata = coredata(ret)
cov.fm <- fit.models(CLASSICAL = covMLE(data), ROBUST = covRob(data,estim = "mcd",quan = .7))plot(cov.fm,which.plots = 3)
![Page 12: ROBUST COVARIANCES Common Risk versus Specific Risk Outliers](https://reader034.vdocuments.us/reader034/viewer/2022051700/5681613d550346895dd0a058/html5/thumbnails/12.jpg)
Robust Covariance Choices in R “robust”
Min. covariance determinant (MCD)
M-estimate (M)
Donoho-Stahel (DS)
Pairwise estimates (PW)– Quadrant correlation and GK versions– Positive definite (Maronna & Zamar, 2002)
For details see the R robust package reference manual. See also Chapter 10.2.2 of Pfaff (2013) Financial Risk Modeling and Portfolio Optimization with R, Wiley.
12
affine equivariant
not affine equivariant
![Page 13: ROBUST COVARIANCES Common Risk versus Specific Risk Outliers](https://reader034.vdocuments.us/reader034/viewer/2022051700/5681613d550346895dd0a058/html5/thumbnails/13.jpg)
13
(1 ) ( , ) t F N H r μ Σ
A natural model for common factor outliers– Market crashes
The Usual Robustness Outliers Model
tT nR rtable of returns with rows
1. Probability of a row containing an outlier is independent of the dimension n, so the majority of the rows of are outlier-free.
2. Fraction of rows that have outliers is unchanged under affine transformations, so use affine equivariant estimators, e.g., MCD
Rr
![Page 14: ROBUST COVARIANCES Common Risk versus Specific Risk Outliers](https://reader034.vdocuments.us/reader034/viewer/2022051700/5681613d550346895dd0a058/html5/thumbnails/14.jpg)
14
Independent Outliers Across Assets (IOA)
Suppose for example that . Then
1, 2, ,( ) , i i nP B the probability of a row not containing an outlier is , which decreases rapidly with increasing p . E.g., for :
tr (1 )n
# of assets n 5 10 15 20prob. clean row .77 .60 .46 .36
N.B. Affine transformations increase the percent of rows with outliers, so no need to restrict attention to affine equivariant estimators.
A natural model for specific risk outliers
(AKMZ, 2002 and AVYZ, 2009) 1 (0) i iB Let if asset is (is not) an outlier.
1 2 , , , ( )n i iB B B P B Assume are independent with
.05
![Page 15: ROBUST COVARIANCES Common Risk versus Specific Risk Outliers](https://reader034.vdocuments.us/reader034/viewer/2022051700/5681613d550346895dd0a058/html5/thumbnails/15.jpg)
Choice of Outliers Model and EstimatorBoth are useful, but:
The usual outliers model handles market events outliers and for these an affine equivariant robust covariance matrix estimator will suffice, e.g., MCD.
The independent outliers across assets model is needed for specific risk outliers, and for these one may need to use a pairwise estimator to avoid breakdown!
Goal: Determine when pairwise robust covariance matrix estimator performs better than MCD, etc.
15
![Page 16: ROBUST COVARIANCES Common Risk versus Specific Risk Outliers](https://reader034.vdocuments.us/reader034/viewer/2022051700/5681613d550346895dd0a058/html5/thumbnails/16.jpg)
Asset Class & Frequency ConsiderationsSpecific risk outliers are more frequent in the case of:
Higher returns frequency, e.g., weekly and daily Smaller market-cap stocks Hedge funds Commodities … ???
16
![Page 17: ROBUST COVARIANCES Common Risk versus Specific Risk Outliers](https://reader034.vdocuments.us/reader034/viewer/2022051700/5681613d550346895dd0a058/html5/thumbnails/17.jpg)
17
Non-Normality Increases with Frequency
-0.4
-0.2
0.0
0.2
0.4
-0.2 0.0 0.2 0.4
MONTHLY11403
-0.4
-0.2
0.0
0.2
-0.1 0.0 0.1 0.2
WEEKLY11403
-0.4
-0.3
-0.2
-0.1
0.0
0.1
-0.05 0.0 0.05
DAILY11403
-0.2
0.0
0.1
0.2
0.3
-0.2 -0.1 0.0 0.1 0.2
MONTHLY20220
-0.2
-0.1
0.0
0.1
0.2
-0.10 0.0 0.05 0.10
WEEKLY20220
-0.1
0-0
.05
0.0
0.05
0.10
-0.06 -0.02 0.02 0.06
DAILY20220
Quantiles of fitted normal distribution
Qua
ntile
s of
sam
ple
data
Comparison of Non-normality over Different Time Scales
![Page 18: ROBUST COVARIANCES Common Risk versus Specific Risk Outliers](https://reader034.vdocuments.us/reader034/viewer/2022051700/5681613d550346895dd0a058/html5/thumbnails/18.jpg)
Outlier Detection Rule for Counting
optimal 90% efficient bias robust location estimate* associated robust scale estimate*
Outliers: returns outside of
Probability of normal return being an outlier: 0.5%
* Use lmRob with intercept only in R package robust
18
ˆ ˆ ˆ ˆ( 2.83, 2.83)s s
s
![Page 19: ROBUST COVARIANCES Common Risk versus Specific Risk Outliers](https://reader034.vdocuments.us/reader034/viewer/2022051700/5681613d550346895dd0a058/html5/thumbnails/19.jpg)
19
Empirical Study of IOA Model Validity Four market-cap groups of 20 stocks, weekly returns 1997 – 2010 in three regimes:
– 1997-01-07 to 2002-12-31– 2003-01-07 to 2008-01-01– 2008-01-08 to 2010-12-28
1. Estimate outlier probability for each asset, and hence the probability that a row is free of outliers under the IOA model.
2. Directly estimate the probability that a row has at least one outlier.
3. Compare results from 1 and 2 across market-caps and regimes.
i 11n
i i
row
![Page 20: ROBUST COVARIANCES Common Risk versus Specific Risk Outliers](https://reader034.vdocuments.us/reader034/viewer/2022051700/5681613d550346895dd0a058/html5/thumbnails/20.jpg)
20
-0.3
-0.1
0.1
0.3
PLX
S-0
.20.
00.
10.
2
BW
INB
-0.2
0.0
0.1
0.2
HG
IC
2000 2005 2010
Index
-0.1
0.0
0.1
0.2
WTS
SMALL-CAPS
4 of the 20 Small-Caps for Entire History
![Page 21: ROBUST COVARIANCES Common Risk versus Specific Risk Outliers](https://reader034.vdocuments.us/reader034/viewer/2022051700/5681613d550346895dd0a058/html5/thumbnails/21.jpg)
21
1 3 5 7 9 11 14 17 20
% OUTLIERS IN EACH ASSET
ASSETS
PE
RC
EN
T
02
46
8
2008 2009 2010 20110
24
68
1012
Index
CO
UN
T
# OF ASSETS WITH AN OUTLIER
Small-Caps Outliers in Third Regime
![Page 22: ROBUST COVARIANCES Common Risk versus Specific Risk Outliers](https://reader034.vdocuments.us/reader034/viewer/2022051700/5681613d550346895dd0a058/html5/thumbnails/22.jpg)
22
Large-Caps Outliers in Third Regime
2008 2009 2010 20110
510
15Index
CO
UN
T
# OF ASSETS WITH AN OUTLIER
1 3 5 7 9 11 14 17 20
% OUTLIERS IN EACH ASSET
ASSETS
PER
CE
NT
01
23
45
6
![Page 23: ROBUST COVARIANCES Common Risk versus Specific Risk Outliers](https://reader034.vdocuments.us/reader034/viewer/2022051700/5681613d550346895dd0a058/html5/thumbnails/23.jpg)
23
Evaluation of IOA Model for Weekly Returns
1997-01-07 to 2002-12-31 MICRO SMALL MID LARGE% Clean Rows IOA Model 32 39 46 59
% Clean Rows Direct Count 37 48 58 69
2003-01-07 to 2008-01-01 MICRO SMALL MID LARGE% Clean Rows IOA Model 33 46 52 57
% Clean Rows Direct Count 37 49 62 66
2008-01-08 to 2010-12-28 MICRO SMALL MID LARGE% Clean Rows IOA Model 23 31 39 46
% Clean Rows Direct Count 43 48 57 62
![Page 24: ROBUST COVARIANCES Common Risk versus Specific Risk Outliers](https://reader034.vdocuments.us/reader034/viewer/2022051700/5681613d550346895dd0a058/html5/thumbnails/24.jpg)
24
1.0
1.5
2.0
2.5
gmv.logmv.lo.mcdgmv.lo.qcmkt
Cum
ulat
ive
Ret
urn
Weekly Returns, Window = 60, Rebalance = Weekly
LONG-ONLY GMV,GMV.MCD, GMV.PW, MKT
-0.1
50.
00
Wee
kly
Ret
urn
1998-03-03 2001-07-03 2005-01-04 2008-07-01Date
-0.5
-0.1
Dra
wdo
wn
![Page 25: ROBUST COVARIANCES Common Risk versus Specific Risk Outliers](https://reader034.vdocuments.us/reader034/viewer/2022051700/5681613d550346895dd0a058/html5/thumbnails/25.jpg)
25
1.0
1.5
2.0 gmv.lo
gmv.lo.mcdgmv.lo.qcmkt
Cum
ulat
ive
Ret
urn
Weekly Returns, Window = 60, Rebalance = Monthly
LONG-ONLY GMV,GMV.MCD, GMV.PW, MKT
-0.1
50.
00
Wee
kly
Ret
urn
1998-03-03 2001-07-03 2005-01-04 2008-07-01Date
-0.5
-0.1
Dra
wdo
wn
![Page 26: ROBUST COVARIANCES Common Risk versus Specific Risk Outliers](https://reader034.vdocuments.us/reader034/viewer/2022051700/5681613d550346895dd0a058/html5/thumbnails/26.jpg)
26Time
0.40.50.60.70.80.9
dvi.gmv
0.40.50.60.70.80.9
dvi.gmv.mcd
0.40.50.60.70.80.9
1998 2000 2002 2004 2006 2008 2010
dvi.gmv.qc
HHI Diversification Index (sum-of-squared wts.)
![Page 27: ROBUST COVARIANCES Common Risk versus Specific Risk Outliers](https://reader034.vdocuments.us/reader034/viewer/2022051700/5681613d550346895dd0a058/html5/thumbnails/27.jpg)
27
library(PerformanceAnalytics)library(robust)source("GmvPortfolios.r")source("btShell.portopt.r")source("btTimes.r")
# Diversification Index Functiondvi =function(x){1-sum(x^2)}
# Input returnsret.all = read.zoo("smallcap_weekly.csv",sep=",",header = T,format = "%m/%d/%Y")mkt = ret.all[,"VWMKT"]ret = ret.all[,1:20]n.assets <- ncol(ret)
# get returns datesall.date = index(ret)
Back-Test Code
![Page 28: ROBUST COVARIANCES Common Risk versus Specific Risk Outliers](https://reader034.vdocuments.us/reader034/viewer/2022051700/5681613d550346895dd0a058/html5/thumbnails/28.jpg)
28
# compute the backtest timest.mw <- btTimes.mw(all.date, 4, 60)
# backtestingweight.gmv.lo <- backtest.weight(ret, t.mw,gmv.lo)$weightweight.gmv.lo.mcd <- backtest.weight(ret, t.mw, gmv.lo.mcd)$weightweight.gmv.lo.qc <- backtest.weight(ret, t.mw, gmv.lo.qc)$weight# The Diversification Index Plotsgmvdat = coredata(weight.gmv.lo)gmvdat.mcd = coredata(weight.gmv.lo.mcd)gmvdat.qc = coredata(weight.gmv.lo.qc)dvi.gmv = apply(gmvdat,1,dvi)dvi.gmv.mcd = apply(gmvdat.mcd,1,dvi)dvi.gmv.qc = apply(gmvdat.qc,1,dvi)dvi.all = cbind(dvi.gmv,dvi.gmv.mcd,dvi.gmv.qc)dvi.all.ts = as.zoo(dvi.all)index(dvi.all.ts) = index(weight.gmv.lo)xyplot(dvi.all.ts, scales = list(y="same"))
![Page 29: ROBUST COVARIANCES Common Risk versus Specific Risk Outliers](https://reader034.vdocuments.us/reader034/viewer/2022051700/5681613d550346895dd0a058/html5/thumbnails/29.jpg)
29
# compute cumulative returns of portfoliogmv.lo <- Return.rebalancing(ret, weight.gmv.lo)gmv.lo.mcd <- Return.rebalancing(ret, weight.gmv.lo.mcd)gmv.lo.qc <- Return.rebalancing(ret, weight.gmv.lo.qc)
# combined returnsret.comb <- na.omit(merge(gmv.lo, gmv.lo.mcd, gmv.lo.qc, mkt, all=F))
# return analysischarts.PerformanceSummary(ret.comb,wealth.index = T, lty = c(1,1,1,4),colorset = c("black","red","blue","black"), cex.legend = 1.3,cex.axis = 1.3, cex.lab = 1.5, main = "Weekly Returns, Window = 60, Rebalance = Monthly \n LONG-ONLY GMV,GMV.MCD, GMV.PW, MKT")
![Page 30: ROBUST COVARIANCES Common Risk versus Specific Risk Outliers](https://reader034.vdocuments.us/reader034/viewer/2022051700/5681613d550346895dd0a058/html5/thumbnails/30.jpg)
30
“Statistics is a science in my opinion, and it is no more a branch of mathematics than are physics, chemistry and economics; for if its methods fail the test of experience – not the test of logic – they will be discarded”
- J. W. Tukey
Thank You!
![Page 31: ROBUST COVARIANCES Common Risk versus Specific Risk Outliers](https://reader034.vdocuments.us/reader034/viewer/2022051700/5681613d550346895dd0a058/html5/thumbnails/31.jpg)
Proprietary, for use only by permission.
Thank You!
31
“Statistics is a science in my opinion, and it is no more a branch of mathematics than are physics, chemistry and economics; for if its methods fail the test of experience – not the test of logic – they will be discarded”
- J. W. Tukey
![Page 32: ROBUST COVARIANCES Common Risk versus Specific Risk Outliers](https://reader034.vdocuments.us/reader034/viewer/2022051700/5681613d550346895dd0a058/html5/thumbnails/32.jpg)
References Alqallaf, Konis, Martin and Zamar (2002). “Scalable robust covariance and
correlation estimates for data mining”, Proceedings of the eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 14-23. ACM.
Scherer and Martin (2005). Modern Portfolio Optimization, Chapter 6.6 – 6.9, Springer
Maronna, Martin, and Yohai (2006). Robust Statistics : Theory and Methods, Wiley. Boudt, Peterson & Croux (2008). “Estimation and Decomposition of Downside Risk
for Portfolios with Non-Normal Returns”, Journal of Risk, 11, No. 2, pp. 79-103. Alqallaf, Van Aelst, Yohai and Zamar (2009). “Propagation of Outliers in Multivariate
Data”, Annals of Statistics, 37(1). p.311-331. Martin, R. D., Clark, A and Green, C. G. (2010). “Robust Portfolio Construction”, in
Handbook of Portfolio Construction: Contemporary Applications of Markowitz Techniques, J. B. Guerard, Jr., ed., Springer.
Martin, R. D. (2012). “Robust Statistics in Portfolio Construction”, Tutorial Presentation, R-Finance 2012, Chicago,
32