jeff bohn - state street at the chief analytics officer forum west coast

GXN-1900

How to communicate complexity with cognitively compelling clarity

Dr. Jeffrey R. Bohn Chief Science Officer Head of GX Labs [email protected]

Limited Access

GXN-1900

Outline

• Introduction and background

• Addressing cognitive biases when communicating analytical output

• Framing analytical output for non-quantitative audiences

• Improving communication of complex output with compelling data visualization

• Balancing simplification and analytical output implication dilution

• Key points to remember

2

GXN-1900

Introduction and background

3

GXN-1900

David Hume A wise man, therefore, proportions his belief to the evidence. -- Of Miracles; Section X, Part I. 87 For if truth be at all within the reach of human capacity, it is certain it must lie very deep and abstruse: and to hope we shall arrive at it without pains, while the greatest geniuses have failed with the utmost pains, must certainly be esteemed sufficiently vain and presumptuous. -- Treatise of Human Nature, Introduction

4

GXN-1900

The wisdom of Archilochus (Greek lyric poet from Paros, lived 680 BCE to 645 BCE)

The fox knows many things, but the hedgehog knows one big thing.

5

GXN-1900

Thinking

Theories

6

GXN-1900

What makes decision-support analytics complex?

• Multi-variate optimization problem– often with controls and results playing out over different time horizons.

• Uncertainty– “risk” defined by known distributions and “Knightian uncertainty” defined by unknown models/data-generating processes

• Interconnectedness– both explicit and implicit.

• Hierarchies of relationships and relevant assumptions

• Separation of point predictions/estimates and distribution estimates

7

GXN-1900

Drivers of model complexity • Nonstationarity:

– Macro-economic regime changes – Firms changing leverage – Governments changing regulation

• Heterogeneous return distributions by asset class – Non-normal – Skewed – Fat tails – Liquidity

• Capturing obscure, but material risks – Correlated exposure to a latent factor (e.g., US housing market) – Sectorally diversified investment-grade bond portfolio may constitute material

concentration risk – Wrong-way, counterparty risk to an over-hedged energy company

• Human reaction to changing dynamics - Bank management reaction to stress environment by changing underwriting standards - Not selling in a down market and distressed-selling in an up market not recognized - Inability or unwillingness to understand a model and its implications may lead to value-

destroying behavior - Traders gaming particularly sensitive aspects of a risk model

8

GXN-1900

Advances in evolution

• Darwin (1859): Natural selection is a key mechanism of evolution based on the differential survival and reproduction of individuals due to phenotype differences.

• Romanes (1895): Neo-darwinism refers to germ-plasm theory advocated by Wallace and Weissmann.

• Huxley (1942): Modern synthesis drew together multiple fields of biology marrying natural selection, genetics, natural population analysis, systematics, etc.

• Today’s models of evolution are much more complex:

– Genetic networks composed of tens to hundreds of genes interact

– “Regulatory” genes behave conditional on the environment

– Some hereditary variations are nonrandom in origin

– Some acquired information is inherited (epigenetic inheritance)

– Evolutionary change can result from instruction as well as selection

Jablonka and Lamb (2014)

9

GXN-1900

Advances in risk modeling

• Capital-asset pricing model

• Markowitzian portfolio theory

• Arrow-Debreu

• Black-Scholes-Merton

• Continuous-time finance

• Value at risk

• Monte-Carlo simulation

• Expected-tail loss

Are these “advances” i.e., model complexification worth the investment?

10

GXN-1900

Cognitive biases Addressing cognitive biases when communicating analytical output

11

GXN-1900

Dealing with priors: Confirmation bias The trouble with most folks isn’t so much their ignorance, as knowing so many things that ain’t so. Josh Billings related by Friedman (1965)

12

GXN-1900

Cognitive biases

• False dichotomy: Presenting two choices such that it seems they are the only possibilities. – Simple vs. complex model – Use no models vs. use only one model

• Perfect as the enemy of the good (or good enough) • Red herrings and missing forest for the trees • Biases

– Affect heuristic: Analyst or executive has “fallen in love with” a particular output so that they minimize model problems and exaggerate model strengths.

– Groupthink – Saliency bias: Overly influence by analogous, past success – Confirmation bias – Availability bias – Anchoring bias – Halo effect: Impression of model author, analyst or even model influences interpretation – Sunk-cost fallacy: A particular model output has driven strategy/investment – Overconfidence – Disaster neglect – Loss aversion

Traps arising from logical fallacies and cognitive biases

13

GXN-1900

Understanding the decision maker: Elephant and the rider

• Elephant: Automatic processes • Rider: Controlled processes • Modeling decision making

– Humean model: Reason is a servant – Platonic model: Reason could and should rule – Jeffersonian model: Head and heart are co-emperors

• “Seeing that” vs. “Reasoning why” • Rationalist delusion: Maintaining healthy skepticism of reason– smarter people

rationalize better People who devote their lives to studying something often come to believe that the object of their fascination is the key to understanding everything. Location 58 Conscious reasoning functions like a press secretary who automatically justifies any position taken by the president. p. 106 And as reasoning is not the source, whence either disputant derives his tenets; it is in vain to expect, that any logic, which speaks not to the affections, will ever engage him to embrace sounder principles. – David Hume

Haidt (2012)

14

GXN-1900

How to persuade an individual to make a decision

Agree on values Disagree on values

Agree on facts Computational decision Negotiate

Disagree on facts Experiment Paralysis or chaos

15

From Koomey (2001) figure 19.1 p. 88.

GXN-1900

Working memory and compelling communication

• Working memory – Operates over a few seconds – Temporary storage – Manipulates attention – Focuses attention – Resists distractions – Guides decision-making

• Can only process 5 to 9 “chunks” of information within working memory at any given moment in time (Miller, 1955)

• Deviating from expectations typically causes the listener to disengage • Working memory dis-fluently “chunks” instead of always focusing on what matters • Working memory “calls” long-term memory to assist in processing; if nothing is there,

cognitive flow is broken– result is likely disengagement

Education is essential to build up long-term memory

16

GXN-1900

Communicating Uncertainty

• Explain signal and noise in specific terms

• Communicate how model disentangles signal and noise

• Identify and root out data biases

• Educate on error bars and confidence intervals

Sampling error does not necessarily equal “uncertainty” in terms of implications of model output

17

GXN-1900

Compelling communication to overcome cognitive biases

• Educate as to model’s usefulness as a function of complexity

• Frame key performance indicators (KPIs)

• Prototype, socialize, productionize

• Avoid big-bang projects– include executives in discovery/iteration process

18

GXN-1900

Moving from discovery to action

Persuade decision makers regarding…

• Credibility

• Likelihood

• Materiality

• Addressibility

Chronic model weaknesses

• No feedback loops

• No thresholds

• Inadequate spill-over effects

19

GXN-1900 GXN-1900

Framing Framing analytical output for non-quantitative audiences

20

GXN-1900

Be transparent as to epistemic nature of analytical output

• What the executive doesn’t know, but is knowable: Model output is available and useful e.g., credible metrics identify risk (in the technical sense.)

• What the executive or the analyst don’t know yet, but is knowable: Proof-of-concept model is available; however, more investment (e.g., data, analysts, systems, tools) is needed.

• What is knowable with uncertainty: Model output is available and potentially useful; however, questions remain as to whether the model itself is specified correctly e.g., metrics reflect Knightian uncertainty. (Bayesian methods may be helpful.)

• What is unknowable: Model output is not available.

• What one chooses not to know: Incentives overpower model output.

21

GXN-1900

Focus on criteria for accepting the validity of analytical output

1. Accuracy [out-of-sample confirmation of estimated probability distribution and contributions of underlying components to that distribution]

2. Consistency (both internal and external) [multi-asset-class, assets & liabilities]

3. Broadness in scope [granularity and comprehensiveness]

4. Simplicity [complex enough to capture dynamics, but simple enough to be diagnosed and communicated to a quantitatively-informed business head]

5. Fruitfulness [output substantively contributes to impactful decisions]

Depending on the theory under evaluation, criteria may contradict each other so a relative weighting may be needed i.e., given a particular circumstance, some criteria are more important than others. Kuhn (1977)

In portfolio risk analysis, we typically add Timeliness to the evaluation process– a successful theory/model/system that cannot provide timely output is useless.

22

GXN-1900

Contextualization: Describe regimes

• Business as usual (BAU) i.e., sustainable growth

• Cyclical (typical up and down growth– but same process and similar trend)

• Structural (move to a different growth path driven by a different process)

• Providing context is critical:

– Benchmark to competitors (cohorts)

– Benchmark to optimal, feasible outcome

– Show time series

– Drill into components on a consistent basis

23

GXN-1900

Separate analytical framework from interpretation Reverse Stress Testing

Values at horizon Search through factor space

Inflation

S&P

Oil Price

GDP

Rates

Macro scenarios

What factor realizations generate a portfolio value in the specified confidence interval of portfolio distribution?

Source: State Street Global Exchange℠

Limited Access

Figure provided for illustrative purposes.

24

GXN-1900

Strategies for cognitively compelling clear framing

• Study audience beforehand

– What is prototypical for them?

– What are their preferences?

– What is their attention span?

• Start with the punchline focusing on 3 most important outcomes

• Develop explanations in a cognitively fluent way from the listener’s perspective

– Familiarity with material

– Education

– Materiality of material to listener’s responsibilities

• Make the messages as simple as possible, but no simpler

• Be prepared to drill transparently into assumptions, logic and evidence

25

GXN-1900 GXN-1900

Data visualization Improving communication of complex output with compelling data visualization

26

GXN-1900

Multi-asset-class portfolio-risk modeling Use-objective focuses on incorporating risk strategy into portfolio management

probability

Start of Tail Expected Tail Loss

stress losses

Portfolio value Expected

Value

Risk-appetite assessment • Model guides executives as to whether allocation is prudent from a risk perspective Sub-portfolio/manager/hedging evaluation • Model assists in evaluating how well components of portfolio contribute to overall risk/return Scenario analysis • Model provide input into strategic discussions on portfolio construction/management

Figure provided for illustrative purposes.

27

GXN-1900

Examples: U.S. residential-real-estate and carbon

• In 2005, bank wholesale and consumer portfolios modeled together exhibited a high degree of exposure to a U.S. residential-real-estate factor.

• Many multi-asset-class portfolios today exhibit a high degree of exposure to a carbon-emissions factor.

• Latent-factor modeling analyzed in terms of types of firms that load on particular factors provide clues as to emerging factors.

28

GXN-1900

Principles of risk-data visualization

• Match output to use cases

– Concentration risk assessment

– Risk appetite assessment (stress testing)

– Position-level limits/allocation

• Prepare for multiple dimensions (e.g., region, sector, asset class, customer type, size)

• Incorporate drill-down capability

• Contextualize output (e.g., benchmarks, time series, scenario-based)

• Use robust statistics (e.g., median, inter-quartile, mean absolute deviation)

• Use techniques to address data difficulties (e.g., Winsorization, shrinkage)

• Target near-instantaneous rendering of decision-support output

Risk data tend to be defined by outliers

29

GXN-1900

Expected Returns vs. Volatility by Exposure Size Looks sophisticated… but is it useful?

Exposure size Exposure size

30 Data are all figurative for illustrative purposes

GXN-1900

• The next slides in this section all come from one chart that provides the ability to select different dimensions.

31 Data are all figurative for illustrative purposes

GXN-1900 Data are all figurative for illustrative purposes

Expected Return vs. Volatility… as a scatter Almost the same information as the 3D visualization

32


The same 3-D plot… introduce Size as a dimension Expected Returns vs. Volatility by Exposure Size

Highest Volatility

Highest Expected Returns

33


The same plot… introduce Color as another dimension Expected Returns vs. Volatility by Exposure Size – Sharpe Ratio as Color

High Sharpe Ratios, but small positions

OK Sharpe Ratios, and larger position

34


Drill down, if you need the numbers

35


Zoom out – aggregate by sector

Software & Computer Services

36


Zoom in – Display every position

37


Zoom in – Display every position Inspect performance clustering

High Volatility, Low Return

High Sharpe Ratio names

Return proportional to Volatility

38


So which is richer, from a data insight perspective?

39


The Loss Distribution … not very insightful

40


Zooming: The ability to filter data ranges Loss distribution

On-the-fly filters

41


Zooming: The ability to filter data ranges Log of the Loss, to eyeball the simulation error

42

GXN-1900

Tools

• Multi-dimensional

• OLAP

• Linked to real-time data sources

• Object oriented

• Data preparation tools becoming more sophisticated

Data visualization and decision-support tools are much improved

43

GXN-1900

Improve visualization independent of model complexity

• Google study (Tuch, et al., 2012) found for websites:

– Visually complex websites are less appealing

– Prototypical websites (for a given category) are more appealing

– Simpler design is rated higher

• What makes analytical output compelling and credible?

– Prototypicality: Basic mental image one’s brain creates to categorize everything with which you interact.

– Cognitive fluency: One’s brain prefers output that is easier to process.

– Mere exposure effect: Familiarity arising from repeated exposure.

– Metric balancing: Too many metrics equals no understanding.

44

GXN-1900

Minding the gap

• Mind melting

– Tables of numbers

– Complicated causal diagrams

– Words

• Mind moving

– Prudently pruned output

– Emphasis on key outcomes while recognizing underlying complexity

– Graphs

45

GXN-1900

Simplification and message dilution Trading off simplification and analytical output implication dilution

46

GXN-1900

Consilience: A frequent driver toward complexity A "jumping together” of knowledge by the linking of facts and fact-based theory across disciplines to create a common groundwork of explanation. Wilson (1999) p. 8

47

GXN-1900

Simplicity and complexity

• Simplicity should be one goal, not a hard criterion. • Usefulness should also be a goal and should be a hard criterion: Thus, simple enough, but no

simpler. This thought can lead to…

• Occam’s Razor: “Among competing hypotheses, the one with the fewest assumptions should be selected.”

– William of Ockham’s actual quote (I think): “Numquam ponenda est pluralitas sine necessitate [Plurality must never be posited without necessity]”

– Bertrand Russell’s version: "Whenever possible, substitute constructions out of known entities for inferences to unknown entities.”

• Not likely to be a global principle– important that model “suitably” explains/predicts data

– “Suitably” implies matching model with use objective

– Over time, more complex models may explain data better and open new vistas e.g., atomic theory, plate tectonics, Black-Scholes, light theory (Newton’s “simpler” particle’s versus Huygen’s waves)– bottom-up, factor-based portfolio simulations?

• Simplicity in concept may belie complexity in reality– e.g., biological evolution, construct a portfolio that finds highest return/risk

48

GXN-1900

Criteria for deciding degree of model complexity

• Ease of communicating actionable model output to quantitatively-informed executives

• Ignores extraneous information

• Balances model uncertainty and credibility of priors

• Minimizes over-fitting risk: Sample size relative to number of parameters

• Resistant to manipulation as a consequence of incentives

49

GXN-1900

Big Short: Collateralized debt obligations (CDOs)

• Simplicity in concept: Portfolio of many assets plus cash waterfall changes risk profile

• Complexity in structure: Rules in cash waterfall; over-collateralization and interest-coverage triggers

• Simplicity in risk ratings: Diversity score (binomial expansion) or Gaussian copula plus “shading” based on a committee

• Complexity in market structure: Borrowers, appraisers, mortgage brokers, commercial banks, mortgage servicers, investment bankers, fund managers, rating agencies, institutional investors, regulators (Fed, NAIC, SEC) lawyers and many consultants

• Simplicity in incentives: Money for nothing– or at the least mis-representation of actual risk relative to fees and spreads

Communication regarding CDO risk to senior executives at financial institutions was mostly not cognitively compelling.

“Real risk was not volatility; real risk was stupid investment decisions.” Lewis (2010)

50

GXN-1900

Failure of a simple CDO model

• Materially underestimated risk of mezzanine and senior tranches

• Often over-simplified waterfall rules– e.g., equity prioritized

• Unrealistic assumptions with respect to default rates

• Herd mentality on CDO ratings

• Correlation dynamics not well represented– i.e., correlation went toward 1 in crisis.

51

GXN-1900

Portfolio Simulation Sampling across the Latent Factors to simulation various risk conditions

N securities

Portfolio

Valuation Model

…

Random draws

Single Position K latent factors

Factor draws

Factor realizations Factor Betas

idiosyncratic

value of

ith security

K = 30 factors N ≈ 10k ~ 500k positions/idiosyncratic draws J ≈ 1m ~ 10m iterations

How much computing power needed?

J iterations

Distribution

Repeat J times

value of

portfolio for one iteration

equity

fixed income

other

Repeat N times

Source: State Street Global Exchange℠

Limited Access

Figure provided for illustrative purposes only.

52

GXN-1900 GXN-1900

Key points to remember

53

GXN-1900 GXN-1900

Thinking

Theories

54

GXN-1900

How to communicate in a cognitively compelling way

• Research audience… – Culture (values and biases) – Education (general and specific to communicated analytics) – Incentives (degree of departure from objectivity)

• Distill message... – Empirical vs. rational/theoretical (where is the model in this iterative evolution?) – Transparent assumptions – Frank assessment of model uncertainty – Link to audience’s narrative/values

• Use intuitive framing and strong visuals… – Relevant metaphors – Examples from different, but similar domains (e.g., health/medical) – Design compelling graphs and figures

55

GXN-1900

Key points to remember for compelling communication • Make as simple as possible/necessary– but no simpler! • Frame within a narrative • Communicate “goldilocks” content– not too obvious and not too obscure • Avoid “quantifauxcation” • Contextualize (across time and across cohorts) • Address biases: Highlight data selection concerns and explain assumptions & process • Use transparency in model estimation process to spark questions and debate • Compare output from multiple models (when possible) • Visualize data and use graphics– encourage interactive diagnostics and drill-down • Emphasize actionable insight • Educate

– Explain key components of analytical process – Teach how to understand confidence intervals (noise vs. signal)

Build on understanding: Descriptive, prescriptive and cognitive Move from analysis (breaking into components) to synthesis (re-assembling with insight)

56

GXN-1900

References 1 • Aikman, David, Piergiorgio Alessandri, Bruno Eklund, Prasanna Gai, Sujit Kapadia, Elizabeth

Martin, Nada Mora, Gabriel Sterne and Matthew Willison, 2009, “Funding liquidity risk in a quantitative model of systemic stability,” Working Paper 372, Bank of England.

• Arrow, Kenneth J. and Gerard Debreu, 1954, "Existence of an equilibrium for a competitive economy,” Econometrica 22 (3), pp. 265–290.

• Bohn, Jeffrey and Roger Stein 2009, Active Portfolio Management in Practice, Wiley. • Diaconis, Persi, 2003, “The problem of thinking too much,” Bulletin American Academy of

Science, Spring, pp. 26-38. • Fender, Ingo and John Kiff, 2004, “CDO rating methodology: Some thoughts on model risk and

its implications,” Monetary and Economic Department, BIS. • Feynman, Richard P., 1974, “Cargo cult science,” Engineering and Science, June, pp. 10-13. • Folinsbee, Kaila E; et al. 2007, "Quantitative approaches to phylogenetics; §5.2: Fount of

stability and confusion: A synopsis of parsimony in systematics,” In Winfried Henke, ed. Handbook of Paleoanthropology: Primate evolution and human origins: Volume 2, Springer, p. 168.

• Gordy, Michael B., 2003, “A risk-factor model foundation for ratings-based bank capital rules,” Journal of Financial Intermediation, 12, pp. 199-232.

• Gray, Dale and Samuel Malone, 2008, Macrofinancial Risk Analysis, Wiley. • Haidt, Jonathan, 2012, The righteous mind: Why good people are divided by politics and

religion, Pantheon Books, New York, NY. • Haldane, Andrew G., 2012, “The dog and the frisbee,” Bank of England speech. • Hamilton, James, 1994, Time Series Analysis, Princeton University Press. • Hansen, Lars P. and Thomas J. Sargent, 2015, “Four types of ignorance,” Journal of Monetary

Economics, 69, pp. 97-113.

57

GXN-1900

References 2 • Hansen, Lars P. and Thomas J. Sargent, 2007, “Recursive robust estimation and control

without commitment,” Journal of Economic Theory, 136, pp. 1-27. • Jablonka, Eva and Marion J. Lamb, 2014, Evolution in four dimensions: genetic, epigenetic,

behavioral, and symbolic variation in the history of life, MIT Press, Cambridge, MA. • Kalirai, Harvir and Martin Scheicher, 2002, “Macroeconomic Stress Testing: Preliminary Evidence for

Austria,” Financial Stability Report 3, Oesterreichische Nationalbank, 58-74. • Knight, Frank H., 1921, Risk, Uncertainty, and Profit, Hart, Schaffner & Marx; Houghton Mifflin

Company: Boston, MA. • Koomey, Jonathan, 2001, Turning numbers into knowledge: Mastering the art of problem solving,

Analytics Press, Oakland, CA. • Kuhn, Thomas S., 1977, The essential tension: Selected studies in scientific tradition change,

University of Chicago Press, Chicago. • Lewis, Michael, 2010, The big short: Inside the doomsday machine, W.W. Norton & Company: New

York, NY. • Lucas, Robert, 1977, “Understanding business cycles,” in K. Brunner and A.H. Metzler, eds.,

Stabilization of the domestic and international economy, Carnegie-Rochester Conference Series on Public Policy, 7729.

• Popper, Karl, 1972, Objective knowledge: An evolutionary approach, Clarendon Press, Oxford. • Rescher, Nicholas, 2004, “Leibniz quantitative epistemology,” Studia Leibnitiana, 36(2), pp. 210-231. • Simon, Herbert A., 1955, “A behavioral model of rational choice,” Quarterly Journal of Economics,

69(1), pp. 99-118. • Stark, Philip B., 2015, “Pay no attention to the model behind the curtain,” Working paper, UC Berkeley. • Vasicek, Oldrich A., 1997, “The distribution of loan portfolio value,” Working paper, KMV.

58

GXN-1900

Disclaimers and Important Risk Information

State Street Global Exchange℠ is a trademark of State Street Corporation (incorporated in Massachusetts) and is registered or has registrations pending in multiple jurisdictions. State Street Associates® is a research partnership between State Street Global Exchange and academia under which this document is produced. State Street Associates is a registered trademark of State Street Corporation. Use of report This document and information herein (together, the “Content”) is for informational, illustrative and/or marketing purposes only and it does not constitute investment research or investment, legal, or tax advice. The Content provided is not, nor should be construed as, any offer or solicitation to buy or sell any product, service, or securities or any financial instrument, and it does not constitute any binding contractual arrangement or commitment for State Street Corporation and its subsidiaries and affiliates (“State Street”) of any kind. The Content provided does not purport to be comprehensive nor intended to replace the exercise of a client’s own careful independent review regarding any corresponding investment or other financial decision. Distribution The Content provided is not intended for retail clients, nor is intended to be relied upon by any person or entity, and is not intended for distribution to or use by any person or entity in any jurisdiction where such distribution or use would be contrary to applicable law or regulation. No permission is granted to reprint, sell, copy, distribute, or modify the Content in any form or by any means without the prior written consent of State Street. Other Important Disclosures The Content provided has been prepared and obtained from sources believed to be reliable at the time of preparation, however it is provided “as-is” and State Street makes no guarantee, representation, or warranty of any kind including, without limitation, as to its accuracy, suitability, timeliness, merchantability, fitness for a particular purpose, non-infringement of third-party rights, or otherwise. Views and opinions expressed herein are those of the author(s) and are subject to change without notice based on market and other conditions and in any event may not reflect the views of State Street. State Street disclaims all liability, whether arising in contract, tort or otherwise, for any claims, losses, liabilities, damages (including direct, indirect, special or consequential), expenses or costs arising from or connected with the Content. The Content provided may contain certain statements that could be deemed forward-looking statements; any such statements or forecasted information are not guarantees or reliable indicators for future performance and actual results or developments may differ materially from those depicted or projected. Past performance is no guarantee of future results. Prospects, clients or counterparties should be aware of the risks of participating in trading foreign exchange, equities, fixed income or derivative instruments or in investments in non-liquid or emerging markets. Prospects, clients or counterparties should be aware that products and services outlined may put their principal and capital at risk and that diversification does not ensure a profit or guarantee against loss. Australia: This communication is made available in Australia by State Street Bank and Trust Company ABN 70 062 819 630, AFSL 239679 and is intended only for wholesale clients, as defined in the Corporations Act 2001.

59

GXN-1900

Brazil: The products in this marketing material have not been and will not be registered with the Comissão de Valores Mobiliários, the Brazilian Securities and Exchange Commission ("CVM"), and any offer of such products is not directed to the general public within the Federative Republic of Brazil ("Brazil"). The information contained in this marketing material is not provided for the purpose of soliciting investments from investors residing in Brazil and no information in this marketing material should be construed as a public offering or unauthorised distribution of the products within Brazil, pursuant to applicable Brazilian law and regulations. To our knowledge, no analyst is related to an individual who works at the issuer of the securities subject to the content of this website; no analyst or their spouse holds, directly or indirectly, securities subject to the reports on this website; no analyst or their spouse is, directly or indirectly, involved in the acquisition, sale or intermediation of securities subject to the reports on this website; no analyst or their spouse has, directly or indirectly, a financial interest in relation to the subject matter of the reports on this website; and the relevant analyst's compensation is not, directly or indirectly, influenced by the revenues arising from the business and financial transactions carried out by the entity to which is associated or otherwise related. Canada: The products and services outlined in this document are generally offered in Canada by State Street Bank and Trust Company and/or by State Street Global Markets Canada Inc. Hong Kong: This communication is made available in Hong Kong by State Street Bank and Trust Company, which accepts responsibility for its contents, and is intended for distribution to professional investors only (as defined in the Securities and Futures Ordinance). Indonesia: This communication is made available in Indonesia by State Street Bank and Trust Company and its affiliates. Neither this communication nor any copy hereof may be distributed in Indonesia or to any Indonesian citizens wherever they are domiciled or to Indonesian residents except in compliance with applicable Indonesian capital market laws and regulations. This communication is not an offer of securities in Indonesia. Any securities referred to in this communication have not been registered with the Capital Market and Financial Institutions Supervisory Agency (BAPEPAM-LK) pursuant to relevant capital market laws and regulations, and may not be offered or sold within the territory of the Republic of Indonesia or to Indonesian citizens through a public offering or in circumstances which constitute an offer within the meaning of the Indonesian capital market law and regulations. Israel: This communication is made available in Israel by State Street Global Markets International Limited, which is not licensed under Israel’s Regulation of Investment Advice, Investment Marketing and Portfolio Management Law, 1995. This communication may only be distributed to or used by investors in Israel which are “eligible clients” as listed in the First Schedule to Israel’s Regulation of Investment Advice, Investment Marketing and Portfolio Management Law 1995. Japan: This communication is made available in Japan by State Street Global Markets (Japan) which is regulated by the Financial Services Agency of Japan as a financial instruments firm. Malaysia: This communication is made available in Malaysia by State Street Global Markets International Limited (“SSGMIL”) which is authorised and regulated by the United Kingdom’s Financial Conduct Authority. SSGMIL is not licensed within or doing business within Malaysia and the activities that are being discussed are carried out off-shore. The written materials do not constitute, and should not be construed as constituting: 1) an offer or invitation to subscribe for or purchase securities or futures in Malaysia or the making available of securities or futures for purchase or subscription in Malaysia; 2) the provision of investment advice concerning securities or futures; or 3) an undertaking by SSGMIL to manage the portfolio of securities or futures contracts on behalf of other persons. Mexico: This communication is distributed by State Street Bank and Trust Company and its affiliates from outside Mexico. State Street is not authorized to act as an investment advisor in Mexico or as a regulated entity under Mexican law. Any products and services outlined herein will be provided from outside Mexico exclusively. New Zealand: This communication is made available in New Zealand by State Street Bank and Trust Company, which accepts responsibility for its contents, and is intended for distribution only to “wholesale clients” as defined in the Financial Advisers Act 2008 (New Zealand). This communication is not intended to constitute “financial advice” (as that term is defined in the Financial Advisers Act), or “advice or assistance” (in terms of section 37(5) of the Securities Markets Act 1988 (New Zealand)), to any person.

60

GXN-1900

Oman: This communication is made available in Oman by State Street Bank and Trust Company and its affiliates. The information contained in this communication is for information purposes and does not constitute an offer for the sale of foreign securities in Oman or an invitation to an offer for the sale of foreign securities. State Street is neither a bank or financial services provider registered to undertake business in Oman and is neither regulated by the Central Bank of Oman nor the Capital Market Authority. This document is confidential and is intended solely for the information of the person to whom it has been delivered. No representation or warranty is given as to the achievement or reasonableness of any research material contained in this communication. Nothing contained in this communication report is intended to constitute Omani investment, legal, tax, accounting, investment or other professional advice. Qatar: This communication is made available in Qatar by State Street Bank and Trust Company and its affiliates. The information in this communication has not been reviewed or approved by the Qatar Central Bank, the Qatar Financial Markets Authority or the Qatar Financial Centre Regulatory Authority, or any other relevant Qatari regulatory body. Singapore: This communication is made available in Singapore by State Street Bank and Trust Company, Singapore Branch (“SSBTS”), which holds a wholesale bank license by the Monetary Authority of Singapore. In Singapore, this communication is only distributed to accredited, institutional investors as defined in the Singapore Financial Advisers Act (“FAA”). Note that SSBTS is exempt from Sections 27 and 36 of the FAA. When this communication is distributed to overseas investors as defined in the FAA, note that SSBTS is exempt from Sections 26, 27, 29 and 36 of the FAA. State Street Bank and Trust Company Limited, Singapore Branch’s Unique Entity Number is T01FC6134G. South Africa: The products and services outlined in this document are made available in South Africa through either State Street Global Markets International Limited or State Street Bank and Trust Company, both of which are authorized in South Africa under the Financial Advisory and Intermediary Services Act, 2002 as a Category I Financial Services Provider; FSP No. 42823 and 42671 respectively. South Korea: This communication is made available in South Korea by State Street Bank and Trust Company and its affiliates, which accept responsibility for its contents, and is intended for distribution to professional investors only. State Street Bank and Trust Company is not licensed to undertake securities business within South Korea, and any activities related to the content hereof will be carried out off-shore and only in relation to off-shore non-South Korea securities. Taiwan: This communication is made available in Taiwan by State Street Bank and Trust Company and its affiliates, which accept responsibility for its contents, and is intended for distribution to professional investors only. State Street Bank and Trust Company is not licensed to undertake securities business within Taiwan, and any activities related to the content hereof will be carried out off-shore and only in relation to off-shore non-Taiwan securities. Turkey: This communication is made available in Turkey by State Street Bank and Trust Company and its affiliates. The information included herein is not investment advice. Investment advisory services are provided by portfolio management companies, brokers and banks without deposit collection licenses within the scope of the investment advisory agreements to be executed with clients. Any opinions and statements included herein are based on the personal opinions of the commentators and authors. These opinions may not be suitable to your financial status and your risk and return preferences. Therefore, an investment decision based solely on the information herein may not be appropriate to your expectations. United Arab Emirates: This communication is made available in United Arab Emirates by State Street Bank and Trust Company and its affiliates. This communication does not, and is not intended to, constitute an offer of securities anywhere in the United Arab Emirates and accordingly should not be construed as such. Nor does the addressing of this research publication to you constitute, or is intended to constitute, the carrying on or engagement in banking, financial and/or investment consultation business in the United Arab Emirates under the rules and regulations made by the Central Bank of the United Arab Emirates, the Emirates Securities and Commodities Authority or the United Arab Emirates Ministry of Economy. Any public offer of securities in the United Arab Emirates, if made, will be made pursuant to one or more separate documents and only in accordance with the applicable laws and regulations. Nothing contained in this communication is intended to endorse or recommend a particular course of action or to constitute investment, legal, tax, accounting or other professional advice. Prospective investors should consult with an appropriate professional for specific advice rendered on the basis of their situation. Further, the information contained within this communication is not intended to lead to the conclusion of any contract of whatsoever nature within the territory of the United Arab Emirates. This communication has been forwarded to you solely for your information, and may not be reproduced or passed on, directly or indirectly, to any other person or published, in whole or in part, for any purpose. This communication is addressed only to persons who are professional, institutional or otherwise sophisticated investors.

61

GXN-1900

United Kingdom and European Union: The products and services outlined herein are only offered to professional clients or eligible counterparties through State Street Bank and Trust Company, London Branch, authorised and regulated by Federal Reserve Board, authorised and subject to limited regulation by the Prudential Regulation Authority and subject to regulation by the Financial Conduct Authority and/or State Street Global Markets International Limited, authorised and regulated by the Financial Conduct Authority Details about the extent of our regulation by the Financial Conduct Authority and Prudential Regulation Authority are available from us on request. Please note that certain foreign exchange business (spot and certain forward transactions) are not regulated by the Financial Conduct Authority.

United States and Latin America: The products and services outlined in this document are generally offered in the United States and in Latin America by State Street Bank and Trust Company and/or by State Street Global Markets, LLC.

Please contact your sales representative for further information.

© 2016, State Street Corporation, All rights reserved.

62

GXN-1900

Appendix

63

GXN-1900

Defining complexity

• Epistemic: Hard to understand conceptually

• Computational: Algorithm hard to understand and/or implement

• Dynamic: System changes over time– sometimes as a function of itself or its users

• Human minds are memory and predicting mechanisms: Does the complexity arise from quantity of “insight chunks” that need to be remembered or from the quantity of steps/inter-relationships in the predictive model?

• Simple heuristics may arise from highly complex models/systems. Complexity may arise from incorporating nuances into interpretations/decisions based on a given heuristic.

64 Appendix

GXN-1900

Three challenges for communicating complexity

• Cognitive: Complexity and uncertainties associated with managing institutional (e.g., pension funds, sovereign wealth funds, insurance companies, commercial banks) financial portfolios approaches (and can exceed) human cognitive capacity.

• Coherency: Communicating across many heterogeneous groups of individuals (e.g., executives, regulators, portfolio managers, risk managers, auditors, etc.) leads to a larger number of narrow “categories” or specializations as the data/analytics/systems/outputs are assembled to produce the decision-support information; this hyper-specialization minimizes misunderstandings in each micro-context at the cost of fragmented macro-perception and increased difficulty for decision-makers to develop a coherent macro-vision of a portfolio’s risk/return profile.

• Contingency: Model outputs depend on a cascade of assumptions which may be contingent on behavior that dynamically changes as macro-economic regimes change– in some cases, an institution’s portfolio decisions may even have a feedback effect creating even more (non-linear induced) complexity.

65 Appendix

GXN-1900

Shift in risk modeling favors more model complexity

Normal Distribution Assumptions

Linear estimation is good enough

Current Mainstream Paradigm (examples)

Focus on 2nd moment Linear Regressions

Bias toward tractable “closed form”

solutions

Examples FastData™

BigComputation™

Machine Learning

Empirical orientation

Enablers

Skewed, Fat-tailed distributions

Focus on Skewness and Tail, generated by

simulations Deep Learning

Need to recognize non-linearities

Empirical approach with recognition of

non-linearity

Examples

New Approaches

Source: State Street Global Exchange℠ 66 Appendix

GXN-1900

Risk versus uncertainty

• Frank Knight (Knight, 1921): "Uncertainty must be taken in a sense radically distinct from the familiar notion of risk, from which it has never been properly separated.... The essential fact is that 'risk' means in some cases a quantity susceptible of measurement, while at other times it is something distinctly not of this character; and there are far-reaching and crucial differences in the bearings of the phenomena depending on which of the two is really present and operating.... It will appear that a measurable uncertainty, or 'risk' proper, as we shall use the term, is so far different from an unmeasurable one that it is not in effect an uncertainty at all.”

• Risk is known unknowns i.e., data generating process, inter-relationships arising from model structure, parameters imply known distributions

• Knightian uncertainty is unknown unknowns i.e., data generating process, model structure and parameters are unknown

67 Appendix

GXN-1900

Model risk

• Mis-specify data-generating process: Fail to estimate the “true” distribution due to the fact that the underlying process differs materially from the model’s process assumptions.

– Second-order assessment of misspecification also important.

– What is range of wrong estimates?

– Are out-of-sample tests showing under-estimates in times of stress?

• Mis-estimate parameters: While the assumed data-generating process may be reasonable, still fail to estimate “true” distribution based on parameterization problems.

– Are there enough data points to credibly estimate parameters?

– Can model structure be used to infer tail events without requisite data?

68 Appendix

GXN-1900

Types of ignorance Hansen and Sargent (2015)

1. “Bayesian decision maker” does not know β, but trusts prior prob. distribution 2. “Robust Bayesian decision maker” does not trust prior distribution for response

coefficient β; but uses operators to twist prior distributions to generate conservative est. 3. “Robust decision maker” uses a multiplier or constraint preferences to express doubt

regarding probability distribution of W conditional on X and U. 4. “Robust decision maker” asserts ignorance as in 3 by adjusting an entropy penalty to

Make model robust to particular alternative probability models.

“The trouble with most folks isn’t so much their ignorance, as knowing so many things that ain’t so.” Josh Billings related by Friedman (1965)

69 Appendix

GXN-1900

Hume’s problem

• When is it reasonable to think the future will be like the past?

• Rephrase– what part of a model’s/system’s/algorithm’s structure will operate in the future in the same way it has in the past?

– Means are difficult to estimate and historical observation may be a poor guide

– Volatility may be easier (than means) to estimate, but still may change so much that historical observation is likely to be a poor guide.

– Underlying correlation structure may change-- but not as much as volatility

– Co-skew?

– Co-kurtosis?

How much can/should non-linear-process-inducing feedback loops and tipping points be included in a model and can historical observation help?

How does model complexification affect strategies for communicating to quantitatively-informed business executives?

70 Appendix

GXN-1900

How do we know a portfolio-risk system/model works?

“Science is no inexorable march to truth, mediated by the collection of objective information and the destruction of ancient superstition. Scientists, as ordinary human beings, unconsciously reflect in their theories the social and political constraints of their times.” -- Stephen Jay Gould

Karl Popper’s process applied to portfolio-risk analysis:

• Problem situation: “Risk-appetite consistent allocation” • Tentative theories: “Hypothesize scenarios” • Error elimination: “Stress the stress test i.e., simulate”

However, falsifying every scenario is not possible; further, Gould argues to beware theory-laden analyses.

71 Appendix

GXN-1900

Manage vagueness to communicate more compellingly

“Vagueness blurring and imprecision effectively provide a protective shell to guard [a] statement against a charge of falsity.” (Rescher, p. 222) • Distinguish overarching “truth claim” from (possibly inaccessible) “true details.” • Avoid recommendation-rejection skepticism by…

– Combating cognitive myopia with context – Clarifying certainty with respect to aspects of output – Separating critical truth-claim components from color-commentary details

• Paradoxically, judicious omission of details (sometime unavoidable) may produce a clearer, more compelling message. Finding the “goldilocks” balance is not always straightforward.

• Examples – Truth claim: I grew up in San Francisco. This vague statement masks the true detail

that I grew up in Danville– loosely part of metropolitan San Francisco. – Truth claim: Portfolio is likely to lose 10% or more of its value in the next structural

recession. This vague statement masks details that a simulation algorithm was used to determine the frequency of losses beyond 10% and the modeler assumes a structural recession occurs 3% (1-in-33 years) of the time, which marks the amount of loss– in this case, 10% of starting portfolio value.

72 Appendix

GXN-1900

Questioning the entire modeling enterprise

•Preproducibility is a prerequisite for attempting to reproduce a result: –Provide an adequate description of an experiment

or analysis for the work to be re-undertaken. –Provide documentation, openness, and

communication.

•Quantifauxcation is to assign a meaningless number, then pretend that since it’s quantitative, it’s meaningful. Usually involves some combination of data, pure invention, ad-hoc models, inappropriate statistics, and logical lacunae.

(From Stark 2015)

73 Appendix

jeff bohn - state street at the chief analytics officer forum west coast

Technology