smu seminar 2014_03_26 v3

Fundamentals Matter – A Brief Introduction to Risk Analysis for

Information Security Southern Methodist University, March 26, 2014

Heather Goodnight, President

Patrick Florer, CTO Risk Centric Security, Inc.

www.riskcentricsecurity.com

Authorized reseller of ModelRisk from Vose Software

Risk Centric Security, Inc. Confidential and Proprietary . Copyright © 2014 Risk Centric Security, Inc . All rights reserved.

Risk Analysis for the 21st Century®

http://www.riskcentricsecurity.com/

• Introductions • What we are going to talk about

o Why Fundamentals Matter / Current State o Definitions

• Risk and the Risk Landscape • Possibility and Probability • Measurement • Variability and Uncertainty • Precision vs. Accuracy • Scales of Measurement: Qualitative vs. Quantitative • Not Enough Data • Monte Carlo Simulation • Modeling Expert Opinion and PERT distributions

Agenda


Heather Goodnight is an accomplished Global Sales and Business Development Consultant. Over the years, her unique, practical insight into problems of risk and opportunity have provided important guidance for organizations both large and small. She is a cofounder of Risk Centric Security and currently serves as President of the Corporation. In 2010, she was appointed to the RIM Council (Responsible Information Council) of the Ponemon Institute. In addition to her role at Risk Centric Security, she serves as Business Development Manager at Triumfant, Inc., a vendor of advanced anti-malware products. Patrick Florer has worked in information technology for almost 35 years. For 17 years, he worked a parallel track in medical outcomes research, analysis, and the creation of evidence-based guidelines for medical treatment. His roles have included IT operations, programming, and systems analysis. From 1986 until now, he has worked as an independent consultant, helping customers with strategic development, analytics, risk analysis, and decision analysis. He is a cofounder of Risk Centric Security and currently serves as Chief Technology Officer. He is a member of the Ponemon Institute RIM council. In 2012, he was appointed Distinguished Fellow of the Ponemon Institute.

Introductions


The Current State of Confusion … .


ROI IRR EPS

EMV EBITDA

≠

Often leads to this …


What is Risk?

Risk Centric Security, Inc. Confidential and Proprietary. Copyright © 2014 Risk Centric Security, Inc . All rights reserved.

What Risk Isn’t!


Vulnerability Threat

Risk = Frequency x Impact


Frequency

Impact

Risk

Risk and Opportunity


Possibility and Probability: Possibility


Possibility and Probability: Probability


What is a Measurement?


Properties of Measurement


Validity Reproducibility

Detail

Sources of Error in Measurement?


Random Error

Errors from Bias

Variability and Uncertainty


Variability

Uncertainty

Precision and Accuracy


Scales of Measurement


Qualitative Quantitative

Qualitative Scales


Nominal/Categorical

Interval Ordinal

HIGH - Red MEDIUM - Orange LOW - Green

First, Second, Third … On a scale of …

Quantitative/Ratio Scales


1, 2, 3, 4, 5, 6, … n

Problems with Qualitative Scales


My Scale High Medium Low Red Orange Green

Your Scale High Medium Low Red Orange Yellow Green

(RED – GREEN + MEDIUM) / Somewhat Likely = ???

Mismatched Scales

Meaningless Calculations

Assessor Disagreements

Problems with Qualitative Scales


Boundary Problems

$2.5M Loss Exposure = Moderate = Yellow $2.5M Loss Exposure = Moderate = Yellow $2.5M Loss Exposure = Moderate = Yellow $7.5M Aggregate Loss Exposure = not so Moderate !

Issues with Loss of Information

Quantitative Scales


2 + 2 - 1 = 3 360 * 10 = 3,600 Sqrt(25) = 5 f(x) = y etc.

Qualitative Methods - Problems Difficulty with arithmetic and statistical operations

From ISO 17999


Qualitative Methods - Problems


Qualitative Methods - Problems

On a scale of 1 to 5, where 1 = least and 5 = most,

please rate … Likert scale (From Wikipedia, the free encyclopedia)

When responding to a Likert questionnaire item, respondents specify their level of agreement or disagreement … In so doing, Likert scaling assumes that distances on each item are equal …


Data


Good Data Bad Data

Big Data

Little Data

How much data is enough data?


How do I get to the mall?

How do we build this?

vs.

Data from Calibrated Estimates


More often than you might think, the data we have to work with comes from Subject Matter Experts (SME’s). How can we improve the accuracy of these SME’s – to a 90% confidence level? With calibration. Example: How much does an iPhone 5s weigh?

Monte Carlo Simulation


The average = $12,500

$2,500 $12,500 $32,000

The range is:

The distributions are:

Monte Carlo Simulation


The Beta Pert Calculator

Minimum: What is the least or lowest (best or worst) numerical estimate that you believe to be reasonable? This will be the smallest number you come up with.

Most Likely:

What is the most likely or most probable numerical estimate in your opinion? This number must fall between the minimum and maximum. It may equal either the minimum or the maximum, but should not equal both



Maximum:

What is the greatest or highest (best or worst) numerical estimate that you believe to be reasonable? Note that “best” or “worst” case estimates could be either minimum or maximum values, depending upon the scenario. In a risk / loss exposure scenario, lower is better, so the minimum represents the lowest loss, or best outcome. The maximum represents the highest loss, or worst outcome. In a sales or opportunity scenario, it’s the reverse: lower is not better, so the minimum represents the worst case. Higher is better, so the maximum represents the best case.



Confidence: On a scale that includes “Very Low”, “Low”, “Average”, “High”, and “Very High”, how confident are you in the accuracy of your estimates? This parameter controls the sampling around the most likely value, and thereby also controls the height of the histogram or slope of the cumulative plot.

For most analyses, using “Average” for the confidence parameter works well. In this instance, “Average” really means having no strong feeling about the matter – being evenly divided between under-confidence and over-confidence.


The Beta Pert Calculator Percentile Tables


The Beta Pert Calculator Percentile Tables

1% of values are <= 10,044 and 99% are > 10,044 10% of values are <= 11,120 and 90% are > 11,120 20% of values are <= 11,658 and 80% are > 11,658 50% of values are <= 13,025 and 50% are > 13,025

The 50th percentile has another name - it’s called the Median.

The Median is the mid-point in a list of values - half of the values in the list are less and half are greater than the Median.


The Beta Pert Calculator Histogram


The Beta Pert Calculator Cumulative Plot


Thank you !

Heather Goodnight President and Cofounder

Patrick Florer CTO and Co-founder

Risk Centric Security, Inc [email protected]

214.828.1172

Authorized reseller of ModelRisk from Vose Software


Risk Analysis for the 21st Century ®

mailto:[email protected]

”We don’t have enough data!” - Sources

Open Security Foundation: datalossdb and osvdb http://www.opensecurityfoundation.org/

Office of Inadequate Security: http://www.databreaches.net/

Identity Theft Resource Center: http://www.idtheftcenter.org/

ISACA: www.isaca.org

ISSA: www.issa.org


http://www.opensecurityfoundation.org/

http://www.databreaches.net/

http://www.idtheftcenter.org/

http://www.isaca.org/

http://www.issa.org/


Mitre Corporation: www.mitre.org

OWASP: http://owasp.com/index.php/Main_Page

Privacy Rights Clearing House: http://www.privacyrights.org/ SANS: www.sans.org The Ponemon Institute: www.ponemon.org


http://www.mitre.org/

http://owasp.com/index.php/Main_Page

http://www.privacyrights.org/

http://www.sans.org/

http://www.ponemon.org/

”We don’t have enough data!” - Sources Conference procedings: Black Hat, RSA, Source Conferences, BSides

Internet tools:

Search engines: Google, Bing, Yahoo, Ask.com

Trend Analyzers:

Google trends: http://www.google.com/trends

Twitter Trends: www.trendistic.com

Amazon: http://www.metricjunkie.com/


http://www.google.com/trends

http://www.trendistic.com/

http://www.metricjunkie.com/

http://www.metricjunkie.com/


Securitymetrics.org – mailing list

Society of Information Risk Analysts (SIRA)

Books: How to Measure Anything – Hubbard The Failure of Risk Management – Hubbard Risk Analysis: A Quantitative Guide – Vose Clinical Epidemiology and Biostatistics – Kramer Data-Driven Security: Analysis, Visualization and Dashboards – Jacobs and Rudis


smu seminar 2014_03_26 v3

Business

accuracy risk centric

cto risk centric security

risk analysis

problems of risk

risk isnt

risk landscape possibility

frequency impact risk

accuracy scales of measurement