fix what matters: bsidesdetroit 2014
DESCRIPTION
Heartbleed has exposed a weakness in the way we assess risk in information security. We use archaic methods and ignore new data when assessing what to fix, and we rarely go back to see what new data is telling us. In this talk, we explore new, data-driven approaches to vulnerability management.TRANSCRIPT
![Page 1: Fix What Matters: BSidesDetroit 2014](https://reader033.vdocuments.us/reader033/viewer/2022052822/554beb77b4c9055a368b4e79/html5/thumbnails/1.jpg)
Fix What Matters: !
Why CVSS Sucks And How To
Do Better
![Page 2: Fix What Matters: BSidesDetroit 2014](https://reader033.vdocuments.us/reader033/viewer/2022052822/554beb77b4c9055a368b4e79/html5/thumbnails/2.jpg)
Once Jailbroke an Iphone 3G
Michael Roytman
Proud Owner of Remote Controlled AirplaneRecently a Naive Grad Student
Data Scientist, Risk I/ODoes Not Wake Up Before 11 CST
qualifications:
![Page 3: Fix What Matters: BSidesDetroit 2014](https://reader033.vdocuments.us/reader033/viewer/2022052822/554beb77b4c9055a368b4e79/html5/thumbnails/3.jpg)
15x better than CVSS
![Page 4: Fix What Matters: BSidesDetroit 2014](https://reader033.vdocuments.us/reader033/viewer/2022052822/554beb77b4c9055a368b4e79/html5/thumbnails/4.jpg)
Probability A Vuln Having Property X Has Observed Breaches
Random Vuln
CVSS 10
Exploit DB
Metasploit
MSP+EDB
0.0 0.1 0.2 0.2 0.3
![Page 5: Fix What Matters: BSidesDetroit 2014](https://reader033.vdocuments.us/reader033/viewer/2022052822/554beb77b4c9055a368b4e79/html5/thumbnails/5.jpg)
PART 1: !
YOU SUCK AT YOUR JOB
!
(and don’t even know it yet)
![Page 6: Fix What Matters: BSidesDetroit 2014](https://reader033.vdocuments.us/reader033/viewer/2022052822/554beb77b4c9055a368b4e79/html5/thumbnails/6.jpg)
Why Are We Here?
Empirical Failures of CVSSProper Remediation Frameworks
CVSS SUCKS
Analytical Failures of CVSS
(+Data Driven Alternatives)
![Page 7: Fix What Matters: BSidesDetroit 2014](https://reader033.vdocuments.us/reader033/viewer/2022052822/554beb77b4c9055a368b4e79/html5/thumbnails/7.jpg)
Remove the Threat
RemediationAccept the Risk
Repair the Vulnerability
![Page 8: Fix What Matters: BSidesDetroit 2014](https://reader033.vdocuments.us/reader033/viewer/2022052822/554beb77b4c9055a368b4e79/html5/thumbnails/8.jpg)
C(ommon) V(ulnerability) S(coring) S(ystem)
“CVSS is designed to rank information system vulnerabilities”
Exploitability/Temporal (Likelihood)
Impact/Environmental (Severity)
The Good: Open, Standardized Scores
![Page 9: Fix What Matters: BSidesDetroit 2014](https://reader033.vdocuments.us/reader033/viewer/2022052822/554beb77b4c9055a368b4e79/html5/thumbnails/9.jpg)
“It is a capital mistake to theorize before one has data.
!
!
!
Insensibly, one begins to twist facts to suit theories, instead of
theories to suit facts.”
![Page 10: Fix What Matters: BSidesDetroit 2014](https://reader033.vdocuments.us/reader033/viewer/2022052822/554beb77b4c9055a368b4e79/html5/thumbnails/10.jpg)
FAIL 1: A Priori Modeling“Following up my previous email, I have tweaked my equation to try to achieve better separation between adjacent scores and to have CCC have a perfect (storm) 10 score...There is probably a way to optimize the problem numerically, but doing trial and error gives one plausible set of parameters...except that the scores of 9.21 and 9.54 are still too close together. I can adjust x.3 and x.7 to get a better separation . . .”
![Page 11: Fix What Matters: BSidesDetroit 2014](https://reader033.vdocuments.us/reader033/viewer/2022052822/554beb77b4c9055a368b4e79/html5/thumbnails/11.jpg)
F2: Data FundamentalismDon’t Ignore What a Vulnerability Is: Creation Bias http://blog.risk.io/2013/04/data-fundamentalism/ !
Jerico/Sushidude @ BlackHat https://www.blackhat.com/us-13/briefings.html#Martin
!
Luca Allodi - CVSS DDOS http://disi.unitn.it/~allodi/allodi-12-badgers.pdf
![Page 12: Fix What Matters: BSidesDetroit 2014](https://reader033.vdocuments.us/reader033/viewer/2022052822/554beb77b4c9055a368b4e79/html5/thumbnails/12.jpg)
F2: Data FundamentalismSince 2006 Vulnerabilities have declined by 26 percent.” http://csrc.nist.gov/groups/SNS/rbac/documents/vulnerability-trends10.pdf !
!
The total number of vulnerabilities in 2013 is up 16 percent so far when compared to what we saw in the same time period in 2012. ” http://www.symantec.com/content/en/us/enterprise/other_resources/b-intelligence_report_06-2013.en-us.pdf
![Page 13: Fix What Matters: BSidesDetroit 2014](https://reader033.vdocuments.us/reader033/viewer/2022052822/554beb77b4c9055a368b4e79/html5/thumbnails/13.jpg)
F3: Stochastic Ignorance
Attackers Change Tactics Daily
![Page 14: Fix What Matters: BSidesDetroit 2014](https://reader033.vdocuments.us/reader033/viewer/2022052822/554beb77b4c9055a368b4e79/html5/thumbnails/14.jpg)
F3: Stochastic Ignorance
![Page 15: Fix What Matters: BSidesDetroit 2014](https://reader033.vdocuments.us/reader033/viewer/2022052822/554beb77b4c9055a368b4e79/html5/thumbnails/15.jpg)
Empirical Failures of CVSSObjective: Remediate the riskiest vulnerabilities
Constraint: Can’t measure impact/priority
Need:
MOAR DATA!!!
![Page 16: Fix What Matters: BSidesDetroit 2014](https://reader033.vdocuments.us/reader033/viewer/2022052822/554beb77b4c9055a368b4e79/html5/thumbnails/16.jpg)
Repair the Vulnerability
![Page 17: Fix What Matters: BSidesDetroit 2014](https://reader033.vdocuments.us/reader033/viewer/2022052822/554beb77b4c9055a368b4e79/html5/thumbnails/17.jpg)
I Love It When You Call Me Big Data50,000,000 Live Vulnerabilities
1,500,000 Assets
2,000 Organizations
![Page 18: Fix What Matters: BSidesDetroit 2014](https://reader033.vdocuments.us/reader033/viewer/2022052822/554beb77b4c9055a368b4e79/html5/thumbnails/18.jpg)
I Love It When You Call Me Big Data
3,000,000 Breaches
![Page 19: Fix What Matters: BSidesDetroit 2014](https://reader033.vdocuments.us/reader033/viewer/2022052822/554beb77b4c9055a368b4e79/html5/thumbnails/19.jpg)
Baseline AllthethingsProbability (You Will Be Breached On A Particular Open Vulnerability)?
=(Open Vulnerabilities | Breaches Occurred On Their CVE) /(Total Open Vulnerabilities)
2%
![Page 20: Fix What Matters: BSidesDetroit 2014](https://reader033.vdocuments.us/reader033/viewer/2022052822/554beb77b4c9055a368b4e79/html5/thumbnails/20.jpg)
Probability A Vuln Having Property X Has Observed Breaches
RANDOM VULN
CVSS 10
CVSS 9
CVSS 8
CVSS 6
CVSS 7
CVSS 5
CVSS 4
Has Patch
0.000 0.010 0.020 0.030 0.040
![Page 21: Fix What Matters: BSidesDetroit 2014](https://reader033.vdocuments.us/reader033/viewer/2022052822/554beb77b4c9055a368b4e79/html5/thumbnails/21.jpg)
PART 2: !
FIX WHAT MATTERS
![Page 22: Fix What Matters: BSidesDetroit 2014](https://reader033.vdocuments.us/reader033/viewer/2022052822/554beb77b4c9055a368b4e79/html5/thumbnails/22.jpg)
Empirical Failures of CVSSObjective: Remediate the riskiest vulnerabilities
Constraint: Can’t measure impact/priority
Need:
MOAR DATA!!!
![Page 23: Fix What Matters: BSidesDetroit 2014](https://reader033.vdocuments.us/reader033/viewer/2022052822/554beb77b4c9055a368b4e79/html5/thumbnails/23.jpg)
Proper Framework
Know which vulnerabilities put you most at risk.
![Page 24: Fix What Matters: BSidesDetroit 2014](https://reader033.vdocuments.us/reader033/viewer/2022052822/554beb77b4c9055a368b4e79/html5/thumbnails/24.jpg)
![Page 25: Fix What Matters: BSidesDetroit 2014](https://reader033.vdocuments.us/reader033/viewer/2022052822/554beb77b4c9055a368b4e79/html5/thumbnails/25.jpg)
![Page 26: Fix What Matters: BSidesDetroit 2014](https://reader033.vdocuments.us/reader033/viewer/2022052822/554beb77b4c9055a368b4e79/html5/thumbnails/26.jpg)
![Page 27: Fix What Matters: BSidesDetroit 2014](https://reader033.vdocuments.us/reader033/viewer/2022052822/554beb77b4c9055a368b4e79/html5/thumbnails/27.jpg)
![Page 28: Fix What Matters: BSidesDetroit 2014](https://reader033.vdocuments.us/reader033/viewer/2022052822/554beb77b4c9055a368b4e79/html5/thumbnails/28.jpg)
![Page 29: Fix What Matters: BSidesDetroit 2014](https://reader033.vdocuments.us/reader033/viewer/2022052822/554beb77b4c9055a368b4e79/html5/thumbnails/29.jpg)
![Page 30: Fix What Matters: BSidesDetroit 2014](https://reader033.vdocuments.us/reader033/viewer/2022052822/554beb77b4c9055a368b4e79/html5/thumbnails/30.jpg)
![Page 31: Fix What Matters: BSidesDetroit 2014](https://reader033.vdocuments.us/reader033/viewer/2022052822/554beb77b4c9055a368b4e79/html5/thumbnails/31.jpg)
Uh, Sports?
Opposing Teams, Specific Players
Gameplay
Scouting Reports, Gametape
Roster, Player Skills
Learning from Losing
![Page 32: Fix What Matters: BSidesDetroit 2014](https://reader033.vdocuments.us/reader033/viewer/2022052822/554beb77b4c9055a368b4e79/html5/thumbnails/32.jpg)
InfoSec?
![Page 33: Fix What Matters: BSidesDetroit 2014](https://reader033.vdocuments.us/reader033/viewer/2022052822/554beb77b4c9055a368b4e79/html5/thumbnails/33.jpg)
Defend Like You’ve Done It Before
Groups, Motivations
Exploits
Vulnerability Definitions
Asset Topology, Actual Vulns on System
Learning from Breaches
![Page 34: Fix What Matters: BSidesDetroit 2014](https://reader033.vdocuments.us/reader033/viewer/2022052822/554beb77b4c9055a368b4e79/html5/thumbnails/34.jpg)
Work With What You’ve Got:
Akamai, Safenet
ExploitDB, Metasploit
NVD, MITRE
![Page 35: Fix What Matters: BSidesDetroit 2014](https://reader033.vdocuments.us/reader033/viewer/2022052822/554beb77b4c9055a368b4e79/html5/thumbnails/35.jpg)
Alternatives
![Page 36: Fix What Matters: BSidesDetroit 2014](https://reader033.vdocuments.us/reader033/viewer/2022052822/554beb77b4c9055a368b4e79/html5/thumbnails/36.jpg)
Probability A Vuln Having Property X Has Observed Breaches
Random Vuln
CVSS 10
Exploit DB
Metasploit
MSP+EDB
0.0 0.1 0.2 0.2 0.3
![Page 37: Fix What Matters: BSidesDetroit 2014](https://reader033.vdocuments.us/reader033/viewer/2022052822/554beb77b4c9055a368b4e79/html5/thumbnails/37.jpg)
Be Better Than The Gap
![Page 38: Fix What Matters: BSidesDetroit 2014](https://reader033.vdocuments.us/reader033/viewer/2022052822/554beb77b4c9055a368b4e79/html5/thumbnails/38.jpg)
I Love It When You Call Me Big Data
!
Spray and Pray => 2% !
CVSS 10 => 4% !
Metasploit + ExploitDB => 30%
![Page 39: Fix What Matters: BSidesDetroit 2014](https://reader033.vdocuments.us/reader033/viewer/2022052822/554beb77b4c9055a368b4e79/html5/thumbnails/39.jpg)
Holler!www.risk.io@mroytman