Download - Security Metrics in Practice Development of a Security Metric System to Rate Enterprise Software
![Page 1: Security Metrics in Practice Development of a Security Metric System to Rate Enterprise Software](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816341550346895dd3d0a3/html5/thumbnails/1.jpg)
Security Metrics in Practice Development of a Security Metric System
to Rate Enterprise Software
Brian Chess [email protected] DeQuan Lee [email protected]
![Page 2: Security Metrics in Practice Development of a Security Metric System to Rate Enterprise Software](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816341550346895dd3d0a3/html5/thumbnails/2.jpg)
Overview
Background: Java Open ReviewWhy Measure?What Can We Measure?What Should We Measure?The Whole Picture
![Page 3: Security Metrics in Practice Development of a Security Metric System to Rate Enterprise Software](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816341550346895dd3d0a3/html5/thumbnails/3.jpg)
Java Open Review
http://opensource.fortifysoftware.com/Over 130+ Open Source Projects74+ millions of lines scanned weekly995 open source defects365 open source bugs *fixed*Responsible for fixes in:
AzureusBlojsom (OS X Server’s blogging software)JforumGroovy on Rails
![Page 4: Security Metrics in Practice Development of a Security Metric System to Rate Enterprise Software](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816341550346895dd3d0a3/html5/thumbnails/4.jpg)
Java Open Review (cont.)
Reviews Open Source Java projects for
Security defectsQualitfy defects
Utilizes Fortify SCA and Findbugs static analysis toolsAllows remote developers to collaborate on security audit process
![Page 5: Security Metrics in Practice Development of a Security Metric System to Rate Enterprise Software](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816341550346895dd3d0a3/html5/thumbnails/5.jpg)
Next Logical Step
Users immediately asked “which is better?”Which project is more secure?
Gathered lots of data over several classes of applicationsCustomers wanted to know how to choose between projects that fulfilled the same function
![Page 6: Security Metrics in Practice Development of a Security Metric System to Rate Enterprise Software](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816341550346895dd3d0a3/html5/thumbnails/6.jpg)
No two projects are alike…
![Page 7: Security Metrics in Practice Development of a Security Metric System to Rate Enterprise Software](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816341550346895dd3d0a3/html5/thumbnails/7.jpg)
Why Measure?
Determine which development group produces more secure softwareWhich software provider should I use?In what way is my security changing?What happens when I introduce this component into my environment?
![Page 8: Security Metrics in Practice Development of a Security Metric System to Rate Enterprise Software](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816341550346895dd3d0a3/html5/thumbnails/8.jpg)
No risk metric for you!
Risk assessments need:Threats, Vulnerabilities, and Controls orEvent Probability and Asset Value
Analyzing the source code only uncovers vulnerabilitiesRisk information should be derived by the consumer of the softwareLacking complete information, we steer clear of risk determinationsWe can’t answer
“How secure am I?”“Where should I place my security resources?”
![Page 9: Security Metrics in Practice Development of a Security Metric System to Rate Enterprise Software](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816341550346895dd3d0a3/html5/thumbnails/9.jpg)
What We Can Measure?
Objective, automated, repeatable, and plentiful:Security defects found in source codeQuality defects found in source codeInput opportunities of software
![Page 10: Security Metrics in Practice Development of a Security Metric System to Rate Enterprise Software](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816341550346895dd3d0a3/html5/thumbnails/10.jpg)
Software Security Metrics
Create a descriptive modelPresent the customer with everything we know about the softwareUtilize project artifacts to gauge the security of software
Utilize existing tool results to present useful, high-level information
Any tool that produces objective, repeatable, and easily digested security information.
Metrics used by customer to:Benchmark software counterpartsDevelop clear path towards improving software security in a measurable fashionFeed into existing risk management system
![Page 11: Security Metrics in Practice Development of a Security Metric System to Rate Enterprise Software](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816341550346895dd3d0a3/html5/thumbnails/11.jpg)
Software Security Metrics
Metrics Fortify SCA can facilitate for projects:Common Vulnerability Scoring System (CVSS)
Fortify SCA can automatically provide most of the Base information for a CVSS scoreAudited issue provides information for Temporal category of CVSSReport consumers can provide the Environmental information for a context sensitive CVSS measurement
Useful metrics SCA can deduce about projects:Lines of CodeCylcomatic Complexity per FunctionDefects/KLOCDefects/External data entry-pointsDefects/Internal data consumption points
![Page 12: Security Metrics in Practice Development of a Security Metric System to Rate Enterprise Software](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816341550346895dd3d0a3/html5/thumbnails/12.jpg)
Categorizing Applications
Applications are categorized to aid in “apples to apples” comparisons of projects
“Show me the best mid-sized CRM system”Genres are a combination of:
Application SizeSmall < 150KLOCMedium > 150KLOC; < 500KLOCLarge > 500KLOC
Application TypeWeb ApplicationApplication ServerComponent/APIStand-alone utilityConsole application
Application IndustryFinancialEngineeringConsumereCommerce
![Page 13: Security Metrics in Practice Development of a Security Metric System to Rate Enterprise Software](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816341550346895dd3d0a3/html5/thumbnails/13.jpg)
“Scoring” Software
Customers requested the information in a condensed mannerA quick “thumbs-up/thumbs-down” view on projectsModeled a “star” system similar to mutual fund ratingsMark Curphey’s OWASP Evaluation and Certification Criteria is a similar effort.
![Page 14: Security Metrics in Practice Development of a Security Metric System to Rate Enterprise Software](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816341550346895dd3d0a3/html5/thumbnails/14.jpg)
Rating System
Morningstar for Software SecurityDeveloping a 5-star/tier rating system
Some subtlety of assessment is lost in exchange for ease of useEach tier has criteria before a project may be promoted to the next tierNOT limited to static analysis tools
![Page 15: Security Metrics in Practice Development of a Security Metric System to Rate Enterprise Software](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816341550346895dd3d0a3/html5/thumbnails/15.jpg)
Stars Explained
Absence of Remote and/or Setuid VulnerabilitiesAbsence of Obvious Reliability IssuesFollow Best PracticesDocumented Secure Development ProcessPassed Independent Security Review
![Page 16: Security Metrics in Practice Development of a Security Metric System to Rate Enterprise Software](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816341550346895dd3d0a3/html5/thumbnails/16.jpg)
Critique of Rating System
Ouch!Ratings are very harsh
Impact is high for even one exploitIf automated tools can find issues, other security defects likely to exist
Ordering in unordered setThe tiered nature implies more importance for some criteriaProject information is loss
A project may have a defined security process, yet have vulnerabilities present
Subjectivity of higher tiersThings become more ambiguous as you move up the tiers
What qualifies as an independent review?
![Page 17: Security Metrics in Practice Development of a Security Metric System to Rate Enterprise Software](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816341550346895dd3d0a3/html5/thumbnails/17.jpg)
Using the system
Search for software that meets a particular security requirement
“Show me 2-star, mid-size, shopping cart software”Use stars to filter out componentsDrill down to underlying metrics to make a more informed decision on component usage
“How does this set of 1-star components compare?”Examine detailed defect informationDoes one project have fewer remote exploits?
Feed metrics into existing risk assessment process to make final determination
Model how the selected software will impact existing risk model
![Page 18: Security Metrics in Practice Development of a Security Metric System to Rate Enterprise Software](https://reader035.vdocuments.us/reader035/viewer/2022062323/56816341550346895dd3d0a3/html5/thumbnails/18.jpg)
Next Steps?
Validate against Closed Source softwareTest against auditor “smell test”
How do the hard numbers compare against security auditors reports?
Explore additional software metrics that may relate to security