something to think about what kinds of metrics are actually being presented? to boards of directors?...
TRANSCRIPT
Something to Think AboutWhat kinds of metrics are actually being presented?
To Boards of Directors? Internal Audit Committees?
What kinds of questions are being asked?
What kinds of decisions are being made?
What success and failures are people having?
What are their uncertainties and challenges?
-Caroline Wong, CISSPSecurity Initiatives Director
Author: Security Metrics, A Beginner’s Guide
Software Security MetricsReading the Barometers
True Security Countermeasures and Audit’s Virtual VectorWednesday 10/28/2015
Neil BahadurManaging Consultant, Chicago
Board of Directors Example Briefing
What must we address?• Regulatory compliance & contractual requirements• Internal audit & governance requirements• The evolving standard of “Due care”
What should we also consider?• Risk-based controls in each of our IT, InfoSec, Data,
and software security initiatives• Defect prevention, defect discovery, and ongoing
process improvement
What are we doing and how is it working?• Current protections and business metrics• Certifications (SOC2, etc.)• Internal audit (SOX / PCI / HIPAA / FFIEC / etc.)
What gaps must we address now?• Strategic programs / elements (including business
justification)• Compliance / contractual gaps• Internal audit findings
What will we do if there is an incident?• IR Plan / Program / Service • Date and results of last incident response test
• Key Elements
Agenda
• “Must-Do” vs “Want-To-Do” obligations• About metrics• Software security foundation
• Numerical data from a Software Security Initiative
• Practical application• Understand the context• Validate that obligations are met
OBLIGATIONS
Governance & Compliance – Must-Do Examples
• Contractual: PCI - DSS• 6.5: Develop applications based on secure coding guidelines.• 11.3.2: Verify that the penetration test includes application-layer penetration tests.
• Regulatory: SOX• Section 404 IT General Controls
• Internal: SSAE-16 SOC2, ISO/IEC 27034-1:2011 • Process focus• Not as the state of security of an application system but as “a process an organization
can perform for applying controls and measurements to its applications in order the manage the risk of using them”;
• What must we address?
Risks & Threats – Want-to-Do Examples
• Protecting our intellectual property• Protecting non-PCI customer PII and loyalty databases• Whether we are a target for hactivism or criminal
attackers• Similar business profile to those recently attacked• Walk-in threat
• Insider threats from employees and business partners• Detailed review of security initiatives: IT, InfoSec, Data,
Software
• What should we also consider?
Sample Directives
• Regulatory• PCI mandates testing and fixing all critical findings
• Contractual• Customer requires we adhere to their coding standards.
• Business practice• Software security group says they adhere to policy stating that all
supported applications must undergo static and dynamic testing
• Insurance• Policy exceptions must be resolved within 1 year for coverage
• What are the obligations that we are striving to meet with Software Security?
ABOUT METRICS
Fitness – As explained by my wife, the personal trainer
• I ask questions and make decisions about my health every day• What should I eat for breakfast? • How much? How often?
• What kind of exercise should I do? • For what length of time? How often?
• I can change my behavior by setting goals and measuring progress• SMART goals• Specific, measurable, actionable, reasonable, time-based
Vocabulary
• Measurement vs. Metric – what’s the difference?
• It is 67 degrees Fahrenheit in San Francisco• I had 2 cups of coffee this morning
• A measurement is the value of a specific characteristic of a given entity• A metric is the aggregation of one or more measurements to create a
piece of business intelligence.• What is the question the metric answers?• What is the decision the metric supports?• What is the environmental context?
6 Habits of Highly Effective Metrics
1. Simple messages work
2. Context matters
3. Consistently aggregate data for decision support
4. Introduce the concept of risk management
5. Get specific
6. Metrics drive behavior
Unhelpful Measurements in Isolation
• We found 23,435 issues this year (24,123 last year)• Same coverage? Same methods? Fixes?
• 62 applications had static or dynamic testing• Each? Both? Portfolio total apps? PCI apps?
• Others?• How could they be tied together to tell the “real” story?
SOFTWARE SECURITY FOUNDATION
What’s a Software Security Initiative (SSI)?
Technical Validation Activities
• Finding variances in adherence to coding standards• SAST? Fortify? Veracode? AppScan Source? SCR?• Examining application source code or binaries and doing analysis• Customize your rules to find things that matter the most to you
• DAST? Burp? AppScan Standard? WebInspect? PT?• Attacking the running application using automated/manual methods
• Mobile? Client-Server? Embedded? IoT? Cloud?• Phone apps, car apps, pacemakers, ATMs, etc., sometimes tech-
specific, sometimes plain old business logic failures
Numerical SSI DataTop Concerns
• Assigning owner for software security
• Educating developers
• Finding design flaws
• Expanding test coverage
• Blocking popular attack vectors
Apps by Language
Java.NETC/C++PHP
July Aug Sept Oct0
5
10
15
20
25
30
35
40
45
Outstanding Vulnerabilities
LowMedHighCritical
Java .NET C/C++ PHP0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
Application TestingPercent Coverage
SASTDASTSCRExternal Pentest
Critical High Med Low0
2
4
6
8
10
12
14
16
18
Application Pen Test FindingsAll PCI Apps – Oct 2015
PRACTICAL APPLICATION
Understand the Context to Validate
• Which reports do you receive from software security?• How frequently do you receive these?
• Do you understand the data?• Do you care about the data?• What do you do with this data?• Do you understand how the data ties into obligations?• How can you get your own data?
Validation Example – Regulatory
• Directive• PCI mandates testing and fixing all critical findings
• What kinds of information would you ask for?• What metrics would be helpful?• What isolated measurements wouldn’t?• How could you get independent data points?• What kinds of tooling/training would you need?
Validation Example – Contractual
• Directive• Customer requires we adhere to their coding standards.
• What kinds of information would you ask for?• What metrics would be helpful?• What isolated measurements wouldn’t?• How could you get independent data points?• What kinds of tooling/training would you need?
Validation Example – Business Practice
• Directive• Software security group created internal policy committing that all
supported applications must undergo static and dynamic testing
• What kinds of information would you ask for?• What metrics would be helpful?• What isolated measurements wouldn’t?• How could you get independent data points?• What kinds of tooling/training would you need?
Validation Example – Insurance
• Directive• Policy exceptions must be resolved within 1 year for coverage
• What kinds of information would you ask for?• What metrics would be helpful?• What isolated measurements wouldn’t?• How could you get independent data points?• What kinds of tooling/training would you need?
Next Step: Is Software Security Working?BUSINESS QUESTION METRIC – BUSINESS INTEL COMMENTS
What’s the impact on production defects of a 10% increase in software security spending for static analysis?
Trend in SSI cost vs.Trend in production software security defects
Comparing the two trendlines is more useful than looking at either in isolation
Which security technology stacks and components harbor the greatest amount of defects?
Discoveries of vulnerability x / App component type
Understanding the prevalence of a vuln for a specific app component type is more useful than counting discovery instances without the environmental context
How long does it take the organization to successfully respond to process variances?
Average days to remediate variance /Variance type
Understanding the time it takes to address variances by variance type is more useful than without the environmental context
How much extra work is caused by the need to triage results (remove false positives, etc) from testing tools?
Static analysis false positives /Tool / Defect type / Tech stack / Analysis rule
Any of the counts alone are less useful than when viewed together for more complete business context
What is the impact of training on software security defects found in various types of testing?
Web app Java code from developers with 8 hours of instruction has 20% fewer defects found by static analysis than code from untrained developers
Look at desired outcomes of training rather than the individual course attendance.
Key Takeaways
• Understand the obligation• Consider the potential data sources• Know measurements from metrics and which you need
to answer the questions that are being asked• Understand the process• Know how to validate
Resources
• Security Metrics: A Beginner’s Guide• CIS Consensus Security Metrics Definitions• Cigital Silver Bullet Podcasts
• Show 91 on Security Metrics
