security metrics [2008]

14/07/2008

Security MetricsPhil Huggins

Security Metrics14/07/2008

Core Text

Security Metrics : Replacing Fear, Uncertainty and Doubt

Andy Jaquith, 2007

0-321-34998-9


Recommended Texts


Growing field

► Areas of interest► Software security ► Modelling► Benchmarking► Return on investment► Breach data

► Standards► ISO / IEC 27004► NIST SP800-55

► Communities► Securitymetrics.org► Metricsexchange.org► Cybersecurity KTN


Securitymetrics.org

► Open mailing list and wiki► Active community► Established by Andy Jaquith► Runs the US based Metricon and MiniMetricon events

each year


Metricsexchange.org

► New open community group► Established by Elizabeth Nichols► Sharing metrics definitions, learning and data ► Early days, big ideas


Cybersecurity KTN – Metrics SIG

► UK Knowledge Trading Networks established by DTI► Promoting collaboration between industry, academia and

government► Metrics Special Interest Group has focused on the

delivery of the Internet Threat Exposure (ITE) Index► Threat and Countermeasure focused metric of exposure► Appears to be aimed at less sophisticated security

practitioners► Risk assessment-lite ?► Currently being developed in an open group


Standards

► NIST SP800 – 55► Exhaustive list of possible security metrics to measure► 99 pages► No real sense of what is a useful metric► Defines useful characteristics to describe a metric

► Performance Goal, Performance Objective, Metric, Purpose, Implementation Evidence, Frequency, Formula, Data Source, Indicator

► ISO/IEC 27004► Currently in draft / closed group► Metrics covering the performance of an ISMS as defined in 27001

and 27002


Types of Security Metrics

► Risk Metrics► Compliance Metrics► Operational Metrics► Quality Metrics► Management Metrics► Business Metrics

► Confusion among practitioners


Focus problems

► Technical Focus► “What do we count?”

► Business Focus► “What do we need to do and why?”

► Counting is the mechanical foundation► The business wants the story the numbers tell

► Metrics are not the answer to funding problems


Other common problems

► Managing to the metric► No longer focused on the result

► Measuring emerging threats► Measuring last years breaches


Questions from the board

► Am I safe?► Can I take responsibility for the actions of my company?► Who handles my data?► Who am I doing business with?► Are they accountable?


Metricon practitioners top 10 metrics

► Data volumes transmitted to competition► Coverage metrics► Availability of business systems► End user perception of security► Legal fees paid out► Total cost of information security► Information asset value► Count of events on systems► Security control success rate► Cost of security monitoring and reporting


Balanced Security Scorecards

► Complete:► People, Process, Technology, Budgeting, Innovation,

Organisational Planning, Operations

► Traditionally include four primary perspectives:► Financial► Customer► Internal Processes► Learning and Growth

► Jaquith has a comprehensive chapter on balanced security scorecards in his book


Geer’s Scorecard

► Finance► Cost of data security per transaction► Downtimes lost to attack by attack class► Data flow per transaction and source► Budget correlation with risk measures

► Process► % of critical systems under a DR plan► % of critical systems obeying the security policy► MTBF & MTTR for security incidents► Frequency of security team internal consultations► Latency to obey security change orders by department


Geer’s Scorecard

► Learning and growth► % of job reviews involving security► % of security workers with training► Ratio of B.U. security staff to central security staff► Timely new system security consultations► % of programs with budgeted security

► Customer► % of SLAs with security standards► % of tested external facing applications► Number of non-employees with access► % of data secure by default► % of customer data outside the data centre


GE Global experience

► Metrics to drive behaviour► Scorecard approach► Business unit drill down and comparison views► Communication plan was key► Built a custom system piecemeal over several years► Started with manual data, automated over time► Now moving to a common platform► Monolithic vs Composite data sources► Centralised vs Business unit data sources


Dept of Veterans Affairs’ experience

► Didn’t have common definitions of:► What IT Security was► What better IT security looked like► The value of security

► Identified the security events that drove perception of security

► Focused on the frequency and impact of those events► Did not ignore uncertainty!► Results-focused


Intel’s experience

► Developed predictive model for future security incidents► Used to provide ROI on ‘reduce the occurrence’ controls

NOT ‘reduce the effect’ controls► Needed to gather current state data first in order to

identify ‘Annual Rate of Occurrence’► 2 years of data from 20+ global locations

► Needed to estimate ‘Single Loss Expectancy’ value for target environment

► Identified limited target groups to pilot controls in first to measure results

► Needed a LOT of data► 87% accurate predictions over a 12 month period


Verizon 2008 Data Breach Investigations Report

► 500 Investigations over 4 Years► 18% of breaches were the result of an unpatched system► 90% of unpatched breaches had had patches publicly

available for 6 months or more► No more would have been prevented by a patch cycle

shorter than a month► There is a lot of useful data in this report


Dan Geer’s counterpoint

► We are losing► The bad guys are in it for the money► Attackers costs are continually falling► Need to start measuring ‘attack metrics’► Focus on increasing their cost of attack► More cost effective to redirect than to resist


Marcus Ranum’s counterpoint

► Statistics only work where:► Population is large► Problems are common and widely shared► Aggressors act consistently

► The only scores that matter are 0% and 100%

► Security is not ‘risk management’ it is ‘complexity management’

14/07/2008

Thank [email protected]