using information security metrics to demonstrate value and drive
TRANSCRIPT
![Page 1: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/1.jpg)
E D U C AU S E S E C U R I T Y P R O F E S S I O N A L S C O N F E R E N C E
M AY 6 - 8 , 2 0 1 4
Using Information Security
Metrics To Demonstrate Value
and Drive Improvements
SHIRLEY C. PAYNE
AVP FOR INFORMATION SECURITY, POLICY, & RECORDS
UNIVERSITY OF VIRGINIA
![Page 2: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/2.jpg)
Copyright Shirley C. Payne 2014.
This presentation leaves copyright of the content to the presenter.
Unless otherwise noted in the materials, uploaded content carries the
Creative Commons Attribution-NonCommercial-ShareAlike license,
which grants usage to the general public with the stipulated criteria.
![Page 3: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/3.jpg)
Seminar Will Answer…
Demonstrate Value & Drive Improvements
What makes a metric effective?
What are the challenges?
Where do I start?
How should I communicate
metrics?
Where can I learn more?
and provide lots of examples…
![Page 4: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/4.jpg)
Demonstrate Value & Drive Improvements
What makes a metric effective?
![Page 5: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/5.jpg)
Provide single-point-in-
time views of specific,
discrete factors
Generated by counting
Objective raw data
Derived by comparing
two+ measurements
taken over time to a
predetermined baseline
Generated by analysis
Objective or subjective
human interpretations of
those data
Measurements and Metrics – same thing?
![Page 6: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/6.jpg)
The Mark of Good Metrics
Metrics should be SMART
Specific Well-defined, using unambiguous wording
Measurable Quantitative when feasible
Attainable Within budgetary and technical limitations
Repeatable Measurements from which metric is derived do
not vary depending on the person taking them.
Time-dependent Takes into consideration measurements from
multiple time slices
George Jelen, “SSE-CMM Security Metrics”
![Page 7: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/7.jpg)
Albert Einstein
Everything that
can be counted
does not
necessarily count;
everything that
counts cannot
necessarily be
counted.
![Page 8: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/8.jpg)
Truly Effective Metrics…
Indicate the degree to which security goals are being met
Show linkage between security and institutional goals
Drive actions taken to improve the overall security program
![Page 9: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/9.jpg)
Rate This Metric
% of servers that are secure has increased fourfold since 2010
S
M
A
R
T
![Page 10: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/10.jpg)
Rate This Metric
% of servers that are secure has increased fourfold since 2010
% of servers with patched operating systems
increased fourfold since 2010
![Page 11: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/11.jpg)
Rate This Metric
% of employees who are aware of security threats doubled last
year
S
M
A
R
T
![Page 12: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/12.jpg)
Rate This Metric
% of employees who are aware of security threats doubled last
year
% of employees completing annual security
awareness training doubled last year
![Page 13: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/13.jpg)
Rate This Metric
Level of faculty frustration w/2-factor authentication compared
to reduced risk of unauthorized data access
![Page 14: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/14.jpg)
Rate This Metric
Level of faculty frustration w/2-factor authentication compared
to reduced risk of unauthorized data access
% of faculty issued UVa’s hardware identity tokens
compared to faculty use of tokens for email login
![Page 15: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/15.jpg)
Rate This Metric
Web application vulnerabilities found during January 2014
penetration test
S
M
A
R
T
![Page 16: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/16.jpg)
Rate This Metric
Web application vulnerabilities found during January 2014
penetration test
Web application vulnerabilities found during
January 2014 penetration test compared to January
2013 and 2012 results
![Page 17: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/17.jpg)
Rate This Metric
% of total IT budget spent on security increased by 2% each of
the past two years
S
M
A
R
T
![Page 18: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/18.jpg)
Rate This Metric
% of total IT budget spent on security increased by 2% each of
the past two years
Since the implementation of xxx product costing
$50K, the occurrence of records with highly
sensitive content stored in poorly secured data
stores has been reduced by 8 million
![Page 19: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/19.jpg)
Rate This Metric
In 2013 there were 98 reported Higher Education breaches
nationwide compared to 5 at this institution.
S
M
A
R
T
![Page 20: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/20.jpg)
Rate This Metric
In 2013 there were 98 reported Higher Education breaches
nationwide compared to 5 at this institution.
In 2013 there were 5 reported breaches, 4 of which
were discovered by internal controls (versus
reported by outsiders)
![Page 21: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/21.jpg)
The Value of “Truly Effective” Security Metrics(internal focus)
Discern effectiveness of particular security program
component
Indicate security of specific system, product, or process
Identify risk in not taking a given action and, thereby, help
prioritize corrective actions
Provide evidence of regulatory compliance
Demonstrate ability of security staff and departments to
address security issues for which they are responsible
![Page 22: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/22.jpg)
Value of “Truly Effective” Security Metrics(external focus)
Provide basis for answering tough questions, such as
Are we more secure today than we were before?
How do we compare to others in this regard?
Are we secure enough?
Raise security awareness among executives and other
stakeholders
Clearly convey value of overall security program relative to
business objectives
![Page 23: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/23.jpg)
What makes a metric effective?
Characteristics of
effective metrics:
SMART
Indicate % to which
security goals are met
Link security to
institutional goals
Drive improvements
Effective Metric
Specific
Measurable
AttainableRepeatable
Time-dependent
![Page 24: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/24.jpg)
What makes a metric effective?
A metric is effective if it can:
Provide insight into IS program effectiveness,
regulatory compliance, and ability to address security
concerns
Help identify risks of not taking certain actions,
providing guidance for future investments.
Provide concrete facts for raising security awareness
Provide credible answers to hard questions about
status and value of IS program
![Page 25: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/25.jpg)
Demonstrate Value & Drive Improvements
What are the challenges?
and provide lots of examples…
![Page 26: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/26.jpg)
The State of Security Metrics
Other disciplines, such as the field of finance, have proven
quantitative methods for determining risk, along with
decision-making frameworks based on established
measures and metrics.
[These] are just emerging for information security, however,
and as in any discipline, require realistic assumptions and
inputs to attain reliable results.
Wayne Jansen, “Directions in Security Metrics Research,” NISTIR 7564; April 2009
![Page 27: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/27.jpg)
75%CISOs say…
![Page 28: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/28.jpg)
53%CISOs say their…
![Page 29: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/29.jpg)
51%CISOs say their…
![Page 30: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/30.jpg)
Why
Not?
![Page 31: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/31.jpg)
18
35
40
48
59
0 10 20 30 40 50 60 70
EXECS NOT INTERESTED
TIME/RESOURCES TO PREP REPORTS FOR EXECS
ONLY COMMUNICATE W/ EXECS ON INCIDENTS
HIGHER PRIORITIES
INFO TOO TECHNICAL FOR EXECS
%
%
![Page 32: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/32.jpg)
Conclusion
“CISO’s talk about the importance of leveraging
metrics to influence business leadership…
Unfortunately, they struggle with the bigger
challenge of producing meaningful metrics
while those they use are rarely aligned with
business goals.”
Rekha Shenoy, Tripwire VP for Marketing & Corporate Development
![Page 33: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/33.jpg)
Let’s Look At The Challenges
Measuring Risk
Determining ROSI
Limited Guidance and Practical Examples
![Page 34: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/34.jpg)
How To Measure Risk?
Risk = Asset Value x Threat x Vulnerability
Asset Value – easiest to measure in some cases, but
how to quantify assets like institutional reputation?
Threat – very hard to measure the potential for harm,
although information from external sources may be
useful.
Vulnerability – sources of good information available,
but not all vulnerabilities can be quantified.
![Page 35: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/35.jpg)
Determining ROSI?
“It’s a good idea in theory, but it’s
mostly bunk in practice… Security
is not an investment that provides
a return… It is an expense that,
hopefully, pays for itself in cost
savings…Security is about loss
prevention, not about earnings. “
Bruce Schneier – September 2, 2008
https://www.schneier.com/blog/srchives/2008/09/
security_roi_1.html
![Page 36: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/36.jpg)
Guatemala Sinkholehttp://news.nationalgeographic.com
![Page 37: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/37.jpg)
“We’re here to suggest not only
that you can use ROSI to sell
security internally, but you must.”
Scott Berinato, “Calculated Risk: Return on Security Investment,
www.csoonline.com
An Alternate View!
![Page 38: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/38.jpg)
Challenge industry assumptions and cultural biases
Rethink Your Assumptions
Find and use data that’s out there
Do the Legwork
Subtract cost from benefits
Do the Math
Scott Berinato, “Calculated Risk: Return on Security Investment, www.csoonline.com
![Page 39: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/39.jpg)
Challenge industry assumptions and cultural biases
Rethink Your Assumptions
• Precision is not the goal
• Think in stochastic, not binary, terms
Fire extinguisher ROI: $3 return for every $1 invested
NOT
Fire extinguisher ROI: $3.14 return for every $2.97 invested
![Page 40: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/40.jpg)
Find and use data that’s out there
Do the Legwork
• Actuarial information, e.g., CERT, Poneman
• Annual data breach reports, e.g., Verizon, privacyrights.org
• Threat trends, e.g., IBM X-Force, Mandiant
• Talk to business managers, e.g., Risk Management Officers, Financial Managers
![Page 41: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/41.jpg)
Subtract cost from benefits
Do the Math
• Annual Loss Expectancy
• Modified ALE
• Other methods
![Page 42: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/42.jpg)
Cost Examples
Lost staff productivity
Loss/compromise of data
Recovery costs
Reputational loss
Fines and lawsuits
Loss of future research grants/contracts
Etc.
Informed by Julia Allen – March 10, 2003 “Making the Business
Case for Information Security: Selling to Senior Managements”
![Page 43: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/43.jpg)
Scenario: Need to determine ROSI on acquisition of
web app vulnerability scanning service
![Page 44: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/44.jpg)
Subtract cost from benefits
Do the Math
• Annual Loss Expectancy: how much $$ lost per year due to security incident
ALE =
average cost of data breach X probability of web app breach next year
ALE = $3.2M X .22 = $704,000
Poneman Study Verizon Study
![Page 45: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/45.jpg)
Subtract cost from benefits
Do the Math
• Modified ALE: ALE w/effect of mitigation measure incorporated
Assumption: Effect of scanning all web apps for vulnerabilities is
that probability of web app breach reduced by half
mALE = $3.2M X .11 = $352,000
![Page 46: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/46.jpg)
Subtract cost from benefits
Do the Math
• Annual Loss Expectancy: how much $$ lost per year due to security incident
• Modified ALE: ALE w/effect of mitigation measure incorporated
COST SAVINGS = ALE – mALE
COST SAVINGS = $704,000 - $352,000 = $352,000
![Page 47: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/47.jpg)
Subtract cost from benefits
Do the Math
• Annual Loss Expectancy: how much $$ lost per year due to security incident
• Modified ALE: ALE w/effect of mitigation measure incorporated
ROSI = BENEFITS - COST
Cost per year of xyz web app service = $80,000
ROSI = $352,000 - $80,000 = $281,600
![Page 48: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/48.jpg)
Let’s Look At The Challenges
Limited Guidance and Practical Examples
![Page 49: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/49.jpg)
Limited Guidance and Practical Examples?
Good News!
ISO 27004
NIST SP 800-55 Rev. 1
CIS Consensus Information Security Metrics
Top 20 Critical Security Controls
![Page 50: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/50.jpg)
ISO/IEC 27004
Published December 2009 (new version planned)
Guidance for developing metrics for evaluating
information security programs
Key sections: Information security measurement overview;
Management responsibilities;
Measures and measurement development;
Measurement operation;
Data analysis and measurement results reporting;
Program evaluation and improvement.
http://www.iso.org/iso/catalogue_detail?csnumber=42106
![Page 51: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/51.jpg)
NIST SP 800-55 Rev. 1
Published July 2008
Specific advice for developing, selecting, and implementing
performance measures
Security controls tied to overall mission
Practical examples
http://csrc.nist.gov/publications/PubsSPs.html
![Page 52: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/52.jpg)
CIS Consensus Information Security Metrics
V1.1.0 published November 2010
Metrics on security outcomes and process performance.
Common definitions for data collection and analysis
Metrics grouped by purpose and audience: management,
operational, technical
Twenty metrics defined in six functions: incident management,
vulnerability management, patch management, application security, configuration
management, financial metrics
https://benchmarks.cisecurity.org/downloads/browse/?category=metrics
![Page 53: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/53.jpg)
Top 20 Critical Security Controls
V5.0 published February 2014
Identifies controls having greatest positive impact on risk
posture
Includes suggested metrics for most controls
http://www.counciloncybersecurity.org
![Page 54: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/54.jpg)
Great News: There’s Now Helpful Guidance
…for/by HIGHER EDUCATION w/EXAMPLES!
EDUCAUSE: 7 Things You Should Know About Security
Metrics article
EDUCAUSE: Guide To Effective Security Metrics
EDUCAUSE: Security Metrics Resource Library
EDUCAUSE: Core Data Services
![Page 55: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/55.jpg)
Also, check out these conference sessions…
![Page 56: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/56.jpg)
What are the challenges?
Lack of common vocabulary and definitions
We don’t speak the language of executives:
Institutional goals
Risks
ROI
Finally, practical guidance and examples!
![Page 57: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/57.jpg)
Demonstrate Value & Drive Improvements
Where do I start?
and provide lots of examples…
![Page 58: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/58.jpg)
Seven-Step Methodology
Review and refine
Create action plan
Determine how to report
Establish benchmarks and targets
Develop strategies for generation
Decide what metrics to generate
Define goal(s) and objectives
![Page 59: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/59.jpg)
Step 1
Define the metrics
program goal(s) and
objectives
Clearly state the end toward which
all metrics and measurements should
be directed
Indicate high level actions that must
be collectively accomplished to meet
the goal(s)
![Page 60: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/60.jpg)
Step 2
Decide what metrics to
generate
Use existing process improvement
framework to determine metrics
![Page 61: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/61.jpg)
Framework Examples
Six Sigma Breakthrough Strategy
Balanced Scorecard
Enterprise Risk Management
Enterprise-level Compliance Tracking
Strong Focus Within Institution On: ROI
On time/on schedule project completion
National rankings
Bond ratings
Etc.
![Page 62: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/62.jpg)
Step 2
Decide what metrics
to generate
Use existing process improvement
framework to determine metrics
In the absence of pre-existing
framework, use top-down or
bottom-up approach for determining
what metrics might be desirable
![Page 63: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/63.jpg)
Top-down Approach
STEPS EXAMPLES
a. Define/list objectives of the overall
security program
To reduce the number of virus infections
within the institution by 30% by 2015
b. Identify metrics that would indicate
progress toward each objective
Current ratio of virus alerts to actual
infections as compared to the baseline
2012 figure
c. Determine measurements needed for
each metric
Number of virus alerts issued to the
organization by month
Number of virus infections reported
![Page 64: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/64.jpg)
Bottom-up Approach
STEPS EXAMPLES
a. Identify measurements that are/could
be collected for this process
Average number of critical
vulnerabilities detected monthly in
servers using xyz scanning tool
b. Determine metrics that could be
generated from the measurements
Change in number of critical
vulnerabilities detected in servers since
xyz scanning tool implemented
c. Determine the association between the
derived metrics and established
objectives of the overall security program
To reduce the number of detectable
vulnerabilities on servers by 95% by
2015.
![Page 65: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/65.jpg)
Step 3
Develop strategies for
generating the metrics
Identify trustworthy sources of data Internal, e.g., IT operations, Audit, Risk Management,
Finance, Compliance, etc.
External, e.g., actuarial data, annual breach stats, etc.
Decide on frequency of data collection
Assign responsibility for assuring
accuracy of raw data
Develop methods for compiling data into
measurements and generating metrics
![Page 66: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/66.jpg)
Step 4
Establish benchmarks,
baselines, and targets
Research observed trends and
recommendations from professional
associations, published research, etc.
Set reachable targets
![Page 67: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/67.jpg)
Step 5
Determine how the
metrics will be
reported
Effective communication of metrics is
obviously key. Don’t over-simplify, but
present clearly.
Vary what is reported and how
depending upon audience
Determine context, format,
frequency, distribution method, and
reporting responsibility
![Page 68: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/68.jpg)
Step 6
Create an action plan
and act on it
Plan and conduct actions needed to
generate metrics; test, verify,
investigate anomalies; implement
Document!
![Page 69: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/69.jpg)
Field Data
Measure ID ISPA-1
Goal Identify levels of serious network-based threats of the kind monitored by the FireEye scanner by network
Measure Since implementation of FireEye monitoring, number of high severity level infections detected on each network
Type Implementation
Formula Number of critical issues identified by FireEye on each network
Target Baseline; comparison
Definition of Measures Contributing to Metrics
•“Critical” is defined by FireEye based on its risk analysis; it represents items receiving a score of 5-7 on a 7 point scale; UVa does not have input on this definition•The networks are defined by the UVa network architecture; the number of devices on each network will vary over time as devices are added and migrated between networks; additional networks may be added to the list of those scanned over time
Frequency Data are collected daily; they will be reported, as appropriate, on a daily, weekly, monthly, quarterly and/or fiscal year basis
Responsible Parties •Information Owner: AVP ISPRO•Information Collector: ISPA team•Information Customer: VP/CIO, AVP ISPRO, Director ISPA
Data Source FireEye console
Reporting Format Bar graph; spreadsheet
![Page 70: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/70.jpg)
Step 7
Establish a formal
program review and
refinement cycle
Doubt about metric accuracy?
Value worth effort to generate?
New metric best practices/guidance
to consider?
Most important: did metrics guide
improvements to overall security
program?
![Page 71: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/71.jpg)
Adjust for Maturity of Security Program
![Page 72: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/72.jpg)
Usefulness of a Given Metric Varies Depending Upon
Maturity of the Security Program
Policies Developed
Procedures Developed
Procedures & Controls
Implemented
Procedures & Controls Tested
Procedures & Controls Integrated
“Performance Measurement Guide for Information Security,” NIST SP 800-55 Revision 1
http://csrc.nist.gov/publications/PubsSPs.html
![Page 73: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/73.jpg)
Usefulness of a Given Metric Varies Depending Upon
Maturity of the Security Program
Policies Developed
Procedures Developed
Procedures & Controls
Implemented
Procedures & Controls Tested
Procedures & Controls Integrated
Useful metrics difficult to produce at this early stage; limited availability of data and collection may be difficult
![Page 74: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/74.jpg)
Usefulness of a Given Metric Varies Depending Upon
Maturity of the Security Program
Policies Developed
Procedures Developed
Procedures & Controls
Implemented
Procedures & Controls Tested
Procedures & Controls Integrated
Primary focus on implementation metrics
• Ex: Increase in # of departments that have mission continuity plans
![Page 75: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/75.jpg)
Usefulness of a Given Metric Varies Depending Upon
Maturity of the Security Program
Policies Developed
Procedures Developed
Procedures & Controls
Implemented
Procedures & Controls Tested
Procedures & Controls Integrated
Primary focus on efficiency and effectiveness metrics
• Ex: % of total departments with updated, tested mission continuity plans
![Page 76: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/76.jpg)
Usefulness of a Given Metric Varies Depending Upon
Maturity of the Security Program
Policies Developed
Procedures Developed
Procedures & Controls
Implemented
Procedures & Controls Tested
Procedures & Controls Integrated
Primary focus on impact metrics
• Ex: Outcome of 48-hour power outage in administration bldg.
![Page 77: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/77.jpg)
Usefulness of a Given Metric Varies Depending Upon
Maturity of the Security Program
Policies Developed
Procedures Developed
Procedures & Controls
Implemented
Procedures & Controls Tested
Procedures & Controls Integrated
Primary focus on implementation metrics
• Ex: Sensitive data scanning tool deployed on all individual desktops/laptops
![Page 78: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/78.jpg)
Usefulness of a Given Metric Varies Depending Upon
Maturity of the Security Program
Policies Developed
Procedures Developed
Procedures & Controls
Implemented
Procedures & Controls Tested
Procedures & Controls Integrated
Primary focus on efficiency and effectiveness metrics
• Ex: # of unapproved storage of sensitive data found on desktops/laptops
![Page 79: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/79.jpg)
Usefulness of a Given Metric Varies Depending Upon
Maturity of the Security Program
Policies Developed
Procedures Developed
Procedures & Controls
Implemented
Procedures & Controls Tested
Procedures & Controls Integrated
Primary focus on impact metrics
• Ex: Reduction in sensitive data exposures due to stolen or vulnerable desktops/laptops
![Page 80: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/80.jpg)
Where do I start?
Leverage existing frameworks for expressing: progress
toward goals, value propositions, process improvements,
etc.
Use systematic approach for defining effective metrics
Adjust metric types as security program matures
Implementation Metrics
Efficiency and Effectiveness Metrics
Impact Metrics
![Page 81: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/81.jpg)
Demonstrate Value & Drive Improvements
How should I communicate
metrics?
and provide lots of examples…
![Page 82: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/82.jpg)
Good News: It’s Now A Hot Topic
![Page 83: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/83.jpg)
“Security is THE issue of
the day and security is
everyone’s responsibility ...”
CIO, Commonwealth of Virginia Information Technology Agency
at 2014 Commonwealth of Virginia Information Security Conference
![Page 84: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/84.jpg)
84
But, how to make your message heard?
![Page 85: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/85.jpg)
Albert Einstein
Things should
be made as
simple as
possible, but not
any simpler
![Page 86: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/86.jpg)
Tip…
Customize your
metrics-based
information for
the audience
![Page 87: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/87.jpg)
Customize for Security Engineers
• Change in #
malware
infections over
time
• # web app
vulnerabilities
detected since
scan tool
implemented
• Mean time
between phish
report and
blocked malicious
sites
![Page 88: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/88.jpg)
Customize for CISO/CIO
• Change in # of
proactive security
consultations
compared to FY13
baseline.
• Since
implementing web
application
security scanning
service, # high
severity level
vulnerabilities
detected declined
90%.
![Page 89: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/89.jpg)
Customize for Executives
• % of IT budget
spent on security
compared to peer
institutions
• Since institution-
wide SSN
remediation
project initiated,
change in ratio of
data security
breaches to total
security incidents
investigated
![Page 90: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/90.jpg)
Tip…
Use effective
visuals
![Page 91: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/91.jpg)
![Page 92: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/92.jpg)
![Page 93: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/93.jpg)
![Page 94: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/94.jpg)
Additional Tips for Communication
Provide right metrics for issue at hand
Provide brief interpretation and analysis
Use specific audience’s language
Link to business goals and objectives
![Page 95: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/95.jpg)
How should I communicate metrics?
Take heart. You now have a receptive audience.
Tailor for the audience
Delivery method is as important as what you have to say
Right metric clearly conveyed
=
Right conclusion & decision
![Page 96: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/96.jpg)
Demonstrate Value & Drive Improvements
Where can I learn more?
and provide lots of examples…
![Page 97: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/97.jpg)
References
Allen, Julia. “Making the Business Case for Information Security: Selling to Senior Managements.” Carnegie Mellon University at InfoSec World, 2003, March 10, 2003
Allen, Julia and Stephani Losi. “The ROI of Security.” Software Engineering Institute, Carnegie Mellon University, October 1, 2006. http://resources.sei.cmu.edu/asset_files/podcast/2006_016_100_47182.pdf
Berinato, Scott, “Calculated Risk: Return on Security Investment,” CSOonline.com, December 9, 2002, http://www.csoonline.com/article/2113094/metrics-budgets/calculated-risk--return-on-security-investment.html
Center for Internet Security. “CIS Consensus Information Security Metrics,” November 2010. https://benchmarks.cisecurity.org/downloads/browse/?category=metrics
Council on Cybersecurity. “Top 20 Information Security Controls,” http://www.counciloncybersecurity.org
Cullinane, Dave. “Security Awareness and Communication in the C-Suite,” EDUCAUSE e-Live Webinar, October 4, 2012. http://www.educause.edu/library/resources/security-awareness-and-communication-c-suite
EDUCAUSE: 7 Things You Should Know About Security Metrics, http://www.educause.edu/library/resources/7-things-you-should-know-about-information-security-metrics
![Page 98: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/98.jpg)
References
EDUCAUSE: Guide To Effective Security Metrics, https://wiki.internet2.edu/confluence/display/2014infosecurityguide/Effective+Security+Metrics
EDUCAUSE: Security Metrics Resource Library, http://www.educause.edu/library/security-metrics
EDUCAUSE Core Data Service, http://www.educause.edu/research-and-publications/research/core-data-service
Hinson, Dr. Gary, “Seven Myths About Security Metrics,” ISSA Journal, July 2006. http://www.noticebored.com/html/metrics.html
ISO/IEC 27004 http://www.iso.org/iso/catalogue_detail?csnumber=42106
Jansen, Wayne. “Directions in Security Metrics Research,” NISTIR 7564; April 2009. http://csrc.nist.gov/publications/nistir/ir7564/nistir-7564_metrics-research.pdf
Jelen, George. “SSE-CMM Security Metrics.” NIST and CSSPAB Workshop, Washington, D.C., 13-14 June 2000. URL: http://csrc.nist.gov/csspab/june13-15/jelen.pdf (10 July 2001).
Payne, Shirley C., “A Guide To Security Metrics,” SANS Reading Room, July 11, 2001, updated June 19, 2006. http://www.sans.org/reading-room/whitepapers/auditing/guide-security-metrics-55
![Page 99: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/99.jpg)
References
“Performance Measurement Guide for Information Security,” NIST SP 800-55 Revision 1 –http://csrc.nist.gov/publications/PubsSPs.html
Poneman Institute. “The 2013 Cost of a Data Breach: Global Analysis,” May 28, 2013. http://www.ponemon.org/library/2013-cost-of-data-breach-global-analysis
Ponemon Institute. “The State of Risk-based Security Management 2013,” http://www.tripwire.com/ponemon/2013/
Schneier, Bruce. “Security ROI,” September 2, 2008 https://www.schneier.com/blog/archives/2008/09/security_roi_1.html
Slater, Derek, “Security Metrics: Critical Issues,” CSOonline.com, Nov 12, 2012, http://www.csoonline.com/article/2123361/metrics-budgets/security-metrics--critical-issues.html
Stafford, Eugene and Christina Torode. “A bleak picture of IT security metrics and fighting malicious attacks,” ISSA conference, Nashville, Tenn., December 11, 2013 http://searchcompliance.techtarget.com/video/A-bleak-picture-of-IT-security-metrics-and-fighting-malicious-attacks
![Page 100: Using Information Security Metrics To Demonstrate Value and Drive](https://reader031.vdocuments.us/reader031/viewer/2022030320/586cbbb31a28abdc3a8bf0e9/html5/thumbnails/100.jpg)
Most Of All…
Keep your eyes on the forest, not the trees!