security measures and metrics
DESCRIPTION
Security Measures and Metrics. Pete Lindstrom Research Director Spire Security. Agenda. Elements of metrics Interlude: Four disciplines Back to metrics ROI/ROSI. Status of security. Difficult to define “good security” Minimal difference between security and “lucky” - PowerPoint PPT PresentationTRANSCRIPT
Security Measures and Metrics
Pete Lindstrom
Research Director
Spire Security
Agenda
Elements of metrics
Interlude: Four disciplines
Back to metrics
ROI/ROSI
Status of security
Difficult to define “good security”
Minimal difference between security and “lucky”
We don’t know how to measure success.
One incident doesn’t necessarily mean “failure”
Key elements of security metrics
Key elements of security metrics
Building BlocksLet’s put them
together…
People:Admins by Department
Costs:Salaries, ConsultingHW, SW, Maint.
Activities:Four Disciplines
Time:Hr/Day
Month/Yr
Resources:User accts,
systems, apps
Agenda
Elements of metrics
Interlude: Four disciplines
Back to metrics
ROI/ROSI
Process Effectiveness MetricsProcess
Effectivenessa.k.a. “doing things
right”
Elements:• Activities• errors
For example:• Accts per person• Vulns per person• Patches per person
error rates
ThreatMANAGEMENT
TrustMANAGEMENT
IdentityMANAGEMENT
VulnerabilityMANAGEMENT
Security reference model
1. Harden the Infrastructure
2. Control sources (users/others)
3. Harden the Process/data
4. Monitor/detectinappropriate and/or
malicious activity
ThreatMANAGEMENT
TrustMANAGEMENT
VulnerabilityMANAGEMENT
IdentityMANAGEMENT
Identity ValidationAccount ManagementPassword Management
Policy ManagementSecurity Arch. DesignTicket Management
Vulnerability AssessmentsPatch ManagementSoftware Security
Four disciplines of security management
Threat IdentificationSecurity MonitoringIncident Management
INLINEINLINE
Intrusion Prevention
AuthenticationUser Access
Control
EncryptionIntegrity
System Access Control
Identity management
Functions
Identify users
Assign accounts/rights
Maintain identity (passwords)
Validate sessions
Authorize access
Vulnerability management
Functions
Scan for exposures
Eliminate vulnerabilities
Remediate vulnerabilities
Mitigate vulnerabilities
Manage compliance
Trust management
Functions
Write policies
Design security
Ensure confidentiality
Ensure integrity
Threat management
Functions
Analyze traffic
Analyze logs
Manage incidents
Conduct forensics
Agenda
Elements of metrics
Interlude: Four disciplines
Back to metrics
ROI/ROSI
Process Effectiveness MetricsProcess
Effectivenessa.k.a. “doing things
right”
Elements:• Activities• errors
For example:• Accts per person• Vulns per person• Patches per person
error rates
Process effectiveness
Error rates
Identity management• Request errors
Vulnerability management• Vulnerabilities remaining
Threat management• Incident response
Trust management• Policy violations
Staff Productivity MetricsStaff
productivitya.k.a. “people doing
things” better
Elements:• People• Activities
For example:• Accts per person• Vulns per person• Patches per person
Staff productivityProductivity and workload for all manual activities
(activities/people) Identity management• Requests per administrator• Account disablements per admin• Password resets per admin
Vulnerability management• Vulnerabilities resolved per administrator
Threat management• Incidents per person
Trust management• Policy changes per person
Cycle Time MetricsCycle Time
a.k.a. avg “time to perform activity x”
Elements:• Time• Activities
For example:• Accts per month• Vulns fixed per month• Patches per month
Process efficiency (cycle time)
Time/activities
Identity management• Request time
Vulnerability management• Remediation time
Threat management• Incident response time
Trust management• Policy creation time
Efficiency MetricsAdmins by
Department
2000 Hours per FTE
Efficiencya.k.a. “people doing
things” quicker
Elements:• People• Activities• Time
For example:• Accts/person/hr• Vulns/person/hr• Patches/person/hr
Cost Effectiveness MetricsAdmins by Department
Salaries, Consulting Fees
Costeffectiveness
a.k.a. “people doing things” cheaper
Elements:• People• Activities• Costs
For example:• Cost per acct• Cost per vuln fixed• Cost per patch
Cost effectivenessDollars/activities; dollars/resources; dollars/demographics Identity management• Cost per request• Cost per password reset
Vulnerability management• Cost per vulnerability• Cost per system setting
Threat management• Cost per incident
Trust management• Cost per policy• Cost per project
When to use metrics
Process effectiveness• Six Sigma
Staff productivity• ROI / promotions
Cycle time• Balanced scorecard
Efficiency• ROI
Cost effectiveness• Activity-based costing• ROI/TCO
Business uses of security
Benchmarking (Balanced scorecard)
Baselining (Six Sigma)
Activity-based costing/Mgt
ROI
Risk management (ROSI)
Missing Element: RISK! Risk Managementa.k.a. “people doing
things” more securely!
Elements:• Activities• Resources
Four Disciplines:• Identity Mgt• Vuln Mgt• Trust Mgt• Threat Mgt
Risk metricsResources/resources; resources/demographics Identity management• User accounts per application
Vulnerability management• Vulnerabilities per resource
Threat management• Incidents per resource
Trust management• Policies per resource
Risk effectiveness
Activities/activities (automated)
Identity management
• Failed logins/total logins
Vulnerability management
• Access denied/total access
Threat management
• Incidents/events
Trust management
Agenda
Elements of metrics
Interlude: Four disciplines
Back to metrics
ROI/ROSI
Examples:Return on Investment (ROI) & Return on Security Investment (ROSI)
The elements of value (Loss)
ROI
• IT productivity (time)
• User productivity (time)
…these also have ROSI value
ROSI
• Legal/regulatory costs (fees/fines)
• Direct revenue
• Stored asset value (intellectual property, financial
assets)
Let’s talk ROI
Keyword is efficiency
Reduced Capital Expenditures (CapEx)• Lower h/w, s/w costs
Scalability, manageability, performance
Reduced Operating Expenditures (OpEx)• Lower IT, end-user costs
(higher productivity)
Productivity
Where users and IT spend their time.
Time-is-money philosophy.
Often the only aspect of loss we quantify.
Basic source of ROI.
Hourly rate x hours of effort.
In order to determine the value of activities, you first have to determine what activities are performed.
Identity management ROI
Provisioning• New employee productivity• Automated account management
Password management• Reduced help desk time• Employee productivity
Web access control• Developer efficiency (build vs. buy)
Trust management ROI
Public Key Infrastructure
• Managing certificates
Virtual Private Networks
• Leased lines
SSL Acceleration
• Hardware efficiency
Vulnerability management ROI
Firewalls
• Reduce ACL management
Vulnerability assess/remediate
• Reduce manual efforts
Patch management
• Automate patching
Software quality
• Reduce bug fixes
Threat management ROI
Antivirus• Recovery of systems
Network IDS• Reduce manual detection/forensics
Host IDS• Manual log efforts
Security Event Management• Aggregation/prioritization of work
Getting to ROI
Identify amount of labor allocated to
individual security activities.
Identify solution and its corresponding
activities.
Identify labor difference with and without
solution.
The roots of ROSI
Our overall objective is to reduce risk.We are relatively “new” to spending on
solutions.We often didn’t really do anything that was
considered a recurring expense (I am guessing a bit here).
But, the Internet has changed all that (or at least made it apparent).
Return on Security Investment
Keyword: Effectiveness
Effectiveness = Reduced risk
Protecting Value and Loss
• Legal/regulatory costs (fees/fines)
• Direct revenue
• Stored asset value (intellectual property,
financial assets)
Legal/regulatory costs
Lawsuits:
• Privacy suits
• Downstream liability
• Legal fees
Regulatory issues:
• Regulatory fines
• Remediation costs
Direct revenue
E-Commerce systems
Level of materiality
Seasons, cycles, forecasts drive expected
losses
Some benchmarks: shrinkage; materiality
(internal controls)
Stored asset value
Stored Value (financial assets)
Stored Knowledge (intellectual property)
Market Cap (or equivalent) – Book Value = Goodwill (intangible assets)
Some % of this Goodwill is attributable to information assets.• Professional services – higher percentage• Contract manufacturing or retail - lower
Determining loss
No physical goods
Ubiquitous supply
Full asset value is not necessarily lost
Look at loss in other ways:
• Type of loss
• For each application/system
Types of losses
How much value would be lost under the following conditions (for each app/dataset)?
Information-centric loss• Modified data (Integrity)• Copied data (Confidentiality)• Deleted data (Availability)
System/App-centric loss• Resource availability (Productivity)• Resource misuse (Liability)
Loss potentialRead Modify Delete Avail Misuse
Asset
Value
H M M L L
Revenue M H H H L
Fines M/H H L L ?
IT Prod. L H M L M
EU Prod. L L M H M
Calculating potential loss
Annual Loss Expectancy = Probability x ValueALE = P x A
(Insurance Industry)
Level One: Calculate overall loss potential in 5 categories.• ALE = P x L(Assets, Revenue, Fines, IT Prod, EU Prod)
Level Two: Take above and factor in types of losses.• ALE = P x (C(A,R,F,I,E); I(A,R,F,I,E); A(A,R,F,I,E))
Level Three: Perform above for all applications/data.• ALE = P x App1(C(A,R,F,I,E); I(A,R,F,I,E); A(A,R,F,I,E))…
Appn(C(A,R,F,I,E); I(A,R,F,I,E); A(A,R,F,I,E))
Getting to ROSI
Determines cost effectiveness of proposed
solution.
Calculate losses with and without solution.
Compare the difference.
