deloitte - kpi and measuring security

“You Can’t Manage It If You Can’t Measure It

ISACAMarch 2006

© Deloitte & Touche LLP and affiliated entities.

Agenda

• Do you know how well your information security program is working?

• Key Performance Indicator (KPI)

• Key Performance Index (KPX)

• Information Collection

• Examples

• Summary


What do we have to be worried about?

The time between the discovery of a vulnerability and the potential exploit is diminishing from months to days if not hours


IT Security Governance Maturity Model

• The Maturity Model is sponsored by the IT Governance Institute

• It is used to rank the maturity of an organization’s practices and standards against industry best practices and standards

• It can be used to help guide an organization on the areas that will improve their overall information security posture


How do you know if you have an information security program that effectively manages risks?

• Obtain a high score on an ISO 17799 assessment?

• Complete regular, active penetration tests with no discovered vulnerabilities?

• Have an acceptably low # of security incidents reported using the Incident Response process?

• Have an effective virus program (few or no infections and any infections are managed effectively with little interruption)?

• Have Measurable Service Level Expectations (SLE) that are consistently being achieved?

• Have an effective IDS program (# and type of alerts are being managed effectively, little impact on the business, in line or better than industry benchmarks)?

• Obtain certification against an information security reference standard (ISO 27001)?


There are several problems to avoid when establishing an information security measurement program

• Lack of management commitment

• Measuring too much, too soon

• Measuring too little, too late

• Measuring the wrong things

• Imprecise metrics definitions

• Using metrics data to evaluate individuals

• Using metrics to motivate, rather than to understand

• Collecting data that is not used

• Lack of communication and training

• Misinterpreting metrics data


Key Performance Indicators (KPIs) can help determine the current status of the information security program

• A key performance indicator is a measure of a particular organizational performance activity, or an important indicator of a precise health condition of an organization

• Used as an indication of the current state of a component of the business to take the “surprise” out of risk

• To be effective, the KPI must be defined as succinctly as possible

• Can be measured as an “improvement” from a known state or a reference standard


A Key Performance Indicator . . .

• Must be something that can be measured and continued to be measured

• Must be precise, meaningful and understandable

• Must be relevant to the business

• May be required by legislation and/or Regulations

• Must have a measurement index that has meaning

• Must have an appropriate life (Stickiness)

• Should be tied to the organization’s vision and strategy


Types of Key Performance Indicators (KPIs)

• Threshold – when an index reaches set targets or falls into set ranges

– e.g., ETS scores on defined risks

• Milestone – when a specific condition is reached

– e.g., certification

• Quantitative – measure of value (number, time, $, etc.)– e.g., number of reported security incidents, lost time due to

viruses

• Qualitative – measure of acceptability or health – e.g., survey ratings, rating of risks


Examples of Key Performance Indicators

• Awareness

• Knowledge of policies, standards and procedures (surveys and tests)

• Risk Assessment

• Depth and breadth of regular risk assessments across the enterprise (When was the last assessment? Qualitative measure of the risks, risk index)

• Risk Management

• Number of incidents reported, amount of loss incurred, number of situations managed

• Audit

• Noted deficiencies against the policy and standards (measured year over year)

• Benchmarks and Certification

• Maintaining/following IT security certifications such as FIPS 140-1, ISO 27001, ISO 15408 (Common Criteria)


Possible Non-Risk Key Performance Indicators (KPIs)

• People– Training & Certifications

– Competence Turnover

• Technology– Currency

– Cost management

– Compliance / licensing

• Investment– Trends per area

• Effectiveness & Return on Investment– Key Risk Indicator experience vs. cost

• Productivity

– Missed Deadlines


KPIs can be used to measure the Effectiveness of Investment (EOI)

• A Return on Investment (ROI) for information security is difficult to measure since risk, and especially risk reduction, is challenging to quantify in terms of dollars.

• The Effectiveness of Investment (EOI) could be the comparison of the effectiveness of the security measures with the value of the investment.

• For example, the number and impact of viruses and worms can be compared with the investment in virus detection technology and support programs.

• A collection of KPIs could be used to measure the EOI for information security


A Key Performance Index (KPX) is a summary or correlation of one or more KPIs that provides an indication of the overall performance of a defined area of the security program

• May prompt the organization to change strategic direction in information security

• Levels may be triggered by a variety of factors

• Must be meaningful and understandable

• Must be relevant to the business

• Must have a measurement index that has meaning

• Must have an appropriate life (Stickiness) and

• Should be tied to the organization’s vision and strategy


Example KPI Format

Any additional information or comments? Is this a requirement from legislation or regulations?

Comments

___ Day ___ Week ___ Month

___ Quarter ___ Year ___ Year+

Frequency

Any potential tools used to support the measurement and reporting process?

Tools

Method used to measure the KPI Method

What does it apply to?Unit/Dept

__ Low __ Medium __ HighEffort

__ Quantitative ___ Qualitative ___ Milestone ___ ThresholdType

Who is this KPI relevant to?Stakeholder

What are the objectives of the KPI – what is it measuring? Why is it important?

Objective

Description of the KPI – what does it address?Description

Short name or title for the KPIKPI Name


Example Key Performance Indicator (KPI)

Need to have confidence in the detection and reporting mechanisms to be able to measure changes to the index over time. A lower index will then mean less risk

Comments

___ Day _X_ Week _X_ Month _X_ Quarter _X_ Year ___ Year+Frequency

IDS and/or security management/reporting softwareTools

Count number of reported security incidents/events at low, medium and high severity over the past week

Method

Information SecurityUnit/Dept

__ Low _X_ Medium __ HighEffort

_X_ Quantitative ___ Qualitative ___ Milestone ___ ThresholdType

CSIO, CIO, Operations Management, Technology ManagementStakeholder

A measure of the relative size and effectiveness of the organizations risk management processes

Objective

Provides a relative index on the current number of reported security incidents/events at differing security levels for the recent reporting week

Description

Weekly Reported Security IncidentsKPI Name


Example Key Performance Index (KPX)

Need to have confidence in the detection and reporting mechanisms to be able to measure changes to the index over time. A lower index will then mean less risk

Comments

___ Day _X_ Week _X_ Month _X_ Quarter _X_ Year ___ Year+Frequency

IDS and/or security management/reporting softwareTools

Count number of reported security incidents/events at low, medium and high severity over a defined time frame

Method

Core SystemsUnit/Dept

__ Low _X_ Medium __ HighEffort

_X_ Quantitative ___ Qualitative ___ Milestone ___ ThresholdType

CSIO,CIOStakeholder

A measure of the relative size and effectiveness of the organizations risk management processes

Objective

Provides a relative index on the current number of reported security incidents/events at differing security levels within a specified time frame

Description

Information Security Risk Management IndexKPI Name


Several automated tools can provide a view of security incidents and trends


Security Incidents - Advanced Forensic Tools


The Information Security Program should include a reporting mechanism that provides a single point of reference for concise, executive-level information for business and technology owners.

The dashboard aims to transform data from operations to actionable information for decision makers

Sample Security DashboardOperator Event View

Reports

Incident Tracking (Ticketing System)

Geographic Threat View

Trend View

Advanced Forensic Tools

Geographical Dashboard View


An analysis of security incidents will contribute to the current status of the Information Security Program


Keep track of each area of concern that is the object of a KPI or KPX definition

Any additional information or comments?Comments

Any required acknowledgement or reporting for this KPI?Reporting

How does the KPI(s) map to the individual performance goals?Map KPI(s) to Performance Goals

What summary index(s) can be defined that is a high-level representation of one or more KPIs that are vitally important to the organization?

KPX(s)

What Key Performance Indicators(s) should be defined for this objective?KPI(s)

What are the measurements that may be available to report on this area?Measurements

What are the key control objectives and controls that should be in place for the organization? The controls should be based on international reference standards

Key Control Objectives and

Controls

What is the main objective – how is it measured? – Why is it important?Objective

What is the Vision and Mission statement that directs IT security?Vision/Mission

Topic - <What is the KPI or area of concern?>

© Deloitte & Touche LLP and affiliated entities.Presentation Name (View / Header and Footer)

An example KPI for Inappropriate Use

Inappropirate Use - KPXThe impact of recorded

inappropriate use events compared to the amount of

IT security awareness training per person.

KPI - 1Number of verified

instances of inappropriate use over a set time

period. (weekly or by reporting period)

KPI - 2

Impact of inappropriate use events to the business in terms of resources and or loss over time (weekly or

by reporting time)

KPI - 3Number of verified

inappropriate use events compared with the number of IT security awareness training days per person

compared over time

Measurement - 1Number of inappropriate use cases opened and

verified

Measurement - 2Amount of service lost to

inappropriate use

Measurement -3Number of IT security

awareness training days


An example KPX for Inappropriate Use

KPX


An example KPI for Intrusion Detection

KPI - 4Cost of the IDS program in relation to the number and impact of detected events

Measurement - 4The number of systems with active monitoring capabilities

KPI - 3Number of IDS program failures Measurement - 5

Number of Sensors per network segment

Measurement - 6Cost of the hardware and/orsoftware to implement intrusiondetection sensors

IDS KPXThe measureable amount of productivity loss attributedto intrusions in relation to thethe number of events and thecost of the IDS program.

KPI - 1Average amount of Loss (productivity time) per intrusion within a set time period (weekly or per reporting period).

Measurement - 1Number of incidents of intrusions detected and reported

Measurement - 3Amount of downtime or productivity loss caused by intrusion incidents.

Measurement - 2Number of incidents of intrusions impacting the organization that were not reported

KPI - 2Number of events caught andprevented by the IDS within aset time period


An example KPX for Threat Management–Intrusion Detection System (IDS)

Number of Resolved Major and Catastrophic Incidents Over Time

Time/ Reporting Period

# of Resolved Major and

Catastrophic Incidents

11 22

33

Number of Major and Catastrophic Incidents Over Time

Time/ Reporting Period

# of Major and


High Risk IncidentsHigh Risk Incidents

Critical IncidentsCritical Incidents

Average Time to Resolve a Number of Major and Catastrophic Incidents

Average Time to Resolve Major and Catastrophic Incidents

# of Resolved Major and


Major Incidents

CatastrophicIncidents

Number of Resolved Major and Catastrophic Incidents

Major IncidentsMajor Incidents

CatastrophicIncidentsCatastrophicIncidents

Number of Resolved Major and Catastrophic Incidents

>4<10 hrs/month/ system productiv ity loss

>4<10 hrs/month/ system productiv ity loss>10hrs/month/ system productiv ity loss>10hrs/month/ system productiv ity loss


Summary

• A good collection of Key Performance Indicators will provide an overview of the current status of risk management within the organization –Use the collection of KPIs as an information security dashboard

• The KPIs can be used to help comply with legislative or regulatory requirements–Provide the information that can be used for reporting purposes

• The KPIs must be carefully selected and defined to be useful –Must be meaningful and measurable

• Effective KPIs can be used to demonstrate good management of risk –For example, KPIs may provide a financial institution the ability to

reduce the percentage of reserve required to offset operational risk defined by the Basel II Accord

Questions?Glen Bruce, [email protected]

Member ofDeloitte Touche Tohmatsu


Deloitte, one of Canada's leading professional services firms, provides audit, tax, consulting, and financial advisory services through more than 6,100 people in 47 offices. Deloitte operates in Québec as Samson Bélair/Deloitte & Touche s.e.n.c.r.l. The firm is dedicated to helping its clients and its people excel. Deloitte is the Canadian member firm of Deloitte Touche Tohmatsu.

Deloitte refers to one or more of Deloitte Touche Tohmatsu, a Swiss Verein, its member firms, and their respective subsidiaries and affiliates. As a Swiss Verein (association), neither Deloitte Touche Tohmatsu nor any of its member firms has any liability for each other's acts or omissions. Each of the member firms is a separate and independent legal entity operating under the names "Deloitte," "Deloitte & Touche," "Deloitte Touche Tohmatsu," or other related names. Services are provided by the member firms or their subsidiaries or affiliates and not by the Deloitte Touche Tohmatsu Verein.

deloitte - kpi and measuring security

Documents