1

USE M ETR I C S DASHBOAR D STO M ANAGE

ENTER P R I SE SEC UR I TY R I SKS

Session 5302

2

S E S S I O N S U P P O R T E D B Y A S I S F O U N D AT I O N A N D A S I S D E F E N S E A N D I N T E L L I G E N C E C O U N C I L 2

PresenterPeter OhlhausenPresidentOhlhausen Research, Inc.

M E T R I C S D A S H B O A R D S

PresenterDaniel McGarveySenior Principal Business Process AnalystAlion Science and Technology

PresenterRichard WeaverChief Security OfficerHead, Security SvcsDept.Johns Hopkins University Applied Physics Laboratory

ModeratorCheryl StoneDirector, Corporate Security & SafetyRAND Corporation

3

O U T L I N E

I. ASIS Foundation metrics research project, Security Metrics Evaluation Tool, and ongoing research

II. Presenting metrics data to C-suite

III. Metrics dashboards for decision making and ROI demonstration

3

M E T R I C S D A S H B O A R D S

4

A S I S F O U N DAT I O N M E T R I C S R E S E A R C H

Persuading Senior Management with Effective, Evaluated Security Metrics (2014)

• Nine criteria for evaluating metrics--Reliability, Validity, Generalizability--Cost, Timeliness, Manipulation--ROI, Org. Relevance, Communication

• Library of evaluated metricshttps://foundation.asisonline.org

• Please contribute your metric at https://www.surveymonkey.com/r/metrics-survey

4

M E T R I C S D A S H B O A R D S

5

S E C U R I T Y M E T R I C S E VA L UAT I O N TO O L ( S E C M E T )

• Discern strong and weak points of a security metric

• Refine metric to optimize its scientific merit, operational reasonableness, and strategic relevance

• More persuasive to senior management

5

M E T R I C S D A S H B O A R D S

6

C R I T E R I O N 5 : T I M E L I N E S S

Extent to which metric data can be gathered in a timely fashion so the results can have an impact.

6

The data for this metric is

out-of-date by the time it

can be gathered and

interpreted; the data

collection process is very

time-consuming; the data is

unlikely to have an impact (as

it does not reflect current

conditions).

2

The data for this metric is

fairly up-to-date by the time

it can be gathered and

interpreted; the data col-

lection process is somewhat

time-consuming; the data is

somewhat likely to have an

impact (as it somewhat

reflects current conditions).

4

The data for this metric is very

up-to-date when gathered

and interpreted; the data

collection process is not time-

consuming; the data is very

likely to have an impact (as it

reflects current conditions).

1 3 5

7

I T ’ S I M P O R TA N T T O T H I N K A B O U T T H E WAY O N E M A K E S D E C I S I O N S

• Halo effect

• Outcome/hindsight bias

• Confirmation bias

• Regression to the mean

• Wet bias

7

M E T R I C S D A S H B O A R D S

A I M F O R L E S S W R O N G N E S S

8

K A R L P O P P E R ,P H I L O S O P H E R O F S C I E N C E

How can we hope to detect and eliminate error? By criticizing the theories or guesses of others and—if we can train ourselves to do so—by criticizing our own theories or guesses.

Conjectures and Refutations: The Growth of Scientific Knowledge, 1963

8

M E T R I C S D A S H B O A R D S

9

O B S E R VAT I O N S O N N E W M E T R I C S

• Metric of completed guard tours: does it discourage stopping to address a problem?

• Metric of driving time saved by conducting investigations long-distance: does it adequately consider quality factors or lean toward speed, convenience, and cost?

9

10

P R E S E N T I N G M E T R I C S TO C - S U I T E

Corporate management tends to view security as overhead (cost center, not production center) and security metrics as merely measuring activity, not value.

Security benefits are difficult to measure compared to the benefits of profit centers. Security professionals often lack the skills or time to create and administer effective metrics.

Thus, current security metrics, in practice, are generally not compelling and are often not taken seriously(Rothke, 2009).

10

11

P R ES E N T I N G M E T R I C S TO C - S U I T E

11

Make Metrics Compelling: an overview

Present metrics that are aligned with the organization’s objectives or risks or that measure the specific issues management is most interested in.

Present metrics that meet measurement standards.

Tell a story.

Use graphics, and keep presentations short.

Present metric data regularly.

12

P R ES E N T I N G M E T R I C S TO C - S U I T E

12

Align with Organizational Objectives and Risks

Risk: Metrics-based approach helps senior management understand the level of risk in site selection and make informed decisions on risk management.

ROI: There is a clear link between reducing shrinkage and saving money. Your metrics must demonstrate that investment in security technology led to reduced losses.

13

13

Risk Vs Return on Investment

14

P R ES E N T I N G M E T R I C S TO C - S U I T E

14

Present Metrics That Meet Measurement Standards

Metrics are quantitative and exude scientific authority. However, if metric is based on invalid or unreliable data, you cannot draw accurate conclusions from it and it will lack external credibility.

A metric that has been properly designed from a scientific point of view and that has been evaluated against a testing tool (such as the Security MET) may appear more valuable and persuasive to senior management.

Using a metric that meets measurement standards also provides an objectivity that aids decision-making.

15

© Pherson associates, llc. all rights reserved. www.pherson.org 15

Risk

Measurement Standards

16

P R ES E N T I N G M E T R I C S TO C - S U I T E

16

Tell a Story

Can be a story about the specific risk that security is attempting to mitigate, as well as the consequences if the event occurs. Be straightforward about risk and uncertainties.

Part of a compelling story is the unfolding of events over time. Metrics can show progress toward meeting a specific strategic goal.

Benchmarking can enrich a story if it is aligned with strategic organizational goals. Benchmarking provides the opportunity to ascertain where company stands on a given metric in relation to its competitors.

17

17

Security Threat

Risk Mitigation

18

P R ES E N T I N G M E T R I C S TO C - S U I T E

18

Use Graphics, and Keep Presentations Short

Keep it simple and clear. Present a few short bullet points—top-level information only, rather than complex charts and graphs.

Less is more.

Pick graphics that get your points across.

One graphic = 1,000 words.

Keep presentation short (but still tell a story).

Present metrics in the style or format management uses.

19

19

20

P R ES E N T I N G M E T R I C S TO C - S U I T E

20

Present Metric Data Regularly

Data ages over time.

Distinguishing metrics that are time-sensitive from those that provide value over time will enhance the overall value of metrics.

Comparing historical data against current data will show trends.

Do not hide painful data from management.

Good metrics are the key to demonstrate ROI.

21

S E C U R I T Y M E T R I C S : W H AT T O M E A S U R E ?S O M E G U I D I N G P R I N C I P L E S

• Be mindful that the process of collecting data and reporting metrics can be extremely time consuming and may unintentionally divert staff from performing work that needs to get done

• Therefore, confine metrics to only those things that provide useful insight into aspects of operations that are actionable and will lead to delivering improved service to customers and/or will reduce security risks

• Make every effort to determine the most critical concerns of senior management, and implement metrics that link to those concerns and that will demonstrate value and return on investment

• To the extent possible, leverage technology and automation to collect and analyze metrics data, thereby avoiding or minimizing manually intensive processes

21

22

T H E $ 6 4 , 0 0 0 Q U E S T I O N : W H AT TO M E A S U R E ?

Example: Enterprise Classified IT Security

- Number of Systems Administrators and ISSOs- Number of Systems (overall)- Number of Networked Systems- Number of WANs- Number of Classified SSP Submissions- Number of Incomplete SSP Submissions Returned - Number of Classified ATOs Received- Number of Users Trained- Number of Authentication Tokens Distributed - Media Write Access Authorized - Number of Privileged Users- Number of Authorized Data Transfer Agents- Number of Classified VTCs Conducted - Number of Mobile Devices- Number of IT related Security Violations and Infractions - Number of Systems Involved in a Classified Spill- Median Number of Days to Receive a Classified ATO- Results of Accreditation/Oversight Inspections- Results of Customer Satisfaction Surveys 22

23

S E L E C T I N G M E T R I C S T H AT B E S T F I T YO U R S E C U R I T Y O P E R AT I O N S ( V 3)

Volume

Numbers (counts) to track and assess level of security activity

Easiest to collect

Useful in defending, adjusting and seeking additional resources

Velocity

Data to capture and assess speed of delivering a security product or service

Useful in evaluating process efficiencies and identifying opportunities for improvement

Helpful in communicating expectations to customers, partners and stakeholders

23

24

S E L E C T I N G M E T R I C S T H AT B E S T F I TYO U R S E C U R I T Y O P E R AT I O N S ( V 3)

Value

Metrics to demonstrate the importance of Security to the overall health and productivity of an organization, capturing key care-abouts of senior management

Harder to identify, develop and measure

Highlights Return on Investment (ROI) by answering “so what” questions

May include Volume and Velocity data but will be outcome oriented

Helpful in providing high-level situational awareness of threats, vulnerabilities and success of mitigating countermeasures

Assists Senior Management in making decisions to accept risk, or to take action to lower risks

Displays/dashboards are useful, and anecdotes (stories, narrative example and explanations) are important to accompany numbers

24

25

S E L E C T I N G M E T R I C S T H AT B E S T F I TYO U R S E C U R I T Y O P E R AT I O N S ( V 3) :

S O M E E X A M P L E SVolume

Visit requests and clearance certifications processed; badges/tokens fabricated and issued; internal access control transactions; security incidents reported; foreign travel and other Security/CI awareness briefings administered

Velocity

Personnel Security clearance cycle time (nomination to indoctrination); IT accreditations (timelines associated with submission of plans to ATO); response time to alarm annunciations and other emergency circumstances

Value

Corporate savings (cost avoidance) attributable to security actions taken; security systems reliability; compliance inspection, audit and red team assessment results; elimination or reduction of undesirable events 25

26

D ATA TO D A S H B O A R DO N E WAY I T ’ S D O N E :

• Data is collected both in real time and on a periodic basis, depending on customer and senior management requirements, and on intended use in security operations, to include adjustment of resources

• Data is collected from a variety of sources: Excel spreadsheets, external SASS (Service Now) and other databases; subsequently using Microsoft SSIS, data is loaded and transformed into Microsoft SQL Server database

• Once data is collected and aggregated in a local data mart, then metrics are calculated and displayed via a SharePoint portal utilizing Power BI

26

27

27

28

28

29

29

30

30

31

31

32

32

33

33

34

34

35

35

36

36

37

37

38

P R E S E N T I N G P I C T O R I A L D I S P L AY S O F M E T R I C S D ATA T O S E N I O R M A N A G E M E N T :

T H E G O L D E N R U L E

CHARTS, GRAPHS, DASHBOARDS, DIAGRAMS, TABLES AND ILLUSTRATIONS SHOULD BE USED ONLY SELECTIVELY

AS A TOOL TO MAKE KEY POINTS

38