Cyber Metrics in the DoDor

How Do We Know What We Don’t Know?

Cyber Metrics in the DoDor

How Do We Know What We Don’t Know?

John S. Bay, Ph.D.Executive DirectorJohn S. Bay, Ph.D.Executive Director

Things People Have Asked MeThings People Have Asked Me

• How much money should I spend this year on cyber defense technologies?

• How many attacks has your firewall repelled this month?

• If I only had a dollar to spend on cyber, where should I spend it?

• Why is cyber research such a slog?

211/12/14

AnswersAnswers(which did not go over well)(which did not go over well)

• How much money have you got?

• We repelled all of them … except that one you read about in the paper

• Spend your dollar on upgrades

• Cyber research is a slog because there is no physics theory underlying it all, liker Maxwells’ Equations or Newton’s Laws

311/12/14

But really … it DEPENDSBut really … it DEPENDS

• The “threat” factor is common in cybersecurity, but mostly not elsewhere

• … and it IS true that there is no useful PHYSICS for the problem

411/12/14

DoD Taxonomy of ThreatsDoD Taxonomy of Threats

5

From: Defense Science Board, Resilient Military Systems and the Advanced Cyber Threat, January 2013

Tier Description

I Practitioners who rely on others to develop the malicious code, delivery mechanisms, and executionstrategy (use known exploits).

II Practitioners with a greater depth of experience, with the ability to develop their own tools (from publically known vulnerabilities).

III Practitioners who focus on the discovery and use of unknown malicious code, are adept at installing user and kernel mode root kits, frequently use data mining tools, target corporate executives akey users (government and industry) for the purpose of stealing personal and corporate data with the expressed purpose of selling the information to other criminal elements.

IV Criminal or state actors who are organized, highly technical, proficient, well funded professionals working in teams to discover new vulnerabilities and develop exploits.

V State actors who create vulnerabilities through an active program to “influence” commercial products and services during design, development or manufacturing, or with the ability to impactproducts while in the supply chain to enable exploitation of networks and systems of interest

VI States with the ability to successfully execute full spectrum (cyber capabilities in combination withall of their military and intelligence capabilities) operations to achieve a specific outcome in political,military, economic, etc. domains and apply at scale.

11/12/14

And The Corresponding CriticalityAnd The Corresponding Criticality

611/12/14

What Might the COSTS Be?What Might the COSTS Be?

711/12/14

So Then, What to Measure?So Then, What to Measure?

• Qualitative– Capabilities– Missions lost

• Quantitative– Performance– Cost• To achieve• Not achieving

811/12/14

Capabilities and MaturityCapabilities and Maturity

911/12/14

Dashboard ApproachDashboard Approach

1011/12/14

““Stoplight Chart” AssessmentsStoplight Chart” Assessments

1111/12/14

See: SPIDERS JCTD

Costs to UsCosts to Us• All vulnerabilities are bugs • All code has bugs• Bugs are expensive• Exploits are cheap the “asymmetry” problem

1211/12/14

Mission-Assurance ApproachMission-Assurance Approach

• Helps focus attention• Requires a “map” o the

mission• Implies a prioritization

on missions (something loses)

• Requires reconfigurable systems and networks

• Is not cheap

13

From: DUSD(I&E) Office, HANDBOOKFor SELF-ASSESSING SECURITY VULNERABILITIES & RISKS of INDUSTRIAL CONTROL SYSTEMSOn DOD INSTALLATIONS, December 201211/12/14

Just Good Enough (Incremental)Just Good Enough (Incremental)ApproachApproach• How long would our red team take to penetrate the

system?– An empirical measure, at best.– Implies a canonical red team

14

prob(first vulnerability is discovered)

time

Bad code

Better code

Gamma distribution?

11/12/14

The Accountability ApproachThe Accountability Approach

• NIST 800-53 guidelines

• The “did we do everything we know how to do” approach

15

From: NIST Special Publication 800-53, rev. 4, Security and Privacy Controls forFederal Information Systems and Organizations, April 201311/12/14

Conclusions: Which is Best?Conclusions: Which is Best?

• None of them. They service somewhat orthogonal purposes.– But they can provide apples-to-apples comparisons

• Can they answer the Generals’ questions?– No– … except maybe the one about the firewall– There is CERTAINLY no satisfactory “physics” to guide anybody

• Cyber Metrics is still an extremely important and high-priority problem for OSD!

1611/12/14