1

CYBER METRICS

Gettingtheconversationstraightbetweentechnicalandnon-technicalactors

SamBandMadelineCarrJune 2018

2

Research Institute in Science of Cyber Security

TheResearchInstituteinScienceofCyberSecurityistheUK’sfirstacademicResearchInstitutetofocusonunderstandingtheoverallsecurityoforganisations,includingtheirconstituenttechnology,peopleandprocesses.RISCSisfocusedongivingorganisationsmoreevidencetoallowthemtomakebetterdecisions,aidingtothedevelopmentofcybersecurityasascience.Itcollectsevidenceaboutwhatdegreeofriskmitigationcanbeachievedthroughaparticularmethod.Thisinvolvesnotonlythecostsofitsintroduction,butongoingcostssuchastheimpactonproductivity–sothatthetotalcostcanbebalancedagainsttheriskmitigationthatisbeingachieved.RISCSmaingoalistomovesecurityfromcommon,establishedpracticetoanevidencebasecomparabletootherevidence-basedsciencesandpracticeslikemedicine.RISCSismanagedbyateambasedintheDepartmentofScience,Technology,EngineeringandPublicPolicy(STEaPP)atUniversityCollegeLondon(UCL).Tofindoutmorevisit:www.riscs.org.uk

Report authors:

Sam B: SamBisaresearcherintheNationalCyberSecurityCentre’sSociotechnicalSecurityGroup(StSG).Samhasspentoverfifteenyearsworkinginavarietyofsecurityroles,includingbackup&recoverysystems,threatassessment,cybersecurityconsultancyandcustomerliaison.Mostrecently,hespenttwoyearsworkingalongsidetheHealth&SocialCaresector.Samisabigbelieverthatpeoplearethegreatestassettotheirorganisation’ssecurityeffortswhenempoweredtoactasleadersandinnovators.

Madeline Carr: DrCarrisAssociateProfessorofInternationalRelationsandCyberSecurityatUniversityCollegeLondonandtheDirectoroftheRISCSInstituteforresearchintothescienceofcybersecurity.SheisalsotheDirectoroftheDigitalPolicyLabwhichsupportspolicymakingtoadapttothepaceofchangeinsociety’sintegrationofdigitaltechnologies.HerresearchlooksatthewaysinwhichnewtechnologybothreinforcesanddisruptsconventionalframeworksforunderstandingInternationalRelationsandtheimplicationsofthisforstateandglobalsecurity,orderandgovernance.DrCarrhaspublishedoncybernorms,InternetFreedom,multi-stakeholderInternetgovernance,andthepublic/privatepartnershipinnationalcybersecuritystrategies(researchfundedbytheBritishCouncil).HerbookUSPowerandtheInternetinInternationalRelationsispublishedwithPalgraveMacMillan.DrCarrisCo-leadontheStandards,GovernanceandPolicystreamoftheUK’s£24MPETRASresearchhubonthecybersecurityoftheInternetofThings.SheisalsothePIonanEPSRCfunded(£480K)projectlookingatthewaysinwhichcybersecuritypolicymakersevaluateevidence,PIonanNCSC/LRFfunded(£1M)‘SupportingtheBoardinCyberRiskDecisionMaking’project,andPIonanEPSRC(£280K)projectlookingatinternationalcooperationoncriticalinfrastructureintheIoT.

Acknowledgements:

Theauthorswouldliketoacknowledgethefollowingpeoplewhohelpedtodevelopandrunthecybermetricsworkshop.UchennaAni,EmmaBowman,IrinaBrass,AlexChung,FejaLesneiwska,KruakaePothong,IneSteenmans,andLeonieTanczer.

3

Summary

OnMay23,2018,RISCSheldaworkshopinLondonthatlookedattheutilityofcybersecuritymetrics.Thepurposeoftheworkshopwastodevelopadeeperunderstandingofthewaysinwhichcybersecuritymetricsareusedindecision-makingmoregenerally,andalsotoraisequestionsabouthowdataisbestpresentedtotheboardandthepolicycommunitymorespecifically.Wewantedtoexplorethepotentialformetricstohelpbutwealsowanttotakeacriticalapproachtotheunderlyingvaluesthatcanshapemetrics–andconsequently,decisions.

4

METHODOLOGY:

Toinvestigatetheutilityofcybersecuritymetricsinthedecisionmakingprocessofindustryandthepolicycommunity,wegatheredagroupof70peoplefromacademia,industry,thepolicycommunityandthetechnicalcommunity.Weaskedthesepeopletoself-identifythemselvesas‘providers’or‘consumers’ofmetricsandtoindividuallyorcollaborativelyrecordtheirresponsestofourquestionsthatweaskedoftheirgroup.Forthecybersecuritymetricsprovidergroup,weaskedthemtopopulatethefollowingtable:

Weaskedthosepeoplewhoidentifiedastheconsumersofcybersecuritymetrics(decisionmakersaboutinvestment,policyetc)torespondtothequestionsonaseparatetable:

AcompletetranscriptoftherecordedresponsesisincludedatAppendixA.Onthefollowingpages,wepresentouranalysisofthefindings.

5

SUMMARY OF FINDINGS

Providingtailoredcybermetricsisanopportunitytoengagewithleadersandshapetheirperceptionsofinformationrisk.Theoutputsofthisworkshopsuggestthatsuccessdependsonprovidersdeliveringmaterialthat:

• genuinelyreducesuncertainty;

• addressesspecificquestions;and,

• usesthelanguageofbusiness.

Thetablebelowbringsoutthetrendsfromourworkshopintermsofwhatisandisnotrequiredbydecision-makers.Belowaresomeofthemostinterestingconclusionsthatwehavedrawnfromthedata.

• Wesawthatsomerequirementsmaybemetwithreticencefrommetricsproviders,perhapsbecausetheyareincendiaryorembarrassing(e.g.rankings,disclosures,overlyambitiousinformationsharingregimes).Providerstendnottowanttodeliverunwelcomenews,forexampleproofthatpastinvestmentshavedeliveredlittlebenefit.Themostsignificanttensionappearstobebetweentheneedtoinformfinancialdecisionsandthereluctancebysomeproviderstodelivermetricswhichover-promiseonthatfront.Theextenttowhichthissomethingtodowithtrustandliability,oralackofmutualunderstandingbetweenconsumerandproviderwouldrequirefurtherstudy.

• Theresponsesalsorevealedasenseofmistrustinmetricsdeliveredbysomecommercialproviders.Whilethesemetricswouldbeincreasinglyusefulasmoreservicesareoutsourced,therewasafeelingamongsomeparticipantsthatcommercialserviceprovidershadn’tnecessarilythevestedinteresttoprovideaccurateortimelymetrics,especiallyifServiceLevelAgreementswerebeingbreached.

• Someconsumerrequirements–whilevalid–aredifficulttoachieve,perhapsbecauseoffinancialconstraints,frailtyofcommercialproductsorlackofqualitydata.Forexample,wesawafewcommentswhichsuggestedthatcommonvulnerabilityscanningtoolslackedaccuracy.Otherexamplesofmetricswhichcouldbedifficulttoproduceincluded:protectivemonitoringandalertingcapabilities,assessingthetruecostofanincident,gainingconfidenceincyberinsurancepolicies,understandingtheoverallcostsfromcybercrime,metricizingtheblockerstosuccessfulGDPRcompliance,anddelivering“micro-narratives”todecision-makers.

• Thereweresomenotableinstanceswhereprovidersofmetricscouldoffernewideas.Forexample,howmuchsystemsdowntimehadbeenencounteredduetorogueeventsoractivitieswhichwerebeyondthecontrolofnetworkmanagers.Someproviderssawvalueinengaginginadialoguewithleaderstorefineprioritiesandbuildconsensus,forexamplewhereexcessivecosthadbeenincurredduetonotinvestingearlierinsecurity.Therewasalsothesuggestiontoobtainaccurateimpactandvulnerabilityscoringthroughtable-topexercising.Someproviderswereinclinedtodeliverquiterevelatorymetrics–conditionalonfeelingsafetodoso–includingthereal“value”ofextendedsupportcontractsandevidenceofsensitivecompromisesandvulnerabilities.

6

WHATCONSUMERSWANT WHATCONSUMERSDON’TWANT

THECOMPETITION:• HowamIperformingagainstmycompetitors?• Whathappenedtomyoppositenumbersin

othercompaniesafterabreach?• Whatwastheimpact/costoftheirmistakes?• Benchmarking:howamIdoingcomparedtomy

peergroup?Whatistheprobabilityofabreachcomparedtomycompetitors?

• Made-upnumbers.• Feelingbombardedby

meaninglessornitty-gritty,technicaldata.

• Snapshotswithnotemporalcontext.

• RAG(Red,Amber,Green).• Unqualifiedopinion.• Biasorexcessivesubjectivity,

especiallywrappedupinsomethingsuccinctorscientific-looking.

• Fear-mongering.• Blame.• Salespitchforsnake-oilor

magicbullets.• Unstableorunrepeatablestats.• Styleoversubstance.• Jargon.• Olddata.• Inappropriateorineffective

visualisationtechniques.• Spin.

IT’SALLABOUTTHEMONEY• Relevance:whatismyreturnoninvestment?• Howdomycyberrisksaffectmyabilitytoraise

capital?• HowmuchriskamIunabletotransfer(e.g.

throughcyberinsurance)?• AmIcarryingcriminalliabilitythatIcan’t

transfer?• Gettingtorootcause:solvemanyproblems

withonefix.• Costofrecoveryvscostofcontrol(proactivevs

reactivepostureoncybersecurity,andcomparativecosts).

• What’stherisk/costassociatedwithdoingnothing?

CYBERSECURITYATAGLANCE• Somecommentspointedtowardsabasic

SecurityOperationscapability• Dashboarding,e.g.networkboundaryactivities

7

THREATS&VULNERABILITIES• Pastbreachesareagoodindicatoroffuture

vulnerabilities.Andmetricsofrealincidentsaremoreinformativethanpotentialones.

• Isthreatintelligenceused?Howusefulisit?Whereelseisitused?Whereisthreatintelrealandvaluable?

• Whataremypriorities,ratherthanjustloomingthreats?Don’tscareme,informme.

• Aretherequickwinsavailable?HowcanImakesomeprogressfast?Helpmeliftopportunitiesoutofthenoise–whatcanIdorightnow?

• Trackingcapabilityoflow-capabilitythreatactors.

• Impact,ratherthanquantity,ofincidentsisimportant.

THEUSER• Areusersabsorbingtraining?E.g.arethey

forwardingsuspiciousemailsontothesecurityteam?

• Securityvsproductivity:spottingwheresecuritypolicyisfatiguingpeopleandimpedingproductivity.

• Cultureandindicationsofuserwellbeingandbehaviour(andtheriskthatpresents).

THIRDPARTIES• Whereissensitivedatagoing?Whomisitbeing

sharedwith?• Howdoyoutrustthestatsprovidedtoyouby

thirdpartyserviceproviders?

8

APPENDIX A: Transcription of Cyber Metrics workshop responses

Part One: Responses from consumers of cyber security metrics

“Keepitcoming!”(Alreadyreceivethisandfindituseful)

• CommonVulnerabilities&Exposures(CVE)vulnerabilitydatabase

• Resultsofcyberdefencematurityassessments

• Detailsofpastincidents

• Timingseriesanalysisofpastbreaches

• RiskassessmentofITchangeprojects

• Patchingstatus

• CyberEssentials

• Consistenttimeseriesmetrics

• Organisations’strategy:relevanttohowthisbusinessmakesitsmoney

• Incidents,butbydept/functiontofindriskareas:simpletotalnumberisnotthatuseful

• RiskassessmentsofITchangeprojects:base,withcontrols,costsandoptions

• Returnoninvestment

• Barrierstouptakeofbehaviour

• Realincidentbreaches,ratherthanpotentialones

• Punishmentsreceivedbyotherboards’directors

• Overviewofattackattempts,especiallyovertime

• Livedashboard,e.g.AVlevels/status

• Metricsintermsweunderstand,e.g.businesslanguage

• Understandingperceptionsofrisk

• Doorganisationsunderstandwhatiscriticalforthem,i.e.whattoprotect?

9

• Malwarenotifications

• Qualitativeanalysisofwhat’suseful

• Whatismyuninsuredrisk?

• Datathatenablesmetodosomething,whatchangescantheorganisationdotorespond?

• Whatincidentshaveaffectedorganisationsandwhatisthetrend?

• Short,sharpandtothepoint

• Howdoesmylevelofuninsuredriskaffectmyabilitytoraisecapital?

• Metricsalignedtosolutions

• Levelofrealvaluableknowledgesharingwithinasector

• Doorganisationsreceiveanyusedthreatintelligence(understand)?

• Datathathasasupportingnarrative

• Surgecapacity,usedspace,e.g.whenit’s>80%,itshouldgiveyouanalert

• Usagepatterns,responsesinshort,mediumandlongterm

• AV/malwarecoverage

• No.ofinstallsuptodate

• Detectionsalerts:typesandpropagation

• Actions:cleanedandquarantined

• Networkcapacity:e.g.usedbandwidth/time>x%,ittriggersanalert

• Positionmetricsasatooltoenhancethebusiness,nottopresenthurdles

• Phishingtests

• Governanceofcybersecurity,seeCyberAssessmentFramework(CAF)

• Howdoesanexpenditureaffectmyuninsuredrisk?

• Bostonconsultancymatrix:Iwanttoknowmoreaboutmy“star”and“cashcow”areasthanmy“dog”ormy“questionmarks”

• Areaswherewearenotcompliant

• Howdowehavetheconfidencetodiscusscyberonceayear,ratherthanonceamonth?

• HowismycybersecuritypostureaffectingmydefencewithrespecttocriminalliabilitythatIcannotconvert?

• Small&Medium-sizedEnterprises(SME)withCyberEssentials(Basic&PLUS),IASMEorotherlevelofsecuritymanagement

10

Wishlist(Don’tcurrentlyreceivethisbutwouldliketo)

• ConfigurationstatusofthecorporateIT:noofdevices,softwarerunning,knownvulnerabilitiesmappedfromCVEdatabase,plusseverity

• Sensordata:typesofattack,accessvectors,effects,impact

• Effectivenessoftrainingandawarenesspackages,e.g.howmanypeopleclickedthelinkduringaspear-phishingcampaign?

• Suitablequantitativemetricsforhumanfactors(notsureifthisispossible)

• Physicalsecurity:notseenmuchinthisarea

• Riskassessmentandassociatedinvestmentplan

• Securebehavioursvstarget/norm,e.g.phishisreported

• Supportinganalysis/datathatgivemoredetailswhenneeded

• Awareness,engagement:minusdone+howwelldone,e.g.lingertime

• Metricsaren’tjustaboutnumbers

• Dayslostperyearduetosecurity‘features’

• Quantitativeandqualitativedata(holisticview)

• Userawarenesslevel

• Robustnessofchangemanagementprocessinorganisation

• TimetofixbytheISP/SystemIntegrator

• ITprojectswithsecuritydesignedasaproportionofmereexistenceofsecuredesignpractice

• Likelihoodofthebreaches(supportedbyrobustmodel,e.g.[couldn’treadit])

• Insecurebehaviours:clickingdodgylinks,useofUSBs,useofdropbox,webmail

• Doingtrainingtooquickly,ignoringawarenessmaterial

• Realincidentdata

• Impactofbreaches

• Metricsthatshowimpactofincidentsnotjustnumberofincidents

• Behavioursthatcouldbeinsecure,whoclickslotsoflinks,whobrowsesalot?

• Anythingthatdemonstratesimpactonfinancialaccountingmetrics

11

• Thirdparties

• Goodbenchmarks

• Threatactoractivity

• Relevantbusinessactivity

• Puttinginformationinthecorporatecontext,thinkaboutourannualreport

• Sector-by-sectormetricswithuniformmethodology

• Qualitative,outcomes-baseddata

• Networkmapsofhowdifferenttypesofattackaremaybecausedbythesamerootcauses

• Aquicksummaryforhighlevelexecutivepack

• Identifiedcybersecurityrequirementstobeimplementedbyusers

• Quantitativeconsequencesfromcasestudiesofsimilarcompanies

• Whenweareinmergersandacquisitions:overviewofsystemintegrationriskaspartofbidcostvsbringingdataontoourexistingsystem

• Numberofsilentconnectionstomyphone/device,theirthreatlevelandactionablestepstoreducerisks

• Resilienceofmydevicestodifferenttypesofattack

• McGraw/BSIMMdata

• Numberofpasswordsre-used/repeatedacrosswebsitesandservices

• Metricsmusthavecontext,otherwisethey’rejuststats

• Mandatorycriteriatobenchmarkcybersecuritystatusofanorganisation

• SecurityactionsIhavedoneright

• Notificationofpersonalimpact,notorganisationalortechnicalimpact

• Quickwinsandlonger-termsolutions

• Realtimedashboard

• Returnoninvestment/cost-benefit

• Pleaseputsystemrisksinbusinesscontext,productivity,costetc

• Productivity,cost-benefitofcontrols,e.g.timetrainingvsvalue

• Developsecurecodingcapability

• Howourcompetitorsaredoing:notseeingmuch

• Whatagoodprocesslookslikeratherthananoutcome

• IfwehaveoutsourcedourIT,whatinformationshouldwecontractourprovidertoreport?Howcanwetrustthem?

• Understandingofeffectivenessofasecuritycontrol

• Quantifiableriskofdoingnothingdifferently

12

• Robuststatsaboutbehaviourratherthanthinking/intention/awareness

• Riskmanagersreport

• Option/negotiation:information/threatalertsuggest/requiresactions/response,butwhatarethealternatives?Thein-betweenoptionsandconsequences?

• Meaningfulconnection:whatdothesenumbers/percentagesmeanintermsofrequiredorrecommendedactions?

• IndicationofpossibleemployeeabusethatmightactuallyindicateanHRissue,e.g.stressorpoormanagement

• Analysiscouchedintermsofbusinesscontinuity

• Securityculture(seeCPNItools)

• Truecostofrecoveryvscostofcontrol

• Wantmore!80%say….[referencetoMadeline’spresentation]

• CostofmeasureimplementedthisFY

• Formulatoturnthreatintelligenceintoriskprofile

• Capabilitylevelofattackeryoucandefendagainst(STIX/TAXII)

• Howmuchproductivitylosttostupidsecuritypolicies?

• Subjectiveassessmentofriskofattack,threatvsmeasures

• Benchmarkingagainstasimilargroupoforganisations

• Capabilitytoachieverecoverytimeandpoint[sic]objectives

• Lessstats,moreinfo

• Changesofnetworktrafficfollowingnewsecuritypolicyimplementation

• Incidentstracedtorootcause

• Howmanyotherpeopleforwardphishingemailsforanalysis?

• Probabilityofabreachinmyindustryformyapplication

• Howservicesareinterrogatingmydata,e.g.howismyemailbeingread?

• Onwardtransmissionofmypersonaldata

• Largecompaniesaretakinganactiveleadershiproleinsupportingtheirindustrysector

• Howdoyoucapture/representtheintentoftheactioninametricform?

• Performanceoftrainedusersthreemonthsaftertraining?!

• Doyouhaveanincidentmanagementplanandhaveyouexercisedit?

13

“Stop!”(Currentlyreceivingthisbutdon’tfindituseful)

• Makingupnumbers

• Numericalstatsoverload

• Singlereports,i.e.notinrelationtopastorfuturemetrics

• Numberofhitsonmyfirewall

• Numberofemployeesclickingonfakephishingemails

• Privacystatementsandpasswordstrength

• RAG(RedAmberGreen)ratings:doesn’tmeanthesamethingtoeveryone

• “Expertopinion”

• Howmanyorgshaveasecuritypolicy?

• Numberofincidentsordetectioneventswithnobaserateofoccurrence

• Cherrypickingdata,i.e.biasedanalysis

• Bullyingwiththreatsof“badthings”

• Machinespatched

• Networkmonitoringstats

• Blamingusersbytellinguswhatthey’vedonewrong

• Userstrained

• Magicbulletsolutions

• Trafficlights

• Anythingthatyoucannotprovetomewillbestableenoughtoinvestinmeasuringovertime

• 3Dpiechartsorbubblecharts

• Uncontextualizednumbers

• Non-contextualstats

• Metricsfullofjargonwithoutexplanations

• Drowningmeindata

• “indexes”thathidecomplex,subjectivemethodologies

14

• SIEM

• AVstats

• Patchedpercentages:I’monlyinterestedineffects

• Nitty-grittydetailofsystempatching

• Tick-boxprocessconfirmation

• Outdateddata

• Anythingqualitative

• Historicevents,e.g.pastebin

• Detailsofindividuallower-significanceincidents/issues

• Progressagainstcompliancerequirement

• Irrelevantdatawheretheresultsarenotsignificant

• Phishingteststats:hugelyvariableifcomparedonetothenext,butusefultocomparebetweendeptstofindriskareas

• Poorpresentationofthegraphics,e.g.poorchoicesofcolour,inappropriatecharttypes

• Incidentnumberswithoutcontext,e.g.6,000incidentsacrosssectorXin2015

15

“Thanksfortheoffer,but….”(Don’treceiveitandwouldn’twanttoifitwereoffered)

• Metricsforafee

• Riskmetricswithoutsolutions

• Metricsthatpromptmorequestionsthananswers:don’tgivemeproblemswithoutsolutions,I’mbusyenoughalready!

• Datathathasbeensanitisedbymiddlemanagement

• Unsolicitedsalespitch:informationgathering,socialengineering

16

Part Two: Responses from providers of cyber security metrics

“Ican/doprovidethis”(Currentlydeliveringandplantocontinue)

• Totalcybersecurityspendasapercentageofrevenue/profit

• Existenceandrehearsalofincidentresponseplan

• BGProutingtables

• DNStracedataaboverecursiveresolver

• Likelihoodoffuturebreach

• Numberofphishingattacksinmyorganisation

• Timetoresumeservicedeliverypost-incident

• Impactofincidentonservicedelivery/BaUprocesses

• Timetoincident/compromisedetection

• Maxnumberofroguechangedaysonmynetwork

• Evidenceofnetworkcompromise

• #comment:categorisepreventionmetrics,detectionmetrics,responsemetrics,andrecoverymetrics

• Totalnumberofknownvulnerabilitiesonthenetwork

• Numberofdetectednetworkintrusions

• Numberofbreachesasaresultofuntargetedandunsophisticatedattacks

• Riskregisterentries,i.e.likelihoodofabreachofcustomerpersonaldata

• Motivationofthreat

• SoftwareinventoryviaSoftwareID(SWID)tags

• Casestudies/scenarios

• Allmetricsareproxiesandsubjecttocalibrationerrors

• Numberofemployeesattending/completingtraining,infosec,phishingetc(doesn’tshowhoweffectiveit’sbeen)

• Evidenceofcompromise

17

• Numberofstaffwithoutadequatesecuritytraining

• Metrics/dataforthesakeofit=comforting

• “value”ofextendedsupport

• Worstcasescenario,e.g.dayswebsitewouldbedown,systemsthatwouldneedtobere-built

• Uptakeof/barrierstopasswordbehaviour(maybe)

• Costofpastbreaches

• Totalcosttoinsure

• Quantifiedinformationonpersonaldata,i.e.whatisourexposure,emailandcarddetailsforonemillioncustomers

• Moneyspentrespondingtopreventableincidents,i.e.withmoreinvestmentinthefirstplace

• Supportstatusofmyestate

• Performanceofsecurebehaviours:reportingincidents,engagingwithawareness/engagementduration

• Cyberbreachessurvey

• Numberofpasswordresetrequests

• Boardengagementwithcybersecurity(FTSE350survey)

• Differentmetricsonthesamesystemfordifferentperspectives

• Endusercompliancewithphishingdetection/avoidancerules

• Numberofcyberincidentspreventedoraverted

• Insiderthreat

• AdamJoinson’sobesitymap

• Accountinglogs(AAA)

• Malwaredetectedandquarantined

• Cyberdefencematurityassessments(policy,e.g.NISTCybersecurityframework,CDCat,IAMaturityModel)

• Syslogs:firewalllogs,visualisationthroughgraphs

• Patchstatus

• Desktopbuild

• Phishesblocked

• PhishingemaillifejourneyinaSankeydiagram

• Capturebysecurityproductvendor

• Readteaming:table-topexerciseswithstakeholdersandsystemownerstoagreeimpactandvulnerabilityscoresforidentifiedattackvectors

• Aretheseconvenientbutnotuseful?

18

• Levelofengagementofemployeesingoodpracticesonsecuritybehaviour,bothpassiveandactive

• Statisticsonsecurityincidentfromsoftware,hardwareandhumananalysis(server&clientsides)

• Statisticsonhumansecuritybehaviourestimatedfromsoftware,hardware,surveys,observations,analysisofdataandreportedincidentsetc

“I’dliketobut…”(Wouldbehappytodeliverbuttherearefeasibilityissues)

• Analyselogsbeforeaproblemoccurs

• Completevulnerabilitymanagement

• Behaviours/securityculture

• Assurancelevels

• Numberofunknownvulnerabilitiesinthenetwork

• Costofanincident

• AmIcompliantwiththetermsofmycyberinsurance?

• Asinglemetricthatcanbecomparedacrossallorganisations

• Costoffuturebreaches

• Accuratecostofcybercrimetoacompany

• GDPRconstraints

• Micro-narratives

• Mismatched/ill-fittingdataprotectionrules

• Instrumentsnotavailable(tooexpensive)

• Feedbacklooponmetrics–stillrelevant?

• Howsecurearewe(withasinglepercentagevalue)?

• Qualitativedata:resourceautomate

• Informationthatinherentlyrepresentsthetask(conceptualratherthanthedata(computerscience))

• Knowingwheninformationisstillrelevantandnotoutofdatetobeofanyuse–createinformeddecision

19

“IcoulddobutI’drathernot”(It’sfeasibletodeliverbutmakesmeuncomfortable)

• Accuracyofvulnerabilityassessmenttools

• Estimatedcostsofsecurityincidentsandcrimeincyberspace,intermsofmonetaryvalue

• RAG(RedAmberGreen)ratings

• Boardengagementbysector

• Rankingsofcompaniesbycybersecuritycapacity

• Fixtimesbysupplier/contractor:incendiaryormisleadingbecauseofServiceLevelAgreements

• CyberEssentialsuptake

• Confidentialityconstraintstodisclosingdata

• Numbersofscansoneachportonawebserver

• Timetofixwebsitevulnerabilitiesinapublicleaguetableoforganisations

• Degree/reportofvulnerabilitiesfixed/recommendationsaddressedpost-penetrationtest

• Resultsoftestingemployeesecurity/awareness

• Daysexposedtodisclosedvulnerabilities

• Costofproviding/managingtechnicalcontrols

• Numberoftimes(threat)intelhasbeensharedviaCISPetc

• Employees’digitalfootprint/corporateinformationexposedviainternet/socialmedia

• Numberor%ofemployeespassingformaleducation/training/certificationetc

• Benchmarksagainstpeersinsameindustry/sector

• Improvement-relatedstatisticsonsecurityincidentsandbehavioursintermsofnumbers

• Thesecanbeusedtogameotherindicators

• Numberofhigherprivilegeaccesses

• Toohardtodisentanglefromvalueproposition

• Netflow/IPFixatorganisation/internetboundary

• DNSTracedatabelowrecursiveresolver

20

• Madeupdataormisleadingdata

• Penetrationtestresults

• Provenance

• Ensurecoverage/samplesize

• Isametricwhichofthese?Evidence,data,measurements,mathematicalsenseofdistancebetweentwoitems(Ithinkwemeanevidence)?

• Haverawdatabutdifficulttoaggregateorvisualise

• Numberofourbackdoors

21

“Weneedtotalk”(Thiscannotbedelivered.Evenifitcouldbe,Iwouldn’twantto)

• Anestimateofthenumberofcyberbreachesprevented

• Anythingthatshowsmeinabadlight

• AgreedmetricsfromUKGovernment(NCSC)forscoringimpactandvulnerability

• Anythingthatclaimstodemonstrateimpactonfinancialaccountingmetrics(ProfitandLoss,balancesheetsetc)

• Awidersharing/collaborativenetworkofexpertspreparedtoshareinformationandworktogether

• Contextualisedquantitativedata

• Numberofcompetitors’backdoors

22

@CityLeadersLab www.cityleadership.net

@RISCS_UK

www.riscs.org.uk www.ucl.ac.uk/steapp

Partners: