an experiment in security decision making

© Copyright 2010 Hewlett-Packard Development Company, L.P. 1© Copyright 2010 Hewlett-Packard Development Company, L.P.

Adrian Baldwin, Yolanta Beres, Marco Casassa Mont, Simon Shiu (all HP Labs) Geoff Duggan, Hilary Johnson (University of Bath)Chris Middup (Open University)

AN EXPERIMENT IN SECURITY DECISION MAKING

© Copyright 2010 Hewlett-Packard Development Company, L.P. 2

CONTEXT– TSB funded trust economics project:• We developed an approach (using economic and mathematical modelling) to help enterprises make “better” security decisions• A series of case studies providing good feedback and anecdotal evidence that were on a good path

– Challenge – can we do better than that?– This paper: • An in depth study of a small group of security professionals (one stakeholder type), on how our approach to security decision making affects them


A RIGOROUS APPROACH TO SECURITY DECISION MAKING

System Model

Problem Architecture

consequences of preferences

problem refinement

things to measure

components of utility

Problem

Preferences

Utility


SDM HYPOTHESESOur methods will positively influence:– the conclusions or decisions made,– the thought process followed,– the justifications given, and– the confidence the stakeholder has in the final

conclusions or decisions made.


SDM EXPERIMENT SCOPE– Measure effect on security professionals/experts (i.e. not

our effect on other stakeholders nor groups/organisations)

– Qualitative in depth study of decision making process (of twelve professionals)

– Bundled economic framing and system modelling as a “single” intervention

– Controlled experiment, i.e. two groups one intervened using our methods, one left as a control


THE SDM PROBLEM– Chose a problem on the security of client infrastructure– Why – we had several similar case studies that meant we knew:• it was a representative current and challenging business security problem• we had decent/realistic empirical data relating to the problem• there are interesting “trade-offs” that meant the answer is subjective and contextual and likely to be different for different stakeholders

– We had 4 decision options that represented different trade-offs– We had to iterate a number of times before we had sufficient

supporting material and a problem we could control, and that was rich enough!


EXPERIMENT DESIGN

5a. Preference/ Economic Framing

5b. Modelling & Results

2. Problem Description

4. Decision Options

6. Choice & Justification

7. Introspection

1. Session Introduction

3. Question & Answers

5. Question & Answers


PHASES1. Session introduction2. Problem description3. Q&A4. Decision options5. (a) Preference

Elicitation(b) Model analysis

6. Choice & Justification7. Introspection

EXPERIMENT PHASES

– Options• Invest in patching• Invest in Host based intrusion prevention (HIPS) technology• Change policy to lock down (remove admin privileges) from users• Do nothing





EXPERIMENT PHASES (INTERVENE PHASE ONLY)

– Identify major outcomes (components of utility)

– Identify appropriate proxy metrics for each outcome

– Prioritise outcomes





EXPERIMENT PHASES

– Describe model of concurrent processes, and how options are explored

– Show (chosen proxy measure) results in 3*3 results tool


DATA ANALYSIS– 173 questions before intervention (from all twelve

participants)– 152 justifications (from all twelve participants)– 6 ordered prioritised outcomes– 12 decision options– 48 Likert scores on confidence (four from each

participant)


THE CHOICES– In the control group: 3 selected Lockdown, 2 selected

HIPS and 1 selected Patching– In the intervention group: 3 selected Lockdown and 3

selected HIPS

– A very security oriented set of options!


CATEGORIZATION OF QUESTIONSSimilar balance between groups

Cost

Compli

ance

Produc

tivity

Evide

nce

Securi

tyOthe

r0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

InterventionControl


CATEGORIZATION OF JUSTIFICATIONSMore balanced business justification for the intervened group

Cost Productivity Security Other0

0.1

0.2

0.3

0.4

0.5

0.6

InterventionControl


SDM HYPOTHESESOur methods will positively influence:•the conclusions or decisions made,•the thought process followed,•the justifications given, and•the confidence the stakeholder has in the final conclusions or decisions made.

SDM RESULTS

WHAT DO THE DATA RESULTS SAY IN RELATION TO OUR ORIGINAL HYPOTHESIS

– Not sufficient evidence that we influenced conclusions or decisions made

– There is evidence we influenced the justifications given• Which in turn suggests we affected their thought processes

– There was a slight (but not significant) increase in confidence in decisions made


SOME FURTHER ANALYSISpotential theoretical explanations

NB on study style: smaller qualitative studies often fertile for early theoretical development

– Security priority in questions (and control group’s justifications) suggest presence of confirmation bias

– The intervened group’s broader justifications suggest our methods managed to counter some of this bias

– The intervened group did not value the economic framing • “i’d made those trade offs already”is at odds with this result - suggests cognitive dissonance


CONCLUSIONS & NEXT STEPS– Encouragement that economic framing improves analysis

• Assume that a study of group decision support would make this results stronger

– Encouragement to use tools to support simultaneous comparison of multiple outcomes and choices

– More cognitive science should be done to complement security economics

– Future analysis• Study ‘question’ data to see methods/structure followed by security profession (compared with ISO27k, hunting for low hanging fruit, ...)

– Future studies• To test the suggested theories• To explore the effect on multi-stakeholder decision making

© Copyright 2010 Hewlett-Packard Development Company, L.P. 1818 © Copyright 2010 Hewlett-Packard Development Company, L.P.

QUESTIONS





EXPERIMENT PHASES (INTERVENE PHASE ONLY)

– Identify major outcomes (components of utility)

– Identify appropriate proxy metrics for each outcome






EXPERIMENT PHASES






EXPERIMENT PHASES

– Describe model of concurrent processes, empirical studies, and how options are explored





EXPERIMENT PHASES– Show results in 3*3 (option to

proxy measure) results tool





EXPERIMENT PHASES– 10 minutes to ask any questions they deem

relevant– Scripted answers (e.g. on history, culture,

processes, architecture, business, regulations etc…)

– Answers to “new” questions were added to the script for future sessions

– After 10 minutes we provided “essential” information that had not been asked about

– This allowed us to collect data on what questions were asked and in what order





EXPERIMENT PHASES

– Choose preferred option– For each option:• Pro’s – reasons why option would be good• Con’s – reasons why option would be bad• Likert scale 1-7 confidence in the option





EXPERIMENT PHASES

– For intervened group• What difference the interventions and tools made

– What information they used to reach their conclusion

– Any strategies they used when asking questions





EXPERIMENT PHASES

– 3 Roles: interviewer, expert and observer

– Interviewer explained and gathered:• Structure of session• Incentives for trying hard• Experience of participant





EXPERIMENT PHASES

– Verbally scripted, web based and written material introducing them to the security role they are being asked to play and the client infrastructure security problem the CISO has.

– Whether/how to deal with rising risk from malware on client infrastructure


DATA ANALYSIS– All questions and justifications were transcribed and put

in ‘random’ order– 3 experts categorised these – differences resolved

through discussion• Relation to ISO 27000• Relation to main business outcomes (compliance, productivity, cost, security risk)

an experiment in security decision making

Documents

control copyright

security professionalsexperts

question answers copyright

decision options6

depth study of decision

experiment design5a

preference economic

rigorous approach