© copyright 2010 hewlett-packard development company, l.p. 1 adrian baldwin, yolanta beres, marco...

© Copyright 2010 Hewlett-Packard Development Company, L.P.    1© Copyright 2010 Hewlett-Packard Development Company, L.P.   

Adrian Baldwin, Yolanta Beres, Marco Casassa Mont, Simon Shiu (all HP Labs) Geoff Duggan, Hilary Johnson (University of Bath)Chris Middup (Open University)

AN EXPERIMENT IN SECURITY DECISION MAKING

© Copyright 2010 Hewlett-Packard Development Company, L.P.    2

CONTEXT

– TSB funded trust economics project:• We developed an approach (using economic and mathematical modelling) to help enterprises make “better” security decisions

• A series of case studies providing good feedback and anecdotal evidence that were on a good path

– Challenge – can we do better than that?

– This paper: • An in depth study of a small group of security professionals (one stakeholder type), on how our approach to security decision making affects them

© Copyright 2010 Hewlett-Packard Development Company, L.P.    3

A RIGOROUS APPROACH TO SECURITY DECISION MAKING

System Model

Problem Architecture

consequences of preferences

problem refinement

things to measure

components of utility

Problem

Preferences

Utility

© Copyright 2010 Hewlett-Packard Development Company, L.P.    4

SDM HYPOTHESES

Our methods will positively influence:

– the conclusions or decisions made,

– the thought process followed,

– the justifications given, and

– the confidence the stakeholder has in the final conclusions or decisions made.

© Copyright 2010 Hewlett-Packard Development Company, L.P.    5

SDM EXPERIMENT SCOPE

– Measure effect on security professionals/experts (i.e. not our effect on other stakeholders nor groups/organisations)

– Qualitative in depth study of decision making process (of twelve professionals)

– Bundled economic framing and system modelling as a “single” intervention

– Controlled experiment, i.e. two groups one intervened using our methods, one left as a control

© Copyright 2010 Hewlett-Packard Development Company, L.P.    6

THE SDM PROBLEM– Chose a problem on the security of client infrastructure

– Why – we had several similar case studies that meant we knew:• it was a representative current and challenging business security problem

• we had decent/realistic empirical data relating to the problem

• there are interesting “trade-offs” that meant the answer is subjective and contextual and likely to be different for different stakeholders

– We had 4 decision options that represented different trade-offs

– We had to iterate a number of times before we had sufficient supporting material and a problem we could control, and that was rich enough!

© Copyright 2010 Hewlett-Packard Development Company, L.P.    7

EXPERIMENT DESIGN

5a. Preference/ Economic Framing

5b. Modelling & Results

2. Problem Description

4. Decision Options

6. Choice & Justification

7. Introspection

1. Session Introduction

3. Question & Answers

5. Question & Answers

© Copyright 2010 Hewlett-Packard Development Company, L.P.    8

PHASES

1. Session introduction

2. Problem description

3. Q&A

4. Decision options

5. (a) Preference Elicitation(b) Model analysis

6. Choice & Justification

7. Introspection

EXPERIMENT PHASES

– Options• Invest in patching

• Invest in Host based intrusion prevention (HIPS) technology

• Change policy to lock down (remove admin privileges) from users

• Do nothing

© Copyright 2010 Hewlett-Packard Development Company, L.P.    9

PHASES

1. Session introduction

2. Problem description

3. Q&A

4. Decision options

5. (a) Preference Elicitation(b) Model analysis

6. Choice & Justification

7. Introspection

EXPERIMENT PHASES (INTERVENE PHASE ONLY)

– Identify major outcomes (components of utility)

– Identify appropriate proxy metrics for each outcome

– Prioritise outcomes

© Copyright 2010 Hewlett-Packard Development Company, L.P.    10

PHASES

1. Session introduction

2. Problem description

3. Q&A

4. Decision options

5. (a) Preference Elicitation(b) Model analysis

6. Choice & Justification

7. Introspection

EXPERIMENT PHASES

– Describe model of concurrent processes, and how options are explored

– Show (chosen proxy measure) results in 3*3 results tool

© Copyright 2010 Hewlett-Packard Development Company, L.P.    11

DATA ANALYSIS

– 173 questions before intervention (from all twelve participants)

– 152 justifications (from all twelve participants)

– 6 ordered prioritised outcomes

– 12 decision options

– 48 Likert scores on confidence (four from each participant)

© Copyright 2010 Hewlett-Packard Development Company, L.P.    12

THE CHOICES

– In the control group: 3 selected Lockdown, 2 selected HIPS and 1 selected Patching

– In the intervention group: 3 selected Lockdown and 3 selected HIPS

– A very security oriented set of options!

© Copyright 2010 Hewlett-Packard Development Company, L.P.    13

CATEGORIZATION OF QUESTIONSSimilar balance between groups

Cost

Compl

ianc

e

Prod

uctiv

ity

Evid

ence

Secu

rity

Other

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

InterventionControl

© Copyright 2010 Hewlett-Packard Development Company, L.P.    14

CATEGORIZATION OF JUSTIFICATIONSMore balanced business justification for the intervened group

Cost Productivity Security Other0

0.1

0.2

0.3

0.4

0.5

0.6

InterventionControl

© Copyright 2010 Hewlett-Packard Development Company, L.P.    15

SDM HYPOTHESES

Our methods will positively influence:

•the conclusions or decisions made,

•the thought process followed,

•the justifications given, and

•the confidence the stakeholder has in the final conclusions or decisions made.

SDM RESULTS

WHAT DO THE DATA RESULTS SAY IN RELATION TO OUR ORIGINAL HYPOTHESIS

– Not sufficient evidence that we influenced conclusions or decisions made

– There is evidence we influenced the justifications given• Which in turn suggests we affected their thought processes

– There was a slight (but not significant) increase in confidence in decisions made

© Copyright 2010 Hewlett-Packard Development Company, L.P.    16

SOME FURTHER ANALYSISpotential theoretical explanations

NB on study style: smaller qualitative studies often fertile for early theoretical development

– Security priority in questions (and control group’s justifications) suggest presence of confirmation bias

– The intervened group’s broader justifications suggest our methods managed to counter some of this bias

– The intervened group did not value the economic framing • “i’d made those trade offs already”

is at odds with this result - suggests cognitive dissonance

© Copyright 2010 Hewlett-Packard Development Company, L.P.    17

CONCLUSIONS & NEXT STEPS– Encouragement that economic framing improves analysis

• Assume that a study of group decision support would make this results stronger

– Encouragement to use tools to support simultaneous comparison of multiple outcomes and choices

– More cognitive science should be done to complement security economics

– Future analysis• Study ‘question’ data to see methods/structure followed by security profession (compared with ISO27k, hunting for low hanging fruit, ...)

– Future studies• To test the suggested theories

• To explore the effect on multi-stakeholder decision making

© Copyright 2010 Hewlett-Packard Development Company, L.P.    1818 © Copyright 2010 Hewlett-Packard Development Company, L.P.   

QUESTIONS

© Copyright 2010 Hewlett-Packard Development Company, L.P.    19

PHASES

1. Session introduction

2. Problem description

3. Q&A

4. Decision options

5. (a) Preference Elicitation(b) Model analysis

6. Choice & Justification

7. Introspection

EXPERIMENT PHASES (INTERVENE PHASE ONLY)

– Identify major outcomes (components of utility)

– Identify appropriate proxy metrics for each outcome

– Prioritise outcomes

© Copyright 2010 Hewlett-Packard Development Company, L.P.    20

PHASES

1. Session introduction

2. Problem description

3. Q&A

4. Decision options

5. (a) Preference Elicitation(b) Model analysis

6. Choice & Justification

7. Introspection

EXPERIMENT PHASES

– Prioritise outcomes

© Copyright 2010 Hewlett-Packard Development Company, L.P.    21

PHASES

1. Session introduction

2. Problem description

3. Q&A

4. Decision options

5. (a) Preference Elicitation(b) Model analysis

6. Choice & Justification

7. Introspection

EXPERIMENT PHASES

– Describe model of concurrent processes, empirical studies, and how options are explored

© Copyright 2010 Hewlett-Packard Development Company, L.P.    22

PHASES

1. Session introduction

2. Problem description

3. Q&A

4. Decision options

5. (a) Preference Elicitation(b) Model analysis

6. Choice & Justification

7. Introspection

EXPERIMENT PHASES– Show results in 3*3 (option to

proxy measure) results tool

© Copyright 2010 Hewlett-Packard Development Company, L.P.    23

PHASES

1. Session introduction

2. Problem description

3. Q&A

4. Decision options

5. (a) Preference Elicitation(b) Model analysis

6. Choice & Justification

7. Introspection

EXPERIMENT PHASES

– 10 minutes to ask any questions they deem relevant

– Scripted answers (e.g. on history, culture, processes, architecture, business, regulations etc…)

– Answers to “new” questions were added to the script for future sessions

– After 10 minutes we provided “essential” information that had not been asked about

– This allowed us to collect data on what questions were asked and in what order

© Copyright 2010 Hewlett-Packard Development Company, L.P.    24

PHASES

1. Session introduction

2. Problem description

3. Q&A

4. Decision options

5. (a) Preference Elicitation(b) Model analysis

6. Choice & Justification

7. Introspection

EXPERIMENT PHASES

– Choose preferred option

– For each option:• Pro’s – reasons why option would be good

• Con’s – reasons why option would be bad

• Likert scale 1-7 confidence in the option

© Copyright 2010 Hewlett-Packard Development Company, L.P.    25

PHASES

1. Session introduction

2. Problem description

3. Q&A

4. Decision options

5. (a) Preference Elicitation(b) Model analysis

6. Choice & Justification

7. Introspection

EXPERIMENT PHASES

– For intervened group• What difference the interventions and tools made

– What information they used to reach their conclusion

– Any strategies they used when asking questions

© Copyright 2010 Hewlett-Packard Development Company, L.P.    26

PHASES

1. Session introduction

2. Problem description

3. Q&A

4. Decision options

5. (a) Preference Elicitation(b) Model analysis

6. Choice & Justification

7. Introspection

EXPERIMENT PHASES

– 3 Roles: interviewer, expert and observer

– Interviewer explained and gathered:• Structure of session

• Incentives for trying hard

• Experience of participant

© Copyright 2010 Hewlett-Packard Development Company, L.P.    27

PHASES

1. Session introduction

2. Problem description

3. Q&A

4. Decision options

5. (a) Preference Elicitation(b) Model analysis

6. Choice & Justification

7. Introspection

EXPERIMENT PHASES

– Verbally scripted, web based and written material introducing them to the security role they are being asked to play and the client infrastructure security problem the CISO has.

– Whether/how to deal with rising risk from malware on client infrastructure

© Copyright 2010 Hewlett-Packard Development Company, L.P.    28

DATA ANALYSIS

– All questions and justifications were transcribed and put in ‘random’ order

– 3 experts categorised these – differences resolved through discussion• Relation to ISO 27000

• Relation to main business outcomes (compliance, productivity, cost, security risk)