judgment and decision making in information systems decision making, sensitivity analysis, and the...

Judgment and Decision Making in Information Systems

Decision Making, Sensitivity Analysis, and the Value of

Information

Yuval Shahar, M.D., Ph.D.

Personal Decision Making:The Party Problem

• Joseph K. invites his friends to a party, but needs to decide on the location:– Outdoors (O) on the grass (completely open)

– On the Porch (P) (covered above, open on sides)

– Inside (I) the living room

• If the weather is sunny (S), outdoors is best, followed by Porch and Indoors; if it rains (R), the living room is best, followed by Porch and Outdoors

The Party Problem:An Ordered Preferences List

O-S (best)

P-S

I-R

I-S

P-R

O-R (worst)

Evaluating The Prospects

O-S (best)

P-S

I-R

I-S

P-R

O-R (worst)

1

0B

W0.95

0.05B

W0.67

0.33B

W0.57

0.43B

W0.32

0.68B

W0

1B

W

The Party Problem: Adding Probabilities

O

P

I

S 0.4

R 0.6

S 0.4

R 0.6

S 0.4

R 0.6

The Party Problem: Substituting Prospects

O

P

I

S 0.4

R 0.6

S 0.4

R 0.6

S 0.4

R 0.6

1

0B

W0

1B

W0.95

0.05B

W

0.32

0.68B

W0.57

0.43B

W

0.67

0.33B

W

The Party Problem: Simplifying The Relationship to Preference Probabilities

O

P

I

0.4

0.6

0.57

0.43

0.63

0.37

B

W

B

W

B

W

The Party Problem: Using the Choice Rule

O

P

I

0.4

0.6

0.57

0.43

0.63

0.37

B

W

B

W

B

W

<=

Expected Preference-Probability Values

O

P

I

S 0.4

R 0.6

S 0.4

R 0.6

S 0.4

R 0.6

Preference Probability

1

0

E-value =0.40

0.32

0.57

0.67

E-value= 0.57

E-value= 0.63

0.95

Computation With Decision Trees: Making the Decision

• Decision trees are “folded back” to the top most (leftmost, or initial) decision

• Computation is performed by averaging expected utility (in this case, preference probability) recursively over tree branches from right to left (bottom up), maximizing utility for every decision made and assuming that this is the expected utility for the subtree that follows the computed decision

The Value of Information (VI)• We often need to decide what would be the next best

piece of information to gather (e.g., within a diagnostic process); that is, what is the best next question to ask (e.g., what would be the result of a urine culture?);

• Alternatively, we might need to decide how much, if at all, is a paticular additional piece of information worth

• The Value of Information (VI) of feature f is the marginal expected utility of an optimal decision made knowing f, compared to making it without knowing f

• The net value of information (NVI) of f = VI(f)-cost(f)• NVI is highly useful when deciding what would be the

next information item, if any, to investigate

Computing the Value of Information: Requirements

• Decision Makers are often faced with the option of getting additional information for a certain cost

• To assess the value of additional information to the decision maker, we need a continuous, real valued utility measure, such as money

• Thus, we need to map preference probabilities to a monetary scale or an equivalent one (such as time)

Introducing Monetary Values: Bringing in The Wizard

• We ask the decision maker for her willingness to pay for a change from any state to any other state, assuming we have a wizard who can perform the change instantaneously

• We can thus create a 1:1 correspondence between preference probabilities and $ values

The Party Problem:Adding Monetary Values

Prospect Preference Probability $ Value

O-S (best) 1 100

P-S 0.95 90

I-R 0.67 50

I-S 0.57 40

P-R 0.32 20

O-R (worst) 0 0

Joseph K.’s Utility Curve

$0 $34 $50K $100K

1

0.5

0

Uti

lity

+

Money

Note: Once we know JK’s utilility curve, we can compute his certain equivalent for ANY deal, e.g., a (<$100K, 50%; $0, 50%> deal, which happens to be $34, using graphical or other methods; or the $ certain equivalent of any outcome (e.g., Outdoor location)

)2

11(

3

4)U(

50

x

x

The Value of Clairvoyance

• To compute the value of information, we assume a clairvoyant who knows with certainty all outcomes and always tells the truth

• However, clairvoyance has a price!

• Thus, the question is, How Much should we pay the clairvoyant for her forecast (e.g., for telling Joseph K. if it will rain or not)

Computing the Value of Clairvoyance

• We build a decision tree comparing the optimal decision without clairvoyance to the optimal decision given that the clairvoyant prophesies any of the possible outcomes (in this case, with 100% accuracy)

• We need to deduct the cost of clairvoyance from the $ value of all outcomes and role the decision tree back as we did before, to determine the expected value of the decision given clairvoyance (and paying for it!)

• We compute e-values using u-values (which represent the decision maker’s utility for each $ value)

• At the end (root node) we convert u-values to $ values• We then know whether the clairvoyance is worth its cost

Computing The $ Value of Clairvoyance

O

P

I

$value U value

O

P

I

O

P

I

“S”

“R”

No clairvoyance

U=0.63 ( $46)

Buy clairvoyance for $15

U =0.67 ( $51)

0.4

0.6

$value $value-$cost100 850 -

1590 7591 540 2550 35100 850 -1590 7591 540 2550 35

100 1 0 090 0.9591 0.3240 0.5750 0.67

1 S

0 R

0.4 S

0.6 R0.4 S

0.6 R

0.6 R

0.4 S

1 S

0 R

0 R

1 S

0 S

1 R

1 R

0 S

0 S

$85

U= 0.40 $26

U= 0.57 $40

U= 0.63 $46

U= 0.63 $46

$75

$25

$-15

$ 5

$35

U= 0.92 $85

U= 0.50 $35

The Value of Partial Clairvoyance

• Assume we have a rain detector which, when the weather is going to be sunny or rainy, predicts Rain or Sun 80% of the time, correspondingly; is it worth $10 to use it?

• In order to compute the value of an uncertain piece of information, we first need to calculate the probability for rain or sun, given each corresponding prediction of the detector

• This involves reversing the tree in which the detector information is usually given, using Bayes theorem

Computing The Value of Partial Clairvoyance (I):

Representing the Detector’s Profile

0.8“ S”

0.2“ R”

0.2“ S”

0.8“ R”

0.4 S

0.6 R

0.32

0.08

0.12

0.48

Computing The Value of Partial Clairvoyance (II):

Calculating Accuracy by Reversing the Tree

0.727 S

0.273 R

0.143 S

0.857 R

0.44 “S”

0.56“ R”

0.32

0.08

0.12

0.48

Computing The Value of Partial Clairvoyance (III):

Calculating the Optimal Decision With and Without Clairvoyance

• To actually calculate the value of uncertain information, we compare the expected utility of the best decision without the information to the expected utility of the decision with the information, as we did before

• We use the distribution of real world states given the semi-clairvoyant’s prediction (using our accuracy calculation)

• In this particular case (80% accuracy), the value without detector is U = 0.627, or $45.83; the value with partial information is U = 0.615, or $44.65

• Thus, for Joseph K., this particular detector is not worth the money

Sensitivity Analysis

• The main insights into a decision are often given by an analysis of the influence of the given probabilities and utilities on the final decision and its value

• We thus get a sense as to how sensitive the expected utility and optimal decisions are to each parameter, and can focus our attention (and perhaps, further elicitation efforts) on the most important aspects of the problem

The Party Problem: Sensitivity Analysis

O

P

I

S p

R 1-p

S p

R 1-p

S p

R 1-p

1

0

U = p

0.323

0.568

0.667

0.950

U = 0.323+0.627p

U = 0.667-0.099p

$value U value

100

0

90

20

40

50

Sensitivity Analysis of the Party Problem

U $

P

0.2

0.4

0.6

0.8

1100

40

O:p

90

P: 0.323+0.628p

I: 0.667+0.0.099p

Free clairvoyance:

0.667+0.333p

0.2 0.4 0.6 0.80.47

20

The value of information

Tornado Diagrams• We can calculate the lower and upper bounds on the expected

utility E(U) of the optimal decision for the full range of changes in each parameter Xi while leaving all other parameters at their nominal (default) value

• We get a quick sense of the potential influence of each parameter on the expected utility

• But:– It ignores the change distribution

– It ignores the dependencies between parameters

X4X12X3X5X7X8

- 0 +