sim presentation 092402
TRANSCRIPT
Copyright, Robust Decisions, 2002
www.robustdecisions.com
1
Increasing Business Value with Decision Management
David G. UllmanPresident, Robust Decisions Inc.
Jeffrey Barnes President, Compelevent Solutions
Pr
Copyright, Robust Decisions, 2002
www.robustdecisions.com
2
Goals
• Define Decision Management
• Describe major Decision Management objectives• Managing uncertainty
• Basing decisions on satisfaction and risk
• Deciding what to do next
• Demonstrate achieving the objects using an RFP example
• Discuss how Decision Management can help you produce more successful decisions
Copyright, Robust Decisions, 2002
www.robustdecisions.com
3
The value of information
DECISIONDECISIONDECISIONDECISION
JudgmentJudgmentJudgmentJudgment
KNOWLEDGEKNOWLEDGEKNOWLEDGEKNOWLEDGE
BehaviorBehaviorBehaviorBehavior
MODELSMODELSMODELSMODELS
RelationshipsRelationshipsRelationshipsRelationships
DATADATADATADATA
Increasing value
Copyright, Robust Decisions, 2002
www.robustdecisions.com
4
What makes critical decisions so difficult?
• There are many stakeholders: customers, influencers, decision makers and approvers
• Their knowledge is uncertain: qualitative, evolving and incomplete
• Their perspectives about what is important varies
• They must base decisions on expected satisfaction and assessment of risk
• Not clear what to do next "The only real problem in life is what to do next."
Arthur C. Clark
Copyright, Robust Decisions, 2002
www.robustdecisions.com
5
A process that ensuresstakeholder perspectives andtheir uncertain knowledge
are effectively utilized in determining what to do next to make thebest possible choices with
known expected satisfaction and risk
Given situation
Goal
What is Decision Management?
Copyright, Robust Decisions, 2002
www.robustdecisions.com
6
• Decisions that were declared final are later revisited • Team members don’t buy into decisions• Team decisions are made by edict or by loudest voice• Not all the expertise of the team members is utilized. • There is little confidence in decisions based on
uncertain information and unknown risk• Decisions are not justified, documented, or reused
Symptoms of Poor Decision Management
Copyright, Robust Decisions, 2002
www.robustdecisions.com
7
Cost of poor decisions
Product development:
• A poor decision - 20 people spend one week reworking a critical problem: $75,000 wasted on burdened salaries alone
Product procurement:
• A poor decision - Wastes product acquisition/implementation costs and creates excessive support costs. Expected added business value delayed, diminished or not realized entirely.
Copyright, Robust Decisions, 2002
www.robustdecisions.com
8
Typical decision problems
• IT investment portfolio
• Outsourcing decisions
• Vendor selection
• Hiring decisions
• Product development
• Partner selection
• Business strategy
• Washing machine selection
Copyright, Robust Decisions, 2002
www.robustdecisions.com
9
Simple example: RFP evaluation
• RFP contains• Detailed list of requirements that define the
ideal system - developed by functional representatives (stakeholders) in the organization
• The responses• 4 proposals submitted
• Your organization has dealt with some of the vendors before
Copyright, Robust Decisions, 2002
www.robustdecisions.com
10
Evaluation criteria given in the RFP
• Cost Criteria, < $100K 30%
• Technical Criteria 50%• System response time, <3 sec
• Training time, > 95% proficiency in 72 hrs
• Ease of use, > 75% satisfaction
• Risk of not meeting criteria, low
• ………..
• Management Criteria 20%• Strong management team
• Experienced with similar problems
• ……….. Total 100%
Whose weightings?
Is risk a useful criterion?
How do you manage qualitative criteria?(they are often the
discriminators)
What if one vendor proposes 90%?
Copyright, Robust Decisions, 2002
www.robustdecisions.com
11
Basic evaluation methods
• Fiat
• Loudest voice
• Competition
• Voting
• Inertia
• Omphaloskepsis
Copyright, Robust Decisions, 2002
www.robustdecisions.com
12
A more sophisticated evaluation method(The Decision Matrix)
Cost
Performance Management
Response time
Training time
Ease of use
Strong mgt team
Proj. Team experience
Total sum/wt
wt .3 .5 .2 1.0
Vendor 1 4 3 2 1 3 3 16/2.8
Vendor 2 4 4 4 3 5 4 24/3.9
Vendor 3 4 3 4 4 4 4 23/3.8
Vendor 4 4 5 5 4 2 2 22/3.9
“The Mechanical Design Process” 3rd
edition, David G. Ullman, McGraw Hill, 2003
Which will you choose?
Measures of satisfaction
Confidence vendor meets criterion5 = very high1 = very low
Copyright, Robust Decisions, 2002
www.robustdecisions.com
13
Decision Management improving the decision matrix
• Minimal additional work
• Inclusion of stakeholders perspectives
• Management of uncertain knowledge and differing perspectives about what is important
• Real time analysis of satisfaction and risk
• Guidance about what to do next to find the vendor most likely to be the best choice
• Capture and document the deliberation for justification and reuse
First topic
Copyright, Robust Decisions, 2002
www.robustdecisions.com
14
Uncertain evaluation
CostResponse
timeTraining
timeEase of
useStrong
mgt teamProj. Team experience
Evaluator 1 4 3 2 4 4 3
Evaluator 2 4 4 3 2 3 3
Evaluator 3 4 3 4 2 1 3
Evaluator 4 4 4 5 1 4 2
How do you manage the differences?Confidence that
vendor meets criterion5 = very high1 = very low
For Vendor 1
Copyright, Robust Decisions, 2002
www.robustdecisions.com
15
How do you manage the differences?
Average?
Discussion?
Voting?
– Averages are always brown.
– Discussions can be dominated by loudest voices.
– How can you account for knowledge differences?
– Move to Florida!
What about uncertainty?
Copyright, Robust Decisions, 2002
www.robustdecisions.com
16
The effect of knowledge in qualitative evaluations
Strong Mgt Team
Evaluator 1 4
Evaluator 2 3
Evaluator 3 1
Evaluator 4 4
Team score including knowledge = 4.1
For Vendor 1 Knowledge scaleExpertExperiencedInformedAmateurWeakUnknowledgeable
Average = 3
Knowledge
Experienced
Informed
Weak
Experienced
Confidence that vendor meets criterion5 = very high1 = very low
Copyright, Robust Decisions, 2002
www.robustdecisions.com
17
The effects of uncertainty during quantitative evaluations
• RFP - Training time > 95% proficiency in 72 hours
• In reality you will be delighted with 95%, but not sure it is achievable. Disgusted if < 85%
• Vendor bids - 95%
• Vendor’s reality – sure she can get 91% + 5%
• Your estimate of vendor’s reality is 87% + 8%
• If you could run an evaluation experiment you might find 90% + 8%
• After the system is in use the reality is 94%
Target uncertainty
Evaluation uncertainty
Changes with time
Copyright, Robust Decisions, 2002
www.robustdecisions.com
18
Next topic
• Minimal additional work
• Inclusion of stakeholders perspectives
• Management of uncertain knowledge and differing perspectives about what is important
• Real time analysis of satisfaction and risk
• Guidance about what to do next to find the vendor most likely to be the best choice
• Capture and document the deliberation for justification and reuse
Decision Management improving the decision matrix
Copyright, Robust Decisions, 2002
www.robustdecisions.com
19
• Risk is not a criterion!!
• Risk is a measure of uncertainty.
• Risk is one measure of how well each alternative meets each criterion.
Risk Analysis
• There is risk that the system response time will be >3 sec
• There is risk that the training time will be < 95% proficiency in 72 hrs
• There is risk that the ease of use will be < 75% satisfaction
• There is risk that the management team is not strong
• There is risk that they are not experienced with similar problems
Copyright, Robust Decisions, 2002
www.robustdecisions.com
20
There are two kinds of risk• Satisfaction Risk – The possibility that an
alternative will not meet a criterion ideal target
• Risk in choosing an alternative that can achieve only 93% proficiency
• Not really a risk measure, but lack of satisfaction
• Downside risk – The possibility that the actual satisfaction will be worse than expected
• We think that it will be 93% but there is a
chance it will only be 90%
• Also upside risk
Copyright, Robust Decisions, 2002
www.robustdecisions.com
21
Weighted Satisfaction
Vendor 1 16/2.8
Vendor 2 24/3.9
Vendor 3 23/3.8
Vendor 4 22/3.9
V4V2
3.9
V3 3.8
It is as important to understand the uncertainty as it is to believe the satisfaction!
Which one will you choose?
Downside risk
Ideal satisfaction - 4.0?
Satisfaction risk
Variation captured by the evaluations of different
stakeholders
Copyright, Robust Decisions, 2002
www.robustdecisions.com
22
Final topic
• Minimal additional work
• Inclusion of stakeholders perspectives
• Management of uncertain knowledge and differing perspectives about what is important
• Real time analysis of satisfaction and risk
• Guidance about what to do next to find the vendormost likely to be the best choice
• Capture and document the deliberation for justification and reuse
Decision Management improving the decision matrix
Copyright, Robust Decisions, 2002
www.robustdecisions.com
23
Traditionally
• Choose vendor with highest score (satisfaction)
• Loose sleep
• Re-bid
"The only real problem in life is what to do next." Arthur C. Clark
Vendor 1 16/2.8
Vendor 2 24/3.9
Vendor 3 23/3.8
Vendor 4 22/3.9
If you choose not to decide, you still have made a choice. Neil Peart,
drummer and lyricist for Rush.
Copyright, Robust Decisions, 2002
www.robustdecisions.com
24
• Satisfaction
• Value of information analysis (a sensitivity analysis)
• Level of consensus
• Level of knowledge or uncertainty
Decision Management basis for “what to do next” analysis
Traditionally all that is available
The needed richness
supplied by decision
management tools
Copyright, Robust Decisions, 2002
www.robustdecisions.com
25
1. Choose an alternative (The team has reached consensus, from all perspectives on a specific alternative)
2. Work toward consensus (Increase team consensus on identified critical evaluations)
3. Increase knowledge (Decrease uncertainty on identified critical evaluations)
4. Refine criteria (Identified critical criteria need to be refined)
Decision Management suggests
Copyright, Robust Decisions, 2002
www.robustdecisions.com
26
A “What to do next report”
Build consensus on Strong management team for Vendor 1
Build consensus on System response time for Vendor 2
Increase team knowledge about Ease of use for Vendor 1
Increase team knowledge about Strong management teamfor Vendor 2
Refine team experience criterion
Copyright, Robust Decisions, 2002
www.robustdecisions.com
27
The deliberation
• Build consensus Strong management team for Vendor 1
• Build consensus System response time for Vendor 2
• Increase knowledgeabout Ease of use for Vendor 3
Based on results team recommends Vendor 3 to decision maker
New what to next report
Initial what to next report
Copyright, Robust Decisions, 2002
www.robustdecisions.com
28
A process that ensuresstakeholder perspectives andtheir uncertain knowledge
are effectively utilized in determining what to do next to make thebest possible choices with
known expected satisfaction and risk
What is Decision Management?
Copyright, Robust Decisions, 2002
www.robustdecisions.com
29
On the next project
You will• Have more successful decisions
• Get more buy-in from stakeholders
• Keep out of trouble
• Document how decisions were reached for justification and reuse
• Deliver on promises made in the business case for the RFP
• Manage uncertainties
• Integrate stakeholder knowledge and perspectives
• Calculate satisfaction and risk
• Analyze what to do next
Copyright, Robust Decisions, 2002
www.robustdecisions.com
30
Tools that support decision
management
• @RISK, Palisades, http://www.palisade.com
• Expert Choice, http://www.expertchoice.com
• Accord™ www.robustdecisions.com
Copyright, Robust Decisions, 2002
www.robustdecisions.com
31
You now get to make the decision…
“What to do next”
Copyright, Robust Decisions, 2002
www.robustdecisions.com
32
For RFP problem.
Copyright, Robust Decisions, 2002
www.robustdecisions.com
33
Robust Decisions Inc.Delivering decision management consulting, training and tools.
Developers of Accord™ the real time, easy to use decision management application for distributed or co-located teams