"let's leave that up to the computer" chris snijders
Post on 14-Jan-2016
31 Views
Preview:
DESCRIPTION
TRANSCRIPT
"Let's leave that up to the computer"
Chris Snijders
Prof. dr. C.C.P. SnijdersDept of Technology Management
Technische Universiteit EindhovenPostbus 513
5600 MB Eindhovenc.c.p.snijders@tue.nl
Chris Snijders
Overview
• Case: Cook county hospital• The science behind it• The computer as a decision
maker
Chris Snijders
Case: Cook county hospital
Emergency Department
-250.000 patients per year
-many persons without insurance
-not enough rooms, overworked staff
-1996: Brendan Reilly director
(see Gladwell, 2005)
Chris Snijders
Problem 1: acute chest pain
Diagnose through:
blood pressure, stethoscope: fluid in the lungs, how long have you been experiencing pain, how does it feel precisely, where does it hurt, does it always hurt or only when you exercise, have you had heart problems before, how about your cholesterol, do you have diabetes, let's look at your ECG, are there any heart problems in the family, do you use drugs, how old are you, are you in shape, do you smoke, do you drink, check appearance: stressed, overweight, ....
High risk : 8
Medium risk : 12
Go home30 p/day
Chris Snijders
Reilly finds Goldman: obv 10,000 cases
Only 4 things matter
ECGBlood pressureFluid in your lungs"unstable angina"
Chris Snijders
Great! So let's do that! Or not...
Implementation: … physicians protest …
A test: 20 cases were given to several physicians
Hardly any agreement between physicians!
Chris Snijders
Reilly tests Goldman’s idea
vs
82% 95%
physician Goldman’s scheme
Chris Snijders
A literature check …
Clinical versus statisticalprediction
For instance (zie Grove et al., 2000)
– Survival probabilities in medical procedures– Probability of recidivism– Probability of success of starting firms– Choice of job candidates– Diagnosing schizofrenia– Predicting school success– …
Chris Snijders
The results …
Over 160 studies
When given the same info,the number of cases in whichthe expert wins = ??
Chris Snijders
… and that is understandable
We emphasize the improbable’ (Stickler)
Confirmation bias (Edwards, Wason)
Hindsight bias (Fischhoff)
Cognitive dissonance (Festinger)
“Dealing with probabilities / Base rate neglect”(Bar-Hillel)
Mental sets (Redelmayer, Tversky)
Our memory fools us (Wagenaar)
Chris Snijders
And there are more of these
"Mental Floating Frankfurters"
Chris Snijders
Restriction 2: Memory
“Where were you, when …”
Shuttle Columbia Crew Lost Feb. 1, 2003
Chris Snijders
Restriction 3: the “availability heuristic”
What is more likely, a plain crash or a car crash?
Chris Snijders
Restriction 4: dealing with probabilities
Suppose: a manager has a good intuition in business:
– when a problem will arise: he gets a gut-feeling that something is wrong with probability 90%
– when no problem will arise: he gets a gut-feeling that something is wrong with probability 10%
On average, there is a problem in 5% of the transactions.
The manager starts a transaction, and he gets a gut-feeling that something might be wrong.
What is the probability that something is indeed wrong?
Chris Snijders
Restriction 4: dealing with probabilities
A murder has been committed. The only evidence available is DNA, found at the murder scene. DNA-research shows a match with your DNA.
The probability that two persons are diagnosed as having the same DNA is about 1 in 100.000.
How likely is it that you are the murderer?
Chris Snijders
Restriction 5: overconfidence
Trivial Pursuit: estimate how many questions you will know
Estimates are generally too high ... and this gets worse with expertise!
Chris Snijders
Restriction 6:Finding non-existent patterns
Chris Snijders
Restriction 7: the noble art of finding a broken leg
Chris Snijders
Decision making =Store, retrieve, combine
Chris Snijders
Our own experiments:purchasing managers
vs
Chris Snijders
A database of purchasing transactions
• Collected at Utrecht and Eindhoven University
• Since 1995• >4000 purchasing transactions• >1500 firms• 300 characteristics per transaction
Chris Snijders
The role of the purchasing manager
Purchasing transaction is to
be started
Purchasing transaction completed
BTW we chose economics, because it is relatively rare in this kind of research
Chris Snijders
Chris Snijders
Main idea• Purchasing professionals should be able to judge, for a given purchasing transaction, how likely it is that problems will arise.
• The test: we present purchasing professionals with a set of purchasing transactions. The professionals then judge how likely it is that problems occur.
• But … these transactions were sampled from our database of purchasing transactions. For all these transactions, we know which and how many problems occurred.
•We can therefore compare the predictions of the professionals with what actually happened.
Chris Snijders
Our test
Case from database
grade
6.2
ManagerStudent
ComputerLay
grade
5.4
grade
6.2
grade
6.9
Chris Snijders
Model-based prediction
• based on cross-validated “Multivariate Adaptive Regression Splines” (of order 1)
Chris Snijders
A set of experiments1. Original test (91 participants)
2. I&L test 1. A call in the Dutch “Tijdschrift voor inkoop en logistiek” (72 participants; webversion)
3. I&L test 2. After publication of the results in I&L, we left the website version open (72 additional participants)
4. MC. MasterClass in Strategic Purchasing and Supply Management, Corsendonk, Belgium (13 participants)
5. Rep03. Replication of original experiment, including small extension (118 participants)
6. PC03. 249 PanelClix participants (opt-in panel in NL)
7. IK05. 148 participants (48 professionals) as part of a class assignment in Information Sciences
Chris Snijders
Chris Snijders
Extensions (1)• “This is not what I do in my regular job”
Ask before during and after test whether professionals feel they are doing a good job.
• “I would have done better if I would have had more information”
Use either 7 or 14 pieces of information.
• “I usually perform better under pressure”
Do it again - now there is pressure.Do it again - with a time constraint.
Chris Snijders
Extensions (2)• Accountability
Have purchasing managers:- explain to the interviewer why they decide as they do- decide in groups of two
• How about differences between persons?
- younger professionals perform a bit better- those who “decide by instinct” perform a bit worse
Chris Snijders
The risk of experience
… trust increases with experience,
… but the test score does not ...
experience experience
trust
score
Chris Snijders
Seasoned vs young proExperienced• Relatively crude• Associating and fast• Comparison with experienced ‘prototypical cases’• Story telling But: • Overgeneralization• May seem efficient, but is not effective
Young pro• More ‘fine-tuning’• Deliberate and slow• Less experienced, and thereby less bothered by ‘prototypical cases’• More abstract• Less mistakes• Decisions may seem less efficient, but are relatively effective
"Recognition Primed
Decision making"
(Klein)
Chris Snijders
Improving experts• Expert system 1 Averaging per case across
professionals. For each case, use the average score of the experts
• Expert system 2 Averaging within professionals, across cases: for each expert, calculate the implicit model they are using. Use that model to predict again (based on volume, length of relation, ability to judge price/quality)
Increase from 0.19 0.23, still < 0.32
No improvement.
Chris Snijders
Improving experts (2)• Averaging professionals with the computer model (50%
pro, 50% model).
• Combining expert system 2 with the computer model (50% expert system 2, 50% model).
Increase from 0.19 0.27, still < 0.32.
Increase from 0.19 0.36 > 0.32 (p=0.024, 1-sided)
Chris Snijders
Kinds of e-purchasing
• Decrease tedious paperwork• Viewing your partners’ supplies• Online market places [auctioning]• Structuring decisions
… but there is an opportunity for …
• ... letting the computer decide! [real business intelligence]
Chris Snijders
For which kinds of decisions?
Purchasing to be done Purchasing done
How do I choose my purchasing team (how many persons, which levels, etc)?
How much time do I invest into writing a detailed list of demands?
How shall I choose my supplier?
What kind of contract shall I use? (tailor-made, standard, ...)
How much time and effort am I going to put into this transaction to make sure that it runs smooth?
Which other ways of trying to make sure that this transaction runs smooth shall I use?
….
1) Quantitative
2 ) Have data, or willingness to collect
3) Returning decisions
From those, choose the ones that are most important to you.
Chris Snijders
Beware
So purchasing managers / experts are no longer necessary?
Other tasks
e-decisionsOther decisions
Chris Snijders
Experts decide in different ways
– are more selective in their search for info– store information faster– use less information, and often this information is
combined in a non-linear way– have a more active pattern of contingent search:
they consider subsets of variables, different subsets in each case
– compare given information with the information in their knowledge base …
– … but choices are often based on over-generalization of specific cases
– use more “broken-leg cues”
Chris Snijders
… but not better
“process performance paradox”:
in a large number of tasks, the experts decide in different ways, but not better than those with a minimal amount of training.
(cf. Camerer and Johnson, 1991)
Chris Snijders
Do not take this to extremes...
Chris Snijders
Models in, intuition out?
No. RPD model works well, for specific tasks and you need to properly train your intuition
– deliberate practice– compile extensive experience bank– obtain accurate and timely feedback– enrich experiences by reviewing prior
ones
Chris Snijders
Gigerenzer: Fast and frugal• Humans use “fast and frugal” strategies (Simon: bounded
rationality, satisficing) or “Probabilistic Mental Models”
by employing simple decision-strategies individuals with limited knowledge are able to arrive at equally, or even more, correct predictions than individuals who have extensive knowledge
• Gigerenzer’s experiment
Which city has the most inhabitants?Arnhem Eindhoven
Try to see how such fast-and-frugal strategies can have a role in the experiments
Chris Snijders
Example: #inhabitants
Predict which of 2 US cities has the most homeless people, based on
– vacancy rate– temperature– unemployment rate– poverty rate– public housing
[all binary variables, cut at the median]
Chris Snijders
Strategies [when training set <> test set]
• Minimalist -• Take-the-best 63
• Dawes’ Rule 61• Multiple regression 60• Bayesian network 65
Chris Snijders
Fast-and-frugal design:left - same - right
Additional factor: time pressure / 48 Purchasing pros, 50 students, 50 super-laypeople
Chris Snijders
Results (1)
All cases Only extremes
Chris Snijders
Results (2)
Chris Snijders
Results (3)(only using the larger differences)
Perfect 100
Model 60Take-the-best 56Linear regression 43Unit regression (std) 43
Random 33
Chris Snijders
Some conclusions
• In our managerial context, model-based decision making clearly outperforms professional decision making
• Good model scores can be achieved by fast-and-frugal strategies...
• ... but the humans in our experiment clearly did not use this kind of logic
Chris Snijders
Implications
– one might be able to teach people to improve their judgment(s)
– in actual managerial practice, there might often be no need to develop fancy models to get a close to optimal improvement
– decision support by creating expert systems that mimic experts makes no sense in this context
– To get this implemented, you need two things: a Goldman and a Reilley
Chris Snijders
Bottom line
Model based prediction can and should be used much more often than it is now.
Chris Snijders
You have to get used to it ...
Chris Snijders
Example: Bordeaux wines
Chris Snijders
Try it yourself (1)
• http://islandia.law.yale.edu/ayres/predictionTools.htm
Chris Snijders
Try it yourself (2)• Life expectancy• Likelihood of marital dissolution• Expected length of a child• Prices of airline-tickets• Risk for and survival probabilities for different
diseases (incl. cancer, diabetes, osteoperosis, etc)
• Probability of bankruptcy• ...
Chris Snijders
Example: life expectancy
http://www.nmfn.com/tn/learnctr--lifeevents--longevity_game
Chris Snijders
Example: breast cancer
Chris Snijders
Example: marriage dissolution
Chris Snijders
Example: bankruptcy
top related