the improbability principle david j. hand imperial college, london and winton capital management...
TRANSCRIPT
![Page 1: The Improbability Principle David J. Hand Imperial College, London and Winton Capital Management This version has been redacted to remove images for copyright](https://reader035.vdocuments.us/reader035/viewer/2022062404/551a1ce255034654788b4741/html5/thumbnails/1.jpg)
The Improbability Principle
David J. HandImperial College, London
and Winton Capital Management
This version has been redacted to remove images for copyright reasons
1
![Page 2: The Improbability Principle David J. Hand Imperial College, London and Winton Capital Management This version has been redacted to remove images for copyright](https://reader035.vdocuments.us/reader035/viewer/2022062404/551a1ce255034654788b4741/html5/thumbnails/2.jpg)
- thinking of someone just before they phone you
- bumping into an old friend in a strange town
- meeting someone who has the same birthday as you
- finding that the number of your house keeps coming up in other situations
2
![Page 3: The Improbability Principle David J. Hand Imperial College, London and Winton Capital Management This version has been redacted to remove images for copyright](https://reader035.vdocuments.us/reader035/viewer/2022062404/551a1ce255034654788b4741/html5/thumbnails/3.jpg)
Things that make you wonder just what is going on?• Is there something we don’t understand about
the way the universe works?
• Are the laws of cause and effect wrong?
• Is something exerting an invisible influence on our lives?
3
![Page 4: The Improbability Principle David J. Hand Imperial College, London and Winton Capital Management This version has been redacted to remove images for copyright](https://reader035.vdocuments.us/reader035/viewer/2022062404/551a1ce255034654788b4741/html5/thumbnails/4.jpg)
• In a way the answer to these questions is yes• But that something is not a mystical
supernatural force• That something is a consequence of solid
probability theory• But looked at from an unusual perspective
The perspective of The Improbability Principle
4
![Page 5: The Improbability Principle David J. Hand Imperial College, London and Winton Capital Management This version has been redacted to remove images for copyright](https://reader035.vdocuments.us/reader035/viewer/2022062404/551a1ce255034654788b4741/html5/thumbnails/5.jpg)
extremely improbable eventsare commonplace
5
![Page 6: The Improbability Principle David J. Hand Imperial College, London and Winton Capital Management This version has been redacted to remove images for copyright](https://reader035.vdocuments.us/reader035/viewer/2022062404/551a1ce255034654788b4741/html5/thumbnails/6.jpg)
• Not a single law, like E = mc2
• But a set of five laws
1.The law of inevitability2.The law of truly large numbers3.The law of selection4.The law of the probability lever5.The law of near enough
6
![Page 7: The Improbability Principle David J. Hand Imperial College, London and Winton Capital Management This version has been redacted to remove images for copyright](https://reader035.vdocuments.us/reader035/viewer/2022062404/551a1ce255034654788b4741/html5/thumbnails/7.jpg)
The law of inevitability:Something must happen
You can’t tell on which blade of grass a golf ball will landBut you know it will land on one of them
You can’t tell which lottery ticket will come upBut you know that one of them will
7
![Page 8: The Improbability Principle David J. Hand Imperial College, London and Winton Capital Management This version has been redacted to remove images for copyright](https://reader035.vdocuments.us/reader035/viewer/2022062404/551a1ce255034654788b4741/html5/thumbnails/8.jpg)
The case of the International Lotto Fund
15 Feb 1992 Virginia State Lottery rollover jackpot $27m A 6/44 lottery: each ticket has a 1 in 7,059,052 chance Spend $7m to guarantee having the winning ticket
International Lotto Fund: Consortium of 2,500 small investorsAustralian, American, European, New Zealand
8
![Page 9: The Improbability Principle David J. Hand Imperial College, London and Winton Capital Management This version has been redacted to remove images for copyright](https://reader035.vdocuments.us/reader035/viewer/2022062404/551a1ce255034654788b4741/html5/thumbnails/9.jpg)
The logistics!
So complicated they managed to buy only 5m tickets Potential disaster - 1 / 4 chance of not buying the
winning ticket
Also maybe they would have to share the jackpot with others
Legal challenge: each ticket must be paid for at the terminal which printed it
9
![Page 10: The Improbability Principle David J. Hand Imperial College, London and Winton Capital Management This version has been redacted to remove images for copyright](https://reader035.vdocuments.us/reader035/viewer/2022062404/551a1ce255034654788b4741/html5/thumbnails/10.jpg)
The law of truly large numbers:with a large enough number of opportunities,
any outrageous thing is likely to happen
probability of being killed by a lightning strike in one year is very small: about 1 in 300,000
but the world has c7bn people
so the chance that no-one will be killed by lightning in one year is infinitesimal
about 24,000 people are killed each year10
![Page 11: The Improbability Principle David J. Hand Imperial College, London and Winton Capital Management This version has been redacted to remove images for copyright](https://reader035.vdocuments.us/reader035/viewer/2022062404/551a1ce255034654788b4741/html5/thumbnails/11.jpg)
The Tantilizer, the Bulgarian State lottery:6th September 2009 4, 15, 23, 24, 35, 4310th September 2009 4, 15, 23, 24, 35, 43
Sports Minister ordered an investigation
But the law of truly large numbers says we should expect such things to happen
It says it’s more likely than not that some two draws will match if two draws per week are made for 43 years
11
![Page 12: The Improbability Principle David J. Hand Imperial College, London and Winton Capital Management This version has been redacted to remove images for copyright](https://reader035.vdocuments.us/reader035/viewer/2022062404/551a1ce255034654788b4741/html5/thumbnails/12.jpg)
Consider the number of lotteries around the world- you’d expect to see duplicate sets of numbers drawn- even on consecutive draws
- it would be amazing if we never saw such repeats
Israel’s Miphal HaPayis lotterysame numbers on 21 Sept and 16 Oct 2010
North Carolina Cash 5 lotterysame numbers on 9 and 11 July 2007
12
![Page 13: The Improbability Principle David J. Hand Imperial College, London and Winton Capital Management This version has been redacted to remove images for copyright](https://reader035.vdocuments.us/reader035/viewer/2022062404/551a1ce255034654788b4741/html5/thumbnails/13.jpg)
The law of truly large numbers produces all sort of other lottery surprises
Maureen Wilcox bought tickets with the winning numbers for both the Massachusetts Lottery and the Rhode Island Lottery
Unfortunately, in each case the winning tickets were for the other lottery
13
![Page 14: The Improbability Principle David J. Hand Imperial College, London and Winton Capital Management This version has been redacted to remove images for copyright](https://reader035.vdocuments.us/reader035/viewer/2022062404/551a1ce255034654788b4741/html5/thumbnails/14.jpg)
The law of selection:you can make things as likely as you
want if you choose after the event
The arrows in the barn
Whole walnuts
Jeane DixonOn 13th May 1956, four years before John Kennedy was elected, Ms Dixon published a prediction in Parade Magazine that the 1960 US Presidential election would be won be a Democrat who would then be assassinated or die in office
14
![Page 15: The Improbability Principle David J. Hand Imperial College, London and Winton Capital Management This version has been redacted to remove images for copyright](https://reader035.vdocuments.us/reader035/viewer/2022062404/551a1ce255034654788b4741/html5/thumbnails/15.jpg)
But Jeane Dixon also predicted• that someone from the Soviet Union would
be the first to walk on the moon• that World War III would begin in 1958
The Jeane Dixon effect: • make enough predictions and you’re sure to
get some right
• tell people about those, and forget the wrong ones
15
![Page 16: The Improbability Principle David J. Hand Imperial College, London and Winton Capital Management This version has been redacted to remove images for copyright](https://reader035.vdocuments.us/reader035/viewer/2022062404/551a1ce255034654788b4741/html5/thumbnails/16.jpg)
Fraudulent share option backdating schemes
Redacted image
Wall St Journal 18 March 2006:1 in 200 million1 in 6 billion1 in 300 billion
16
![Page 17: The Improbability Principle David J. Hand Imperial College, London and Winton Capital Management This version has been redacted to remove images for copyright](https://reader035.vdocuments.us/reader035/viewer/2022062404/551a1ce255034654788b4741/html5/thumbnails/17.jpg)
The law of the probability lever:slight changes can make highly
improbable events almost certain
c.100 people get killed by lightning in the US each year
probability of getting killed by lightning in the US is about 1 in 3 million
= 1/10th worldwide probability
Because: US buildings are better protected people spend more time inside
17
![Page 18: The Improbability Principle David J. Hand Imperial College, London and Winton Capital Management This version has been redacted to remove images for copyright](https://reader035.vdocuments.us/reader035/viewer/2022062404/551a1ce255034654788b4741/html5/thumbnails/18.jpg)
Major Walter Summerford: 1918: knocked from his horse by lighting in
Flanders – legs temporarily paralyzed 1924: moved to Canada and took up fishing, where
the tree he was under was struck by lighting, paralyzing his right side
1930: walking in park: completely paralyzed by a lightning strike
1932: died - not from a lightning strike 1936: oversight remedied: his gravestone was
struck by lightning18
![Page 19: The Improbability Principle David J. Hand Imperial College, London and Winton Capital Management This version has been redacted to remove images for copyright](https://reader035.vdocuments.us/reader035/viewer/2022062404/551a1ce255034654788b4741/html5/thumbnails/19.jpg)
Roy Sullivan: struck by lightning 1942: lost big toe nail 1969: lost eyebrows 1970: seared left shoulder 1972: hair set on fire 1973: regrown hair burnt 1976: ankle injured 1977: chest and stomach burned
Roy Sullivan was a Virginia park ranger
19
![Page 20: The Improbability Principle David J. Hand Imperial College, London and Winton Capital Management This version has been redacted to remove images for copyright](https://reader035.vdocuments.us/reader035/viewer/2022062404/551a1ce255034654788b4741/html5/thumbnails/20.jpg)
Financial crashes
Sebastian Mallaby, in More Money than God, writing about the year 1987
‘... an event such as the crash would not be anticipated to occur even if the stock market were to remain open for twenty billion years..’
20
![Page 21: The Improbability Principle David J. Hand Imperial College, London and Winton Capital Management This version has been redacted to remove images for copyright](https://reader035.vdocuments.us/reader035/viewer/2022062404/551a1ce255034654788b4741/html5/thumbnails/21.jpg)
Roger Lowenstein, in When Genius Failed,writing about the year 1998
‘... the figures implied it would take a so-called ten sigma event ... for the firm to lose all its capital in one year..’
21
![Page 22: The Improbability Principle David J. Hand Imperial College, London and Winton Capital Management This version has been redacted to remove images for copyright](https://reader035.vdocuments.us/reader035/viewer/2022062404/551a1ce255034654788b4741/html5/thumbnails/22.jpg)
Bill Bonner, in Money Week,writing about the year 2007
‘... things were happening then that were only supposed to happen about once in every 100,000 years’
22
![Page 23: The Improbability Principle David J. Hand Imperial College, London and Winton Capital Management This version has been redacted to remove images for copyright](https://reader035.vdocuments.us/reader035/viewer/2022062404/551a1ce255034654788b4741/html5/thumbnails/23.jpg)
David Gartman in The Gartman Letter,writing about the year 2010
‘... What we witnessed yesterday was a series of movements of utterly unprecedented proportions, with currency price changes that are at the 6th and 7th and 8th standard deviations from the norm...’
23
![Page 24: The Improbability Principle David J. Hand Imperial College, London and Winton Capital Management This version has been redacted to remove images for copyright](https://reader035.vdocuments.us/reader035/viewer/2022062404/551a1ce255034654788b4741/html5/thumbnails/24.jpg)
Gaussian vs Cauchy probabilities of extreme values
Standard Gaussian distribution Prob (x > 10) = 1 in 1023
Cauchy distribution Prob (x > 10) = 1 in 30
The second is 4,200,000,000,000,000,000,000time as probable as the first
24
![Page 25: The Improbability Principle David J. Hand Imperial College, London and Winton Capital Management This version has been redacted to remove images for copyright](https://reader035.vdocuments.us/reader035/viewer/2022062404/551a1ce255034654788b4741/html5/thumbnails/25.jpg)
25
The law of near enough:events which are sufficiently similar are
regarded as identical
1986 Bill Shaw survived a train crash that killed 13 people
2001 Ginny Shaw, Bill’s wife, survived a train crash that killed 10 people
0.1 fatalities per billion passenger miles in UK- How long a gap between their accidents?- Husband and wife or more distant relations?
![Page 26: The Improbability Principle David J. Hand Imperial College, London and Winton Capital Management This version has been redacted to remove images for copyright](https://reader035.vdocuments.us/reader035/viewer/2022062404/551a1ce255034654788b4741/html5/thumbnails/26.jpg)
26
Telepathy
Sir Alister Hardy carried out a very large famous experiment to test for telepathic ability
Transmitters looked at drawings which receivers tried to identify
But when is the “received” image sufficiently similar to the “transmitted” one to count as a match?
![Page 27: The Improbability Principle David J. Hand Imperial College, London and Winton Capital Management This version has been redacted to remove images for copyright](https://reader035.vdocuments.us/reader035/viewer/2022062404/551a1ce255034654788b4741/html5/thumbnails/27.jpg)
27
“...selected examples of ostensible ‘direct hits’ in the telepathy experiments where drawings were used as targets...”
Redacted images
![Page 28: The Improbability Principle David J. Hand Imperial College, London and Winton Capital Management This version has been redacted to remove images for copyright](https://reader035.vdocuments.us/reader035/viewer/2022062404/551a1ce255034654788b4741/html5/thumbnails/28.jpg)
28
Redacted Images
![Page 29: The Improbability Principle David J. Hand Imperial College, London and Winton Capital Management This version has been redacted to remove images for copyright](https://reader035.vdocuments.us/reader035/viewer/2022062404/551a1ce255034654788b4741/html5/thumbnails/29.jpg)
29
Each law by itself can make apparently highly improbable events happen
But when the laws work together ...
For example:How to be a successful stock tipster
- the law of inevitability- the law of selection
![Page 30: The Improbability Principle David J. Hand Imperial College, London and Winton Capital Management This version has been redacted to remove images for copyright](https://reader035.vdocuments.us/reader035/viewer/2022062404/551a1ce255034654788b4741/html5/thumbnails/30.jpg)
30
The human mindThe conjunction fallacy
John initially took a degree in mathematics, and followed it with a PhD in astrophysics. After that, he worked in the physics department of a university for a while but then found a job in the back room of an algorithmic trading company, developing highly sophisticated statistical models for predicting movements of the financial markets. In his spare time he attends science fiction conventions.
Which has the higher probability?
A: John is married with two children
B: John is married with two children, and likes to spend his evenings tackling mathematical puzzles and playing computer games
![Page 31: The Improbability Principle David J. Hand Imperial College, London and Winton Capital Management This version has been redacted to remove images for copyright](https://reader035.vdocuments.us/reader035/viewer/2022062404/551a1ce255034654788b4741/html5/thumbnails/31.jpg)
31
Law of transposed conditional
Base rate fallacy
Possibility effect
Hindsight bias
The law of small numbers
. . . . . .
And, perhaps above all, pareidoliaSeeing patterns in random data
![Page 32: The Improbability Principle David J. Hand Imperial College, London and Winton Capital Management This version has been redacted to remove images for copyright](https://reader035.vdocuments.us/reader035/viewer/2022062404/551a1ce255034654788b4741/html5/thumbnails/32.jpg)
32
Beyond the everyday
To life, the universe, and everything
Evolution: the law of truly large numbers the law of selection
The universe and the anthropic principle the ultimate instance of the law of selection
![Page 33: The Improbability Principle David J. Hand Imperial College, London and Winton Capital Management This version has been redacted to remove images for copyright](https://reader035.vdocuments.us/reader035/viewer/2022062404/551a1ce255034654788b4741/html5/thumbnails/33.jpg)
33
Using the improbability principle
Balancing probabilitiesBending probabilitiesBelieving probabilitiesAnd Beyond
![Page 34: The Improbability Principle David J. Hand Imperial College, London and Winton Capital Management This version has been redacted to remove images for copyright](https://reader035.vdocuments.us/reader035/viewer/2022062404/551a1ce255034654788b4741/html5/thumbnails/34.jpg)
34
Using the improbability principle
Balancing probabilitiesBending probabilitiesBelieving probabilitiesAnd Beyond
Choose a die andthrow it 100 times
All 100 show 6
Do you think the die is white or black?
![Page 35: The Improbability Principle David J. Hand Imperial College, London and Winton Capital Management This version has been redacted to remove images for copyright](https://reader035.vdocuments.us/reader035/viewer/2022062404/551a1ce255034654788b4741/html5/thumbnails/35.jpg)
35
Using the improbability principle
Balancing probabilitiesBending probabilitiesBelieving probabilitiesAnd Beyond
Buy one 6/49 lottery ticket per week for 20 yearsProb jackpot = 2 in 26893
Buy 52×20 (different) tickets in one weekProb jackpot = 2 in 26892
52×20 different tickets!
Except: allow for The Improbability Principle:In April 2013 Harry Black bought two British Columbia Lottery tickets with the same numbers, and won
![Page 36: The Improbability Principle David J. Hand Imperial College, London and Winton Capital Management This version has been redacted to remove images for copyright](https://reader035.vdocuments.us/reader035/viewer/2022062404/551a1ce255034654788b4741/html5/thumbnails/36.jpg)
36
Using the improbability principle
Balancing probabilitiesBending probabilitiesBelieving probabilitiesAnd Beyond
Why movie sequels are less likely to be big hits than the originals
Why US TV programs are better in the UK; and UK programs are better in the US
Why the gambler’s fallacy is wrong
![Page 37: The Improbability Principle David J. Hand Imperial College, London and Winton Capital Management This version has been redacted to remove images for copyright](https://reader035.vdocuments.us/reader035/viewer/2022062404/551a1ce255034654788b4741/html5/thumbnails/37.jpg)
37
Using the improbability principle
Balancing probabilitiesBending probabilitiesBelieving probabilitiesAnd Beyond
![Page 38: The Improbability Principle David J. Hand Imperial College, London and Winton Capital Management This version has been redacted to remove images for copyright](https://reader035.vdocuments.us/reader035/viewer/2022062404/551a1ce255034654788b4741/html5/thumbnails/38.jpg)
Why extremely improbable eventsare commonplace
The law of inevitability The law of truly large numbers The law of selection The law of the probability lever The law of near enough
And the human mind
![Page 39: The Improbability Principle David J. Hand Imperial College, London and Winton Capital Management This version has been redacted to remove images for copyright](https://reader035.vdocuments.us/reader035/viewer/2022062404/551a1ce255034654788b4741/html5/thumbnails/39.jpg)
The Improbability Principle
Why incredibly unlikely things keep happening
David J. HandBantam Press
39