Hi Harry, after your seminar at Monash yesterday I speculated on how well 4-out-of-5 confirming experiments can be considered to support a hypothesis.

Matt Coller School of Geography and Environmental Science

Monash University --

I've crunched out some numbers, to work out the exact probabilities. I started with a couple of assumptions:

Assume to begin with, the hypothesis has a 50:50 chance of being right or wrong.

0 positives: 0.002% likelihood of being correct

That a typical experiment would have a 10% chance of giving a false positive or false negative result.

1 positive: 0.13% likelihood of being correct

2 positives: 10% likelihood of being correct

3 positives: 90% likelihood of being correct

4 positives: 99.86% likelihood of being correct

5 positives: 99.998% likelihood of being correct

So if you get four positive answers, and only one disagrees, it is a thousand times more likely that the experiment that disagrees has yielded a false negative, rather than that the all the other four are giving false positives

Matt says: Even I was a bit surprised just how strongly 4 positives out of 5 support a hypothesis… (99.9%).

ie. you can be 99.9% certain that your hypothesis is correct.

So it turns out your (HK’s) rule of thumb has a good mathematical basis indeed!

What matters is not so much how likely it is that four out of five experiments would give true answers, but how very unlikely it is that those four experiments would give false positives (which is 0.04%).

Hi Harry, I met you briefly after your wonderful seminar at Monash yesterday, and speculated on how well 4 out of 5 confirming experiments can be considered to support a hypothesis. I've crunched some numbers, to work out the exact probabilities. I started with a couple of assumptions:

That to begin with, the hypothesis has a 50:50 chance of being right or wrong.

That a typical experiment would have a 10% chance of giving a false positive or false negative result.

Here's how the results from five experiments would confirm or debunk your hypothesis:

0 positives: hypothesis has 0.002% likelihood of being correct 1 positives: hypothesis has 0.13% likelihood of being correct 2 positives: hypothesis has 10% likelihood of being correct 3 positives: hypothesis has 90% likelihood of being correct 4 positives: hypothesis has 99.86% likelihood of being correct 5 positives: hypothesis has 99.998% likelihood of being correct

Even I was a bit surprised just how strongly 4 positives out of 5 will support a hypothesis (99.9%). I realise what matters is not so much how likely it is that four out of five experiments would give true answers, but how very unlikely it is that those four experiments would give false positives (which is 0.04%). So if you get four concurrent answers, and one which disagrees, it's a thousand times more likely that only one experiment is giving a false negative, rather than that the other four are giving false positives - ie. you can be 99.9% certain that your hypothesis is correct. So it turns out your rule of thumb has a good mathematical basis indeed! All the best, Matt Coller School of Geography and Environmental Science Monash University --