ioannidis 2005

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Why most published research findings are falseArticle by John P. A. Ioannidis (2005)

Aurelien Madouasse

November 4, 2011

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Plan

1 Context

2 Introduction

3 Modelling FrameworkHypothesis testingBiasMultiple testingComments

4 Corollaries

5 Conclusion

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Context

• The author: John P.A. Ioannidis

• C.F. Rehnborg Chair in Disease Prevention at StanfordUniversity (US)

• Professor of Medicine and Director of the StanfordPrevention Research Center (US)

• Chaired the Department of Hygiene and Epidemiology atthe University of Ioannina School of Medicine (Greece)

• Has a 51 page CV

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Context

• The author: John P.A. Ioannidis• C.F. Rehnborg Chair in Disease Prevention at Stanford

University (US)• Professor of Medicine and Director of the Stanford

Prevention Research Center (US)• Chaired the Department of Hygiene and Epidemiology at

the University of Ioannina School of Medicine (Greece)• Has a 51 page CV

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Context

• The journal: PLoS Medicine

• Public Library of Science• Peer reviewed• Open Access• Publication fee: US$2900

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Context

• The journal: PLoS Medicine• Public Library of Science• Peer reviewed• Open Access• Publication fee: US$2900

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Context

• The article (Checked 2011-10-22)

• Views: 410,087• Citations:

• CrossRef: 312• PubMed Central: 118• Scopus: 579• Web of Science: 585

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Context

• The article (Checked 2011-10-22)• Views: 410,087• Citations:

• CrossRef: 312• PubMed Central: 118• Scopus: 579• Web of Science: 585

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Plan

1 Context

2 Introduction

3 Modelling FrameworkHypothesis testingBiasMultiple testingComments

4 Corollaries

5 Conclusion

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Introduction

• Published research findings sometimes refuted bysubsequent evidence

• Increasing concern false findings may be the majority

• This should no be surprising

• Here is why . . .

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Plan

1 Context

2 Introduction

3 Modelling FrameworkHypothesis testingBiasMultiple testingComments

4 Corollaries

5 Conclusion

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Hypothesis testing

• Consider a parameter measured in a population ofindividuals with a disease:

• Before treatment

• After treatment (Here assuming the treatment has an effect)

Some Parameter

Fre

quen

cy

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Hypothesis testing

• Consider a parameter measured in a population ofindividuals with a disease:

• Before treatment• After treatment (Here assuming the treatment has an effect)

Some Parameter

Fre

quen

cy

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Hypothesis Testing

• We want to know whether the treatment has an effect

• We make a hypothesis

• H0: The treatment has no effect

• We test our hypothesis

• We get a result

• If H0 were true, the probability of observing our datawould be . . .

• p(data|H0) = p − value

• We draw a conclusion

• If p(data|H0) > 0.05 we accept H0 → No effect• If p(data|H0) ≤ 0.05 we reject H0 → Effect

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Hypothesis Testing

• We want to know whether the treatment has an effect

• We make a hypothesis

• H0: The treatment has no effect

• We test our hypothesis

• We get a result

• If H0 were true, the probability of observing our datawould be . . .

• p(data|H0) = p − value

• We draw a conclusion

• If p(data|H0) > 0.05 we accept H0 → No effect• If p(data|H0) ≤ 0.05 we reject H0 → Effect

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Hypothesis Testing

• We want to know whether the treatment has an effect

• We make a hypothesis• H0: The treatment has no effect

• We test our hypothesis

• We get a result

• If H0 were true, the probability of observing our datawould be . . .

• p(data|H0) = p − value

• We draw a conclusion

• If p(data|H0) > 0.05 we accept H0 → No effect• If p(data|H0) ≤ 0.05 we reject H0 → Effect

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Hypothesis Testing

• We want to know whether the treatment has an effect

• We make a hypothesis• H0: The treatment has no effect

• We test our hypothesis

• We get a result

• If H0 were true, the probability of observing our datawould be . . .

• p(data|H0) = p − value

• We draw a conclusion

• If p(data|H0) > 0.05 we accept H0 → No effect• If p(data|H0) ≤ 0.05 we reject H0 → Effect

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Hypothesis Testing

• We want to know whether the treatment has an effect

• We make a hypothesis• H0: The treatment has no effect

• We test our hypothesis

• We get a result

• If H0 were true, the probability of observing our datawould be . . .

• p(data|H0) = p − value

• We draw a conclusion

• If p(data|H0) > 0.05 we accept H0 → No effect• If p(data|H0) ≤ 0.05 we reject H0 → Effect

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Hypothesis Testing

• We want to know whether the treatment has an effect

• We make a hypothesis• H0: The treatment has no effect

• We test our hypothesis

• We get a result• If H0 were true, the probability of observing our data

would be . . .

• p(data|H0) = p − value

• We draw a conclusion

• If p(data|H0) > 0.05 we accept H0 → No effect• If p(data|H0) ≤ 0.05 we reject H0 → Effect

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Hypothesis Testing

• We want to know whether the treatment has an effect

• We make a hypothesis• H0: The treatment has no effect

• We test our hypothesis

• We get a result• If H0 were true, the probability of observing our data

would be . . .• p(data|H0) = p − value

• We draw a conclusion

• If p(data|H0) > 0.05 we accept H0 → No effect• If p(data|H0) ≤ 0.05 we reject H0 → Effect

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Hypothesis Testing

• We want to know whether the treatment has an effect

• We make a hypothesis• H0: The treatment has no effect

• We test our hypothesis

• We get a result• If H0 were true, the probability of observing our data

would be . . .• p(data|H0) = p − value

• We draw a conclusion

• If p(data|H0) > 0.05 we accept H0 → No effect• If p(data|H0) ≤ 0.05 we reject H0 → Effect

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Hypothesis Testing

• We want to know whether the treatment has an effect

• We make a hypothesis• H0: The treatment has no effect

• We test our hypothesis

• We get a result• If H0 were true, the probability of observing our data

would be . . .• p(data|H0) = p − value

• We draw a conclusion• If p(data|H0) > 0.05 we accept H0 → No effect

• If p(data|H0) ≤ 0.05 we reject H0 → Effect

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Hypothesis Testing

• We want to know whether the treatment has an effect

• We make a hypothesis• H0: The treatment has no effect

• We test our hypothesis

• We get a result• If H0 were true, the probability of observing our data

would be . . .• p(data|H0) = p − value

• We draw a conclusion• If p(data|H0) > 0.05 we accept H0 → No effect• If p(data|H0) ≤ 0.05 we reject H0 → Effect

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Hypothesis Testing

• This framework assumes that we accept to be wrong . . .

sometimes

TruthTrue relationship No relationship

Trial

Relationship 1 − β αNo relationship β 1 − α

• α = probability of declaring a relationship when there isnone - Type I error

• β = probability of finding no relationship when there isone - Type II error

• 1− β = probability of finding a relationship when there isone - Power

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Hypothesis Testing

• This framework assumes that we accept to be wrong . . .

sometimes

TruthTrue relationship No relationship

Trial

Relationship 1 − β αNo relationship β 1 − α

• α = probability of declaring a relationship when there isnone - Type I error

• β = probability of finding no relationship when there isone - Type II error

• 1− β = probability of finding a relationship when there isone - Power

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Hypothesis Testing

• This framework assumes that we accept to be wrong . . .

sometimes

TruthTrue relationship No relationship

Trial

Relationship 1 − β αNo relationship β 1 − α

• α = probability of declaring a relationship when there isnone - Type I error

• β = probability of finding no relationship when there isone - Type II error

• 1− β = probability of finding a relationship when there isone - Power

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Hypothesis Testing

• This framework assumes that we accept to be wrong . . .

sometimes

TruthTrue relationship No relationship

Trial

Relationship 1 − β αNo relationship β 1 − α

• α = probability of declaring a relationship when there isnone - Type I error

• β = probability of finding no relationship when there isone - Type II error

• 1− β = probability of finding a relationship when there isone - Power

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Hypothesis Testing

• This framework assumes that we accept to be wrong . . .

sometimes

TruthTrue relationship No relationship

Trial

Relationship 1 − β αNo relationship β 1 − α

• α = probability of declaring a relationship when there isnone - Type I error

• β = probability of finding no relationship when there isone - Type II error

• 1− β = probability of finding a relationship when there isone - Power

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Hypothesis Testing

• This framework assumes that we accept to be wrong . . .

sometimes

TruthTrue relationship No relationship

Trial

Relationship 1 − β αNo relationship β 1 − α

• α = probability of declaring a relationship when there isnone - Type I error

• β = probability of finding no relationship when there isone - Type II error

• 1− β = probability of finding a relationship when there isone - Power

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Hypothesis Testing

TruthTrue relationship No relationship

Trial

Relationship 1 − β αNo relationship β 1 − α

• For a given hypothesis, whether we get it wrong dependson:

• Whether the hypothesis is true• The magnitude of the effect• The values we choose for α and β

Some Parameter

Fre

quen

cy

Some Parameter

Fre

quen

cy

Some Parameter

Fre

quen

cy

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Modelling the Framework for FalsePositive Findings

TruthTrue relationship No relationship

Trial

Relationship 1 − β αNo relationship β 1 − α

Total p 1 − p

• Central point of the paper

• Consider a population of possible hypotheses• Among these hypotheses, a proportion p are True• Hypothesis testing can be seen as testing for a disease in

Epidemiology• 1− β is the sensitivity• 1− α is the specificity• We can define a positive predictive value

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Modelling the Framework for FalsePositive Findings

TruthTrue relationship No relationship

Trial

Relationship 1 − β αNo relationship β 1 − α

Total p 1 − p

• Central point of the paper• Consider a population of possible hypotheses

• Among these hypotheses, a proportion p are True• Hypothesis testing can be seen as testing for a disease in

Epidemiology• 1− β is the sensitivity• 1− α is the specificity• We can define a positive predictive value

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Modelling the Framework for FalsePositive Findings

TruthTrue relationship No relationship

Trial

Relationship 1 − β αNo relationship β 1 − α

Total p 1 − p

• Central point of the paper• Consider a population of possible hypotheses• Among these hypotheses, a proportion p are True

• Hypothesis testing can be seen as testing for a disease inEpidemiology

• 1− β is the sensitivity• 1− α is the specificity• We can define a positive predictive value

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Modelling the Framework for FalsePositive Findings

TruthTrue relationship No relationship

Trial

Relationship 1 − β αNo relationship β 1 − α

Total p 1 − p

• Central point of the paper• Consider a population of possible hypotheses• Among these hypotheses, a proportion p are True• Hypothesis testing can be seen as testing for a disease in

Epidemiology

• 1− β is the sensitivity• 1− α is the specificity• We can define a positive predictive value

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Modelling the Framework for FalsePositive Findings

TruthTrue relationship No relationship

Trial

Relationship 1 − β αNo relationship β 1 − α

Total p 1 − p

• Central point of the paper• Consider a population of possible hypotheses• Among these hypotheses, a proportion p are True• Hypothesis testing can be seen as testing for a disease in

Epidemiology• 1− β is the sensitivity

• 1− α is the specificity• We can define a positive predictive value

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Modelling the Framework for FalsePositive Findings

TruthTrue relationship No relationship

Trial

Relationship 1 − β αNo relationship β 1 − α

Total p 1 − p

• Central point of the paper• Consider a population of possible hypotheses• Among these hypotheses, a proportion p are True• Hypothesis testing can be seen as testing for a disease in

Epidemiology• 1− β is the sensitivity• 1− α is the specificity

• We can define a positive predictive value

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Modelling the Framework for FalsePositive Findings

TruthTrue relationship No relationship

Trial

Relationship 1 − β αNo relationship β 1 − α

Total p 1 − p

• Central point of the paper• Consider a population of possible hypotheses• Among these hypotheses, a proportion p are True• Hypothesis testing can be seen as testing for a disease in

Epidemiology• 1− β is the sensitivity• 1− α is the specificity• We can define a positive predictive value

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Modelling the Framework for FalsePositive Findings

TruthTrue relationship No relationship

Trial

Relationship 1 − β αNo relationship β 1 − α

Total p 1 − p

• Positive predictive value

• Ioannidis uses R = p1−p

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Modelling the Framework for FalsePositive Findings

TruthTrue relationship No relationship

Trial

Relationship 1 − β αNo relationship β 1 − α

Total p 1 − p

• Positive predictive value

PPV =p(1− β)

p(1− β) + (1− p)α

• Ioannidis uses R = p1−p

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Modelling the Framework for FalsePositive Findings

TruthTrue relationship No relationship

Trial

Relationship 1 − β αNo relationship β 1 − α

Total p 1 − p

• Positive predictive value

• Ioannidis uses R = p1−p

PPV =R

1+R × (1− β)R

1+R × (1− β) + 11+R × α

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Modelling the Framework for FalsePositive Findings

TruthTrue relationship No relationship

Trial

Relationship 1 − β αNo relationship β 1 − α

Total p 1 − p

• Positive predictive value

• Ioannidis uses R = p1−p

PPV =R(1− β)

R(1− β) + α

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Bias

• Among the studies that should have been reported asnegative

• A proportion u are reported as positive because of bias

TruthTrue relationship No relationship

Trial

Relationship 1 − β αNo relationship β 1 − α

Total p 1 − p

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Bias

• Among the studies that should have been reported asnegative

• A proportion u are reported as positive because of bias

TruthTrue relationship No relationship

Trial

Relationship 1 − β + uβ α + u(1 − α)No relationship (1 − u)β (1 − u)(1 − α)

Total p 1 − p

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Bias

TruthTrue relationship No relationship

Trial

Relationship 1 − β + uβ α + u(1 − α)No relationship (1 − u)β (1 − u)(1 − α)

Total p 1 − p

• Positive predictive value

• Ioannidis uses R = p1−p

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Bias

TruthTrue relationship No relationship

Trial

Relationship 1 − β + uβ α + u(1 − α)No relationship (1 − u)β (1 − u)(1 − α)

Total p 1 − p

• Positive predictive value

PPV =p(1− β + uβ)

p(1− β + uβ) + (1− p)(α + u(1− α))

• Ioannidis uses R = p1−p

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Bias

TruthTrue relationship No relationship

Trial

Relationship 1 − β + uβ α + u(1 − α)No relationship (1 − u)β (1 − u)(1 − α)

Total p 1 − p

• Positive predictive value

• Ioannidis uses R = p1−p

PPV =R(1− β) + uβR

R + α− βR + u − uα + uβR

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Bias

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Pre−study odds

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Pre−study odds

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Pre−study odds

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Pre−study odds

Pos

t−st

udy

prob

abili

ty (

PP

V)

u = 0.05u = 0.2u = 0.5u = 0.8

0.0 0.2 0.4 0.6 0.8 1.00.

00.

20.

40.

60.

81.

0

Pre−study probability

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.00.

00.

20.

40.

60.

81.

0

Pre−study probability

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.00.

00.

20.

40.

60.

81.

0

Pre−study probability

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.00.

00.

20.

40.

60.

81.

0

Pre−study probability

Pos

t−st

udy

prob

abili

ty (

PP

V)

u = 0.05u = 0.2u = 0.5u = 0.8

Power = 0.8

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Bias

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Pre−study odds

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Pre−study odds

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Pre−study odds

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Pre−study odds

Pos

t−st

udy

prob

abili

ty (

PP

V)

u = 0.05u = 0.2u = 0.5u = 0.8

0.0 0.2 0.4 0.6 0.8 1.00.

00.

20.

40.

60.

81.

0

Pre−study probability

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.00.

00.

20.

40.

60.

81.

0

Pre−study probability

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.00.

00.

20.

40.

60.

81.

0

Pre−study probability

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.00.

00.

20.

40.

60.

81.

0

Pre−study probability

Pos

t−st

udy

prob

abili

ty (

PP

V)

u = 0.05u = 0.2u = 0.5u = 0.8

Power = 0.5

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Bias

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Pre−study odds

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Pre−study odds

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Pre−study odds

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Pre−study odds

Pos

t−st

udy

prob

abili

ty (

PP

V)

u = 0.05u = 0.2u = 0.5u = 0.8

0.0 0.2 0.4 0.6 0.8 1.00.

00.

20.

40.

60.

81.

0

Pre−study probability

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.00.

00.

20.

40.

60.

81.

0

Pre−study probability

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.00.

00.

20.

40.

60.

81.

0

Pre−study probability

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.00.

00.

20.

40.

60.

81.

0

Pre−study probability

Pos

t−st

udy

prob

abili

ty (

PP

V)

u = 0.05u = 0.2u = 0.5u = 0.8

Power = 0.2

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Testing by Several IndependentTeams

• Increases the probability of a positive finding . . . by chance

• Positive findings more likely to be published

• Association with publication bias?

• Positive findings more likely to receive attention

• Probability of at least one positive finding:

1 - probability of negative findings only

TruthTrue relationship No relationship

Trial

Relationship 1 − βn 1 − (1 − α)n

No relationship βn (1 − α)n

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Testing by Several IndependentTeams

• Increases the probability of a positive finding . . . by chance

• Positive findings more likely to be published• Association with publication bias?

• Positive findings more likely to receive attention

• Probability of at least one positive finding:

1 - probability of negative findings only

TruthTrue relationship No relationship

Trial

Relationship 1 − βn 1 − (1 − α)n

No relationship βn (1 − α)n

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Testing by Several IndependentTeams

• Increases the probability of a positive finding . . . by chance

• Positive findings more likely to be published• Association with publication bias?

• Positive findings more likely to receive attention

• Probability of at least one positive finding:

1 - probability of negative findings only

TruthTrue relationship No relationship

Trial

Relationship 1 − βn 1 − (1 − α)n

No relationship βn (1 − α)n

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Testing by Several IndependentTeams

• Increases the probability of a positive finding . . . by chance

• Positive findings more likely to be published• Association with publication bias?

• Positive findings more likely to receive attention

• Probability of at least one positive finding:

1 - probability of negative findings only

TruthTrue relationship No relationship

Trial

Relationship 1 − βn 1 − (1 − α)n

No relationship βn (1 − α)n

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Testing by Several IndependentTeams

• Increases the probability of a positive finding . . . by chance

• Positive findings more likely to be published• Association with publication bias?

• Positive findings more likely to receive attention

• Probability of at least one positive finding:

1 - probability of negative findings only

TruthTrue relationship No relationship

Trial

Relationship 1 − βn 1 − (1 − α)n

No relationship βn (1 − α)n

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Testing by Several IndependentTeams

• Increases the probability of a positive finding . . . by chance

• Positive findings more likely to be published• Association with publication bias?

• Positive findings more likely to receive attention

• Probability of at least one positive finding:

1 - probability of negative findings only

TruthTrue relationship No relationship

Trial

Relationship 1 − βn 1 − (1 − α)n

No relationship βn (1 − α)n

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Testing by Several IndependentTeams

TruthTrue relationship No relationship

Trial

Relationship 1 − βn 1 − (1 − α)n

No relationship βn (1 − α)n

Total p 1 − p

• Positive predictive value

• Ioannidis uses R = p1−p

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Testing by Several IndependentTeams

TruthTrue relationship No relationship

Trial

Relationship 1 − βn 1 − (1 − α)n

No relationship βn (1 − α)n

Total p 1 − p

• Positive predictive value

PPV =p(1− βn)

p(1− βn) + (1− p)(1− (1− α)n)

• Ioannidis uses R = p1−p

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Testing by Several IndependentTeams

TruthTrue relationship No relationship

Trial

Relationship 1 − βn 1 − (1 − α)n

No relationship βn (1 − α)n

Total p 1 − p

• Positive predictive value

• Ioannidis uses R = p1−p

PPV =R(1− βn)

R + 1− ((1− α)n + Rβn)

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Testing by Several IndependentTeams

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Pre−study odds

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Pre−study odds

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Pre−study odds

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Pre−study odds

Pos

t−st

udy

prob

abili

ty (

PP

V)

n = 1n = 5n = 10n = 50

0.0 0.2 0.4 0.6 0.8 1.00.

00.

20.

40.

60.

81.

0

Pre−study probability

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.00.

00.

20.

40.

60.

81.

0

Pre−study probability

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.00.

00.

20.

40.

60.

81.

0

Pre−study probability

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.00.

00.

20.

40.

60.

81.

0

Pre−study probability

Pos

t−st

udy

prob

abili

ty (

PP

V)

n = 1n = 5n = 10n = 50

Power = 0.8

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Testing by Several IndependentTeams

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Pre−study odds

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Pre−study odds

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Pre−study odds

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Pre−study odds

Pos

t−st

udy

prob

abili

ty (

PP

V)

n = 1n = 5n = 10n = 50

0.0 0.2 0.4 0.6 0.8 1.00.

00.

20.

40.

60.

81.

0

Pre−study probability

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.00.

00.

20.

40.

60.

81.

0

Pre−study probability

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.00.

00.

20.

40.

60.

81.

0

Pre−study probability

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.00.

00.

20.

40.

60.

81.

0

Pre−study probability

Pos

t−st

udy

prob

abili

ty (

PP

V)

n = 1n = 5n = 10n = 50

Power = 0.5

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Testing by Several IndependentTeams

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Pre−study odds

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Pre−study odds

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Pre−study odds

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Pre−study odds

Pos

t−st

udy

prob

abili

ty (

PP

V)

n = 1n = 5n = 10n = 50

0.0 0.2 0.4 0.6 0.8 1.00.

00.

20.

40.

60.

81.

0

Pre−study probability

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.00.

00.

20.

40.

60.

81.

0

Pre−study probability

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.00.

00.

20.

40.

60.

81.

0

Pre−study probability

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.00.

00.

20.

40.

60.

81.

0

Pre−study probability

Pos

t−st

udy

prob

abili

ty (

PP

V)

n = 1n = 5n = 10n = 50

Power = 0.2

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Comments on the framework

• The use of odds instead of probabilities makes the articlehard to follow

• Odds

• Max 1 on the plots i.e. p ≤ 0.5• Plausible values?

• It would be great if the framework could be formallyassessed for various scientific fields!

• Typical values for p and u in Veterinary Epidemiology???• Is it possible to design a study to estimate these???• Problem: Gold Standard

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Comments on the framework

• The use of odds instead of probabilities makes the articlehard to follow

• Odds

• Max 1 on the plots i.e. p ≤ 0.5• Plausible values?

• It would be great if the framework could be formallyassessed for various scientific fields!

• Typical values for p and u in Veterinary Epidemiology???• Is it possible to design a study to estimate these???• Problem: Gold Standard

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Comments on the framework

• The use of odds instead of probabilities makes the articlehard to follow

• Odds• Max 1 on the plots i.e. p ≤ 0.5

• Plausible values?

• It would be great if the framework could be formallyassessed for various scientific fields!

• Typical values for p and u in Veterinary Epidemiology???• Is it possible to design a study to estimate these???• Problem: Gold Standard

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Comments on the framework

• The use of odds instead of probabilities makes the articlehard to follow

• Odds• Max 1 on the plots i.e. p ≤ 0.5• Plausible values?

• It would be great if the framework could be formallyassessed for various scientific fields!

• Typical values for p and u in Veterinary Epidemiology???• Is it possible to design a study to estimate these???• Problem: Gold Standard

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Comments on the framework

• The use of odds instead of probabilities makes the articlehard to follow

• Odds• Max 1 on the plots i.e. p ≤ 0.5• Plausible values?

• It would be great if the framework could be formallyassessed for various scientific fields!

• Typical values for p and u in Veterinary Epidemiology???• Is it possible to design a study to estimate these???• Problem: Gold Standard

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Comments on the framework

• The use of odds instead of probabilities makes the articlehard to follow

• Odds• Max 1 on the plots i.e. p ≤ 0.5• Plausible values?

• It would be great if the framework could be formallyassessed for various scientific fields!

• Typical values for p and u in Veterinary Epidemiology???

• Is it possible to design a study to estimate these???• Problem: Gold Standard

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Comments on the framework

• The use of odds instead of probabilities makes the articlehard to follow

• Odds• Max 1 on the plots i.e. p ≤ 0.5• Plausible values?

• It would be great if the framework could be formallyassessed for various scientific fields!

• Typical values for p and u in Veterinary Epidemiology???• Is it possible to design a study to estimate these???

• Problem: Gold Standard

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Comments on the framework

• The use of odds instead of probabilities makes the articlehard to follow

• Odds• Max 1 on the plots i.e. p ≤ 0.5• Plausible values?

• It would be great if the framework could be formallyassessed for various scientific fields!

• Typical values for p and u in Veterinary Epidemiology???• Is it possible to design a study to estimate these???• Problem: Gold Standard

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Comments on the framework

• Link between magnitude of the effect, α, β and samplesize

• Trade off between α and β• Smaller effects require bigger samples

Some Parameter

Fre

quen

cy

Some Parameter

Fre

quen

cy

Some Parameter

Fre

quen

cy

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Comments on the framework

• Link between magnitude of the effect, α, β and samplesize

• Trade off between α and β

• Smaller effects require bigger samples

Some Parameter

Fre

quen

cy

Some Parameter

Fre

quen

cy

Some Parameter

Fre

quen

cy

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Comments on the framework

• Link between magnitude of the effect, α, β and samplesize

• Trade off between α and β• Smaller effects require bigger samples

Some Parameter

Fre

quen

cy

Some Parameter

Fre

quen

cy

Some Parameter

Fre

quen

cy

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Comments on the framework

• The corollaries follow from the proposed model

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Plan

1 Context

2 Introduction

3 Modelling FrameworkHypothesis testingBiasMultiple testingComments

4 Corollaries

5 Conclusion

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Corollary 1

The smaller the studies conducted in a scientific field, theless likely the research findings are to be true

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Corollary 2

The smaller the effect sizes in a scientific field, the lesslikely the research findings are to be true

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Corollary 3

The greater the number and the lesser the selection oftested relationships in a scientific field, the less likely theresearch findings are to be true

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Corollary 4

The greater the flexibility in designs, definitions, outcomesand analytical modes in a scientific field, the less likely theresearch findings are to be true

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Corollary 5

The greater the financial and other interests and prejudicesin a scientific field, the less likely the research findings areto be true

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Corollary 6

The hotter a scientific field (with more scientific teamsinvolved), the less likely the research findings are to be true

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Plan

1 Context

2 Introduction

3 Modelling FrameworkHypothesis testingBiasMultiple testingComments

4 Corollaries

5 Conclusion

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

How can we improve the situation?

• Cannot draw firm conclusions based on a single positiveresult

• It is possible to test for something until we find what wewant!

• And this is more likely to receive attention

• Selecting research questions• Avoid marketing driven questions• Importance of pre study odds

• Increase power• Larger samples

• For research questions with high pre-study odds• To test major concepts rather than narrow specific

questions

• Research standards