ioannidis 2005

Why mostpublishedresearch

findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Why most published research findings are falseArticle by John P. A. Ioannidis (2005)

Aurelien Madouasse

November 4, 2011


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Plan

1 Context

2 Introduction

3 Modelling FrameworkHypothesis testingBiasMultiple testingComments

4 Corollaries

5 Conclusion


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Context

• The author: John P.A. Ioannidis

• C.F. Rehnborg Chair in Disease Prevention at StanfordUniversity (US)

• Professor of Medicine and Director of the StanfordPrevention Research Center (US)

• Chaired the Department of Hygiene and Epidemiology atthe University of Ioannina School of Medicine (Greece)

• Has a 51 page CV


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Context

• The author: John P.A. Ioannidis• C.F. Rehnborg Chair in Disease Prevention at Stanford

University (US)• Professor of Medicine and Director of the Stanford

Prevention Research Center (US)• Chaired the Department of Hygiene and Epidemiology at

the University of Ioannina School of Medicine (Greece)• Has a 51 page CV


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Context

• The journal: PLoS Medicine

• Public Library of Science• Peer reviewed• Open Access• Publication fee: US$2900


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Context

• The journal: PLoS Medicine• Public Library of Science• Peer reviewed• Open Access• Publication fee: US$2900


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Context

• The article (Checked 2011-10-22)

• Views: 410,087• Citations:

• CrossRef: 312• PubMed Central: 118• Scopus: 579• Web of Science: 585


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Context

• The article (Checked 2011-10-22)• Views: 410,087• Citations:

• CrossRef: 312• PubMed Central: 118• Scopus: 579• Web of Science: 585


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Plan

1 Context

2 Introduction


4 Corollaries

5 Conclusion


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Introduction

• Published research findings sometimes refuted bysubsequent evidence

• Increasing concern false findings may be the majority

• This should no be surprising

• Here is why . . .


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Plan

1 Context

2 Introduction


4 Corollaries

5 Conclusion


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Hypothesis testing

• Consider a parameter measured in a population ofindividuals with a disease:

• Before treatment

• After treatment (Here assuming the treatment has an effect)

Some Parameter

Fre

quen

cy


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Hypothesis testing

• Consider a parameter measured in a population ofindividuals with a disease:

• Before treatment• After treatment (Here assuming the treatment has an effect)

Some Parameter

Fre

quen

cy


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Hypothesis Testing

• We want to know whether the treatment has an effect

• We make a hypothesis

• H0: The treatment has no effect

• We test our hypothesis

• We get a result

• If H0 were true, the probability of observing our datawould be . . .

• p(data|H0) = p − value

• We draw a conclusion

• If p(data|H0) > 0.05 we accept H0 → No effect• If p(data|H0) ≤ 0.05 we reject H0 → Effect


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Hypothesis Testing


• We make a hypothesis• H0: The treatment has no effect


• We get a result

• If H0 were true, the probability of observing our datawould be . . .





findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Hypothesis Testing




• We get a result• If H0 were true, the probability of observing our data

would be . . .





findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Hypothesis Testing





would be . . .• p(data|H0) = p − value




findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Hypothesis Testing






• We draw a conclusion• If p(data|H0) > 0.05 we accept H0 → No effect

• If p(data|H0) ≤ 0.05 we reject H0 → Effect


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Hypothesis Testing






• We draw a conclusion• If p(data|H0) > 0.05 we accept H0 → No effect• If p(data|H0) ≤ 0.05 we reject H0 → Effect


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Hypothesis Testing

• This framework assumes that we accept to be wrong . . .

sometimes

TruthTrue relationship No relationship

Trial

Relationship 1 − β αNo relationship β 1 − α

• α = probability of declaring a relationship when there isnone - Type I error

• β = probability of finding no relationship when there isone - Type II error

• 1− β = probability of finding a relationship when there isone - Power


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Hypothesis Testing


Trial


• For a given hypothesis, whether we get it wrong dependson:

• Whether the hypothesis is true• The magnitude of the effect• The values we choose for α and β

Some Parameter

Fre

quen

cy

Some Parameter

Fre

quen

cy

Some Parameter

Fre

quen

cy


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Modelling the Framework for FalsePositive Findings


Trial


Total p 1 − p

• Central point of the paper

• Consider a population of possible hypotheses• Among these hypotheses, a proportion p are True• Hypothesis testing can be seen as testing for a disease in

Epidemiology• 1− β is the sensitivity• 1− α is the specificity• We can define a positive predictive value


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion



Trial


Total p 1 − p

• Central point of the paper• Consider a population of possible hypotheses

• Among these hypotheses, a proportion p are True• Hypothesis testing can be seen as testing for a disease in



findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion



Trial


Total p 1 − p

• Central point of the paper• Consider a population of possible hypotheses• Among these hypotheses, a proportion p are True

• Hypothesis testing can be seen as testing for a disease inEpidemiology

• 1− β is the sensitivity• 1− α is the specificity• We can define a positive predictive value


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion



Trial


Total p 1 − p

• Central point of the paper• Consider a population of possible hypotheses• Among these hypotheses, a proportion p are True• Hypothesis testing can be seen as testing for a disease in

Epidemiology

• 1− β is the sensitivity• 1− α is the specificity• We can define a positive predictive value


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion



Trial


Total p 1 − p


Epidemiology• 1− β is the sensitivity

• 1− α is the specificity• We can define a positive predictive value


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion



Trial


Total p 1 − p


Epidemiology• 1− β is the sensitivity• 1− α is the specificity

• We can define a positive predictive value


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion



Trial


Total p 1 − p




findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion



Trial


Total p 1 − p

• Positive predictive value

• Ioannidis uses R = p1−p


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion



Trial


Total p 1 − p


PPV =p(1− β)

p(1− β) + (1− p)α



findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion



Trial


Total p 1 − p



PPV =R

1+R × (1− β)R

1+R × (1− β) + 11+R × α


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion



Trial


Total p 1 − p



PPV =R(1− β)

R(1− β) + α


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Bias

• Among the studies that should have been reported asnegative

• A proportion u are reported as positive because of bias


Trial


Total p 1 − p


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Bias

• Among the studies that should have been reported asnegative

• A proportion u are reported as positive because of bias


Trial

Relationship 1 − β + uβ α + u(1 − α)No relationship (1 − u)β (1 − u)(1 − α)

Total p 1 − p


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Bias


Trial


Total p 1 − p




findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Bias


Trial


Total p 1 − p


PPV =p(1− β + uβ)

p(1− β + uβ) + (1− p)(α + u(1− α))



findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Bias


Trial


Total p 1 − p



PPV =R(1− β) + uβR

R + α− βR + u − uα + uβR


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Bias

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Pre−study odds

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Pre−study odds

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Pre−study odds

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Pre−study odds

Pos

t−st

udy

prob

abili

ty (

PP

V)

u = 0.05u = 0.2u = 0.5u = 0.8

0.0 0.2 0.4 0.6 0.8 1.00.

00.

20.

40.

60.

81.

0

Pre−study probability

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.00.

00.

20.

40.

60.

81.

0


Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.00.

00.

20.

40.

60.

81.

0


Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.00.

00.

20.

40.

60.

81.

0


Pos

t−st

udy

prob

abili

ty (

PP

V)

u = 0.05u = 0.2u = 0.5u = 0.8

Power = 0.8


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Bias

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Pre−study odds

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Pre−study odds

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Pre−study odds

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Pre−study odds

Pos

t−st

udy

prob

abili

ty (

PP

V)

u = 0.05u = 0.2u = 0.5u = 0.8

0.0 0.2 0.4 0.6 0.8 1.00.

00.

20.

40.

60.

81.

0


Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.00.

00.

20.

40.

60.

81.

0


Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.00.

00.

20.

40.

60.

81.

0


Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.00.

00.

20.

40.

60.

81.

0


Pos

t−st

udy

prob

abili

ty (

PP

V)

u = 0.05u = 0.2u = 0.5u = 0.8

Power = 0.5


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Bias

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Pre−study odds

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Pre−study odds

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Pre−study odds

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Pre−study odds

Pos

t−st

udy

prob

abili

ty (

PP

V)

u = 0.05u = 0.2u = 0.5u = 0.8

0.0 0.2 0.4 0.6 0.8 1.00.

00.

20.

40.

60.

81.

0


Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.00.

00.

20.

40.

60.

81.

0


Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.00.

00.

20.

40.

60.

81.

0


Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.00.

00.

20.

40.

60.

81.

0


Pos

t−st

udy

prob

abili

ty (

PP

V)

u = 0.05u = 0.2u = 0.5u = 0.8

Power = 0.2


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Testing by Several IndependentTeams

• Increases the probability of a positive finding . . . by chance

• Positive findings more likely to be published

• Association with publication bias?

• Positive findings more likely to receive attention

• Probability of at least one positive finding:

1 - probability of negative findings only


Trial

Relationship 1 − βn 1 − (1 − α)n

No relationship βn (1 − α)n


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion


• Increases the probability of a positive finding . . . by chance

• Positive findings more likely to be published• Association with publication bias?

• Positive findings more likely to receive attention

• Probability of at least one positive finding:

1 - probability of negative findings only


Trial




findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion



Trial



Total p 1 − p




findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion



Trial



Total p 1 − p


PPV =p(1− βn)

p(1− βn) + (1− p)(1− (1− α)n)



findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion



Trial



Total p 1 − p



PPV =R(1− βn)

R + 1− ((1− α)n + Rβn)


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion


0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Pre−study odds

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Pre−study odds

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Pre−study odds

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Pre−study odds

Pos

t−st

udy

prob

abili

ty (

PP

V)

n = 1n = 5n = 10n = 50

0.0 0.2 0.4 0.6 0.8 1.00.

00.

20.

40.

60.

81.

0


Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.00.

00.

20.

40.

60.

81.

0


Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.00.

00.

20.

40.

60.

81.

0


Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.00.

00.

20.

40.

60.

81.

0


Pos

t−st

udy

prob

abili

ty (

PP

V)

n = 1n = 5n = 10n = 50

Power = 0.8


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion


0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Pre−study odds

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Pre−study odds

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Pre−study odds

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Pre−study odds

Pos

t−st

udy

prob

abili

ty (

PP

V)

n = 1n = 5n = 10n = 50

0.0 0.2 0.4 0.6 0.8 1.00.

00.

20.

40.

60.

81.

0


Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.00.

00.

20.

40.

60.

81.

0


Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.00.

00.

20.

40.

60.

81.

0


Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.00.

00.

20.

40.

60.

81.

0


Pos

t−st

udy

prob

abili

ty (

PP

V)

n = 1n = 5n = 10n = 50

Power = 0.5


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion


0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Pre−study odds

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Pre−study odds

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Pre−study odds

Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Pre−study odds

Pos

t−st

udy

prob

abili

ty (

PP

V)

n = 1n = 5n = 10n = 50

0.0 0.2 0.4 0.6 0.8 1.00.

00.

20.

40.

60.

81.

0


Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.00.

00.

20.

40.

60.

81.

0


Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.00.

00.

20.

40.

60.

81.

0


Pos

t−st

udy

prob

abili

ty (

PP

V)

0.0 0.2 0.4 0.6 0.8 1.00.

00.

20.

40.

60.

81.

0


Pos

t−st

udy

prob

abili

ty (

PP

V)

n = 1n = 5n = 10n = 50

Power = 0.2


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Comments on the framework

• The use of odds instead of probabilities makes the articlehard to follow

• Odds

• Max 1 on the plots i.e. p ≤ 0.5• Plausible values?

• It would be great if the framework could be formallyassessed for various scientific fields!

• Typical values for p and u in Veterinary Epidemiology???• Is it possible to design a study to estimate these???• Problem: Gold Standard


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion



• Odds• Max 1 on the plots i.e. p ≤ 0.5

• Plausible values?




findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion



• Odds• Max 1 on the plots i.e. p ≤ 0.5• Plausible values?




findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion





• Typical values for p and u in Veterinary Epidemiology???

• Is it possible to design a study to estimate these???• Problem: Gold Standard


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion





• Typical values for p and u in Veterinary Epidemiology???• Is it possible to design a study to estimate these???

• Problem: Gold Standard


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion







findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion


• Link between magnitude of the effect, α, β and samplesize

• Trade off between α and β• Smaller effects require bigger samples

Some Parameter

Fre

quen

cy

Some Parameter

Fre

quen

cy

Some Parameter

Fre

quen

cy


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion



• Trade off between α and β

• Smaller effects require bigger samples

Some Parameter

Fre

quen

cy

Some Parameter

Fre

quen

cy

Some Parameter

Fre

quen

cy


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion



• Trade off between α and β• Smaller effects require bigger samples

Some Parameter

Fre

quen

cy

Some Parameter

Fre

quen

cy

Some Parameter

Fre

quen

cy


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion


• The corollaries follow from the proposed model


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Plan

1 Context

2 Introduction


4 Corollaries

5 Conclusion


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Corollary 1

The smaller the studies conducted in a scientific field, theless likely the research findings are to be true


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Corollary 2

The smaller the effect sizes in a scientific field, the lesslikely the research findings are to be true


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Corollary 3

The greater the number and the lesser the selection oftested relationships in a scientific field, the less likely theresearch findings are to be true


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Corollary 4

The greater the flexibility in designs, definitions, outcomesand analytical modes in a scientific field, the less likely theresearch findings are to be true


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Corollary 5

The greater the financial and other interests and prejudicesin a scientific field, the less likely the research findings areto be true


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Corollary 6

The hotter a scientific field (with more scientific teamsinvolved), the less likely the research findings are to be true


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

Plan

1 Context

2 Introduction


4 Corollaries

5 Conclusion


findings arefalse

AurelienMadouasse

Context

Introduction

ModellingFramework

Hypothesistesting

Bias

Multiple testing

Comments

Corollaries

Conclusion

How can we improve the situation?

• Cannot draw firm conclusions based on a single positiveresult

• It is possible to test for something until we find what wewant!

• And this is more likely to receive attention

• Selecting research questions• Avoid marketing driven questions• Importance of pre study odds

• Increase power• Larger samples

• For research questions with high pre-study odds• To test major concepts rather than narrow specific

questions

• Research standards

ioannidis 2005

Technology