improving the internal validity of experiments in focal ischaemia malcolm macleod centre for...
TRANSCRIPT
![Page 1: Improving the internal validity of experiments in focal ischaemia Malcolm Macleod Centre for Clinical Brain Sciences, University of Edinburgh](https://reader035.vdocuments.us/reader035/viewer/2022062417/5515127a550346c77d8b4b81/html5/thumbnails/1.jpg)
Improving the internal validity of
experiments in focal ischaemia
Malcolm MacleodCentre for Clinical Brain Sciences, University of Edinburgh
![Page 2: Improving the internal validity of experiments in focal ischaemia Malcolm Macleod Centre for Clinical Brain Sciences, University of Edinburgh](https://reader035.vdocuments.us/reader035/viewer/2022062417/5515127a550346c77d8b4b81/html5/thumbnails/2.jpg)
1026
1026 interventions in experimental stroke
O’C
olli
ns
et
al A
nn N
euro
l 2006
![Page 3: Improving the internal validity of experiments in focal ischaemia Malcolm Macleod Centre for Clinical Brain Sciences, University of Edinburgh](https://reader035.vdocuments.us/reader035/viewer/2022062417/5515127a550346c77d8b4b81/html5/thumbnails/3.jpg)
1026603
1026 interventions in experimental stroke
Tested in focal ischaemia
O’C
olli
ns
et
al A
nn N
euro
l 2006
![Page 4: Improving the internal validity of experiments in focal ischaemia Malcolm Macleod Centre for Clinical Brain Sciences, University of Edinburgh](https://reader035.vdocuments.us/reader035/viewer/2022062417/5515127a550346c77d8b4b81/html5/thumbnails/4.jpg)
1026883374
1026 interventions in experimental stroke
Effective in focal ischaemia
O’C
olli
ns
et
al A
nn N
euro
l 2006
![Page 5: Improving the internal validity of experiments in focal ischaemia Malcolm Macleod Centre for Clinical Brain Sciences, University of Edinburgh](https://reader035.vdocuments.us/reader035/viewer/2022062417/5515127a550346c77d8b4b81/html5/thumbnails/5.jpg)
1026883550
97 18
1026 interventions in experimental stroke
Tested in clinical trial
O’C
olli
ns
et
al A
nn N
euro
l 2006
![Page 6: Improving the internal validity of experiments in focal ischaemia Malcolm Macleod Centre for Clinical Brain Sciences, University of Edinburgh](https://reader035.vdocuments.us/reader035/viewer/2022062417/5515127a550346c77d8b4b81/html5/thumbnails/6.jpg)
1026883550
97 171 3
1026 interventions in experimental stroke
Effective in clinical trial
O’C
olli
ns
et
al A
nn N
euro
l 2006
![Page 7: Improving the internal validity of experiments in focal ischaemia Malcolm Macleod Centre for Clinical Brain Sciences, University of Edinburgh](https://reader035.vdocuments.us/reader035/viewer/2022062417/5515127a550346c77d8b4b81/html5/thumbnails/7.jpg)
What’s my problem?
• I want to improve the outcome for my patients with stroke
• To get that, I want to conduct high quality clinical trials of interventions which have a reasonable chance of actually working in humans
• But which of the remaining 929 interventions should I choose?
![Page 8: Improving the internal validity of experiments in focal ischaemia Malcolm Macleod Centre for Clinical Brain Sciences, University of Edinburgh](https://reader035.vdocuments.us/reader035/viewer/2022062417/5515127a550346c77d8b4b81/html5/thumbnails/8.jpg)
It’s not just my problem …
![Page 9: Improving the internal validity of experiments in focal ischaemia Malcolm Macleod Centre for Clinical Brain Sciences, University of Edinburgh](https://reader035.vdocuments.us/reader035/viewer/2022062417/5515127a550346c77d8b4b81/html5/thumbnails/9.jpg)
“…you will meet with several observations and experiments which, though communicated for true
by candid authors or undistrusted eye-witnesses, or perhaps recommended by your own experience,
may, upon further trial, disappoint your expectation, either not at all succeeding, or at least varying much
from what you expected”
Robert Boyle (1693)Concerning the Unsuccessfulness of Experiments
![Page 10: Improving the internal validity of experiments in focal ischaemia Malcolm Macleod Centre for Clinical Brain Sciences, University of Edinburgh](https://reader035.vdocuments.us/reader035/viewer/2022062417/5515127a550346c77d8b4b81/html5/thumbnails/10.jpg)
• One which describes some biological truth in the system being studied
• Internal validity: the extent to which an experiment accurately describes what happened in that model system
• Can be inferred by extent of reporting of measures to avoid common biases
What is a Valid Experiment?
![Page 11: Improving the internal validity of experiments in focal ischaemia Malcolm Macleod Centre for Clinical Brain Sciences, University of Edinburgh](https://reader035.vdocuments.us/reader035/viewer/2022062417/5515127a550346c77d8b4b81/html5/thumbnails/11.jpg)
• One which considers all available supporting animal data
• One which considers the likelihood of publication bias
• One which tests a drug under circumstances similar to those in which efficacy has been demonstrated in animal models
What is a Valid Translational strategy?
![Page 12: Improving the internal validity of experiments in focal ischaemia Malcolm Macleod Centre for Clinical Brain Sciences, University of Edinburgh](https://reader035.vdocuments.us/reader035/viewer/2022062417/5515127a550346c77d8b4b81/html5/thumbnails/12.jpg)
Potential sources of bias in animal studies
• Internal validityProblem Solution
Selection Bias Randomisation
Performance Bias Allocation Concealment
Detection Bias Blinded outcome assessment
Attrition bias Reporting drop-outs/ ITT analysis
False positive report bias Adequate sample sizes
Aft
er
Cro
ssle
y e
t al, 2
008,
Wach
old
er,
2004
![Page 13: Improving the internal validity of experiments in focal ischaemia Malcolm Macleod Centre for Clinical Brain Sciences, University of Edinburgh](https://reader035.vdocuments.us/reader035/viewer/2022062417/5515127a550346c77d8b4b81/html5/thumbnails/13.jpg)
Internal validityDopamine Agonists in models
of PD
Ferg
uso
n e
t al, in d
raft
![Page 14: Improving the internal validity of experiments in focal ischaemia Malcolm Macleod Centre for Clinical Brain Sciences, University of Edinburgh](https://reader035.vdocuments.us/reader035/viewer/2022062417/5515127a550346c77d8b4b81/html5/thumbnails/14.jpg)
Internal validityDopamine Agonists in models
of PD
Ferg
uso
n e
t al, in d
raft
![Page 15: Improving the internal validity of experiments in focal ischaemia Malcolm Macleod Centre for Clinical Brain Sciences, University of Edinburgh](https://reader035.vdocuments.us/reader035/viewer/2022062417/5515127a550346c77d8b4b81/html5/thumbnails/15.jpg)
Internal validityDopamine Agonists in models
of PD
Ferg
uso
n e
t al, in d
raft
![Page 16: Improving the internal validity of experiments in focal ischaemia Malcolm Macleod Centre for Clinical Brain Sciences, University of Edinburgh](https://reader035.vdocuments.us/reader035/viewer/2022062417/5515127a550346c77d8b4b81/html5/thumbnails/16.jpg)
Internal ValidityRandomisation and blinding in studies of
hypothermia in experimental stroke
van d
er
Worp
et
al B
rain
2007
Randomisation
Yes No
Blinded outcome
assessment
Yes NoEffi
cacy
47%39%47%37%
Effi
cacy
![Page 17: Improving the internal validity of experiments in focal ischaemia Malcolm Macleod Centre for Clinical Brain Sciences, University of Edinburgh](https://reader035.vdocuments.us/reader035/viewer/2022062417/5515127a550346c77d8b4b81/html5/thumbnails/17.jpg)
Stem cells in experimental stroke
Lees
et
al, in d
raft
![Page 18: Improving the internal validity of experiments in focal ischaemia Malcolm Macleod Centre for Clinical Brain Sciences, University of Edinburgh](https://reader035.vdocuments.us/reader035/viewer/2022062417/5515127a550346c77d8b4b81/html5/thumbnails/18.jpg)
Infarct Volume
Internal ValidityRandomisation, allocation concealment and
blinding in studies of Stem cells in experimental stroke
Neurobehavioural score
Lees
et
al, in d
raft
![Page 19: Improving the internal validity of experiments in focal ischaemia Malcolm Macleod Centre for Clinical Brain Sciences, University of Edinburgh](https://reader035.vdocuments.us/reader035/viewer/2022062417/5515127a550346c77d8b4b81/html5/thumbnails/19.jpg)
Internal Validity NXY-059
Macl
eod
et
al, 2
008
![Page 20: Improving the internal validity of experiments in focal ischaemia Malcolm Macleod Centre for Clinical Brain Sciences, University of Edinburgh](https://reader035.vdocuments.us/reader035/viewer/2022062417/5515127a550346c77d8b4b81/html5/thumbnails/20.jpg)
Internal ValidityAttrition bias
![Page 21: Improving the internal validity of experiments in focal ischaemia Malcolm Macleod Centre for Clinical Brain Sciences, University of Edinburgh](https://reader035.vdocuments.us/reader035/viewer/2022062417/5515127a550346c77d8b4b81/html5/thumbnails/21.jpg)
Internal ValidityFalse positive reporting bias
• The positive predictive value of any test result depends on– p (α)– Power (1-ß)– Pre-test probability of a positive result
aft
er
Wach
old
er,
2004
![Page 22: Improving the internal validity of experiments in focal ischaemia Malcolm Macleod Centre for Clinical Brain Sciences, University of Edinburgh](https://reader035.vdocuments.us/reader035/viewer/2022062417/5515127a550346c77d8b4b81/html5/thumbnails/22.jpg)
Internal ValidityFalse positive reporting bias
• The positive predictive value of any test result depends on– p (α) (0.05)– Power (1-ß) (0.30)– Pre-test probability of a positive result
(0.50)
Positive predictive value = 0.67i.e. only 2 out of 3 statistically positive studies
are truly positive
aft
er
Wach
old
er,
2004
![Page 23: Improving the internal validity of experiments in focal ischaemia Malcolm Macleod Centre for Clinical Brain Sciences, University of Edinburgh](https://reader035.vdocuments.us/reader035/viewer/2022062417/5515127a550346c77d8b4b81/html5/thumbnails/23.jpg)
Chances that data from any given animal will be non-
contributory
Number of animals Power % animals wasted
4 18.6% 81.4%
8 32.3% 67.7%
16 56.4% 43.6%
32 85.1% 14.9%
assume simple two group experiment seeking 30% reduction in infarct volume, observed SD
40% of control infarct volume
![Page 24: Improving the internal validity of experiments in focal ischaemia Malcolm Macleod Centre for Clinical Brain Sciences, University of Edinburgh](https://reader035.vdocuments.us/reader035/viewer/2022062417/5515127a550346c77d8b4b81/html5/thumbnails/24.jpg)
Chances of wasting an animal
Number of animals per group
0 10 20 30 40
% a
nim
als
was
ted
0
20
40
60
80
100
![Page 25: Improving the internal validity of experiments in focal ischaemia Malcolm Macleod Centre for Clinical Brain Sciences, University of Edinburgh](https://reader035.vdocuments.us/reader035/viewer/2022062417/5515127a550346c77d8b4b81/html5/thumbnails/25.jpg)
BetterWorse
Pre
cisi
on
• All outcomes– 29 publications– 109 experiments– 1596 animals– Improved outcome by 31% (27-35%)
External Validity Publication Bias for FK506
Macl
eod
et
al, JC
BFM
200
5
![Page 26: Improving the internal validity of experiments in focal ischaemia Malcolm Macleod Centre for Clinical Brain Sciences, University of Edinburgh](https://reader035.vdocuments.us/reader035/viewer/2022062417/5515127a550346c77d8b4b81/html5/thumbnails/26.jpg)
External Validity Hypertension in studies of NXY-059 in experimental
stroke
Macl
eod
et
al, S
troke
in
pre
ss
Infarct volume:– 9 publications– 29 experiments– 408 animals– 44% (35-53%) improvement
Hypertension:– 7% of animal studies– 77% of patients in the
(neutral) SAINT II study
![Page 27: Improving the internal validity of experiments in focal ischaemia Malcolm Macleod Centre for Clinical Brain Sciences, University of Edinburgh](https://reader035.vdocuments.us/reader035/viewer/2022062417/5515127a550346c77d8b4b81/html5/thumbnails/27.jpg)
External Validity Hypertension in studies of tPA in experimental
stroke
Pere
l et
al B
MJ 2007
Comorbidity
“Normal” BP
Effi
cacy
-2%25%
Infarct Volume:– 113 publications– 212 experiments– 3301 animals– Improved outcome by 24% (20-28)
Hypertension:– 9% of animal studies– Specifically exclusion criterion in (positive)
NINDS study
![Page 28: Improving the internal validity of experiments in focal ischaemia Malcolm Macleod Centre for Clinical Brain Sciences, University of Edinburgh](https://reader035.vdocuments.us/reader035/viewer/2022062417/5515127a550346c77d8b4b81/html5/thumbnails/28.jpg)
Quality of Translation tPA and tirilazad
• Both appear to work in animals
• tPA works in humans but tirilazad doesn’t
• Time to treatment: tPA:– Animals – median 90 minutes– Clinical trial – median 90 minutes
• Time to treatment: tirilazad– Animals – median 10 minutes– Clinical trial - >3 hrs for >75% of patients
Sena e
t al, S
troke
2007;
Pere
l et
al B
MJ 200
7
![Page 29: Improving the internal validity of experiments in focal ischaemia Malcolm Macleod Centre for Clinical Brain Sciences, University of Edinburgh](https://reader035.vdocuments.us/reader035/viewer/2022062417/5515127a550346c77d8b4b81/html5/thumbnails/29.jpg)
Chose your patients – tPA: Effect of time to treatment on
efficacy
Pere
l et
al B
MJ 2007
; La
nce
t 2
004
![Page 30: Improving the internal validity of experiments in focal ischaemia Malcolm Macleod Centre for Clinical Brain Sciences, University of Edinburgh](https://reader035.vdocuments.us/reader035/viewer/2022062417/5515127a550346c77d8b4b81/html5/thumbnails/30.jpg)
Publication bias
Randomisation Co-morbidity
bias
Reported efficacy
How much efficacy is left?
26%32% 20% 5%
![Page 31: Improving the internal validity of experiments in focal ischaemia Malcolm Macleod Centre for Clinical Brain Sciences, University of Edinburgh](https://reader035.vdocuments.us/reader035/viewer/2022062417/5515127a550346c77d8b4b81/html5/thumbnails/31.jpg)
AnimalStudies
Systematic Review
AndMeta-analysis
• how powerful is the treatment?
• what is the quality of evidence?
• what is the range of evidence?
• is there evidence of a publication bias?
• What are the conditions of maximum efficacy?
Clinical Trial
Summarising data from animal experiments
![Page 32: Improving the internal validity of experiments in focal ischaemia Malcolm Macleod Centre for Clinical Brain Sciences, University of Edinburgh](https://reader035.vdocuments.us/reader035/viewer/2022062417/5515127a550346c77d8b4b81/html5/thumbnails/32.jpg)
Resources and acknowledgements
• www.camarades.info/index_files/papers.htm• www.camarades.info/index_files/talks.htm• www.camarades.info/index_files/resources.htm
• Chief Scientist Office, Scotland
• Emily Sena, Evie Ferguson, Jen Lees, Hanna Vesterinen
• David Howells, Bart van der Worp, Uli Dirnagl, Philip Bath