CAMARADES: Bringing evidence to translational medicine

Optimizing the Predictive Value of Pre-Clinical ResearchSession 3: Reviewer Perspective

Malcolm MacleodCollaborative Approach to Meta-Analysis and Review of Animal Data from Experimental Studies andCentre for Clinical Brain Sciences, University of Edinburgh

CAMARADES: Bringing evidence to translational medicine

Quality is important

Blinded assessment of behavioural outcome

No Yes

Impr

ovem

ent

in b

ehav

iour

al o

utco

me

(S

tand

ardi

sed

Eff

ect

Siz

e)

0.0

0.2

0.4

0.6

0.8

1.0

1.2EAE

PD

AD

CAMARADES: Bringing evidence to translational medicine

Prevalence of reporting of some measures to improve validity

RandomisationBlinded Outcome

AssessmentSample Size calculation

Stroke 36% 29% 3%

MND 31% 20% <1%

AD 15% 25% 0%

PD 12% 15% 0%

EAE 8% 15% <1%

Glioma 14% 0% 0%

Pain 14% 25% 0%

CAMARADES: Bringing evidence to translational medicine

Quality by Journal

• 4584 full publications curated on CAMARADES

• Reporting the efficacy of an intervention in an animal disease model

• Journals contributing more than 100 publications

– Brain Research– Experimental Neurology– JCBFM*– Journal of Immunology– Journal of Neuroimmunology– Journal of Neuroscience– Neuroscience– PNAS– Stroke

JOURNAL

A B C D E F G H I

NU

MB

ER

OF

PU

BL

ICA

TIO

NS

0

50

100

150

200

250

CAMARADES: Bringing evidence to translational medicine

Quality by Journal

SAMPLE SIZE CALCULATION

JOURNAL

A B C D E F G H I

PR

OP

OR

TIO

N

0.0

0.2

0.4

0.6

0.8

1.0 COMPLIANCE WITH REGULATIONS

JOURNAL

A B C D E F G H I

PR

OP

OR

TIO

N

0.0

0.2

0.4

0.6

0.8

1.0

CAMARADES: Bringing evidence to translational medicine

Does high Impact Factor reflect high quality research?

• 563 publications in focal cerebral ischaemia• Weak association between Impact Factor and

quality (adjusted r2 = 0.06)• Weaker association between number of citations

received by that publication and quality (adjusted r2 = 0.004)

CAMARADES: Bringing evidence to translational medicine

Change over time

• EAE: some improvement over time: 26 years per point increment in quality

• AD: some improvement over time: 24 years per point improvement in study quality

Study quality of in vivo studies selected from random sample of 1000

publications from PubMed

RandomisationBlinded outcome assessmentBlinded conduct of experimentConcealment of allocation sequenceSample size calculationConflict of Interest statement

CAMARADES: Bringing evidence to translational medicine

CAMARADES: Bringing evidence to translational medicine

Publication bias

n

expts

Estimated

unpublished

Reported efficacy

Corrected efficacy

Stroke – infarct volume 1359 214 31.3% 23.8%

EAE - neurobehaviour 1892 505 33.1% 15.0%

EAE – inflammation 818 14 38.2% 37.5%

EAE – demyelination 290 74 45.1% 30.5%

EAE – axon loss 170 46 54.8% 41.7%

AD – Water Maze 80 15 0.688 sd 0.498 sd

AD – plaque burden 632 154 0.999 sd 0.610 sd

- 32%20%

CAMARADES: Bringing evidence to translational medicine

Dimensions of assessment

• Internal validity• Efficacy• Generalisability

1 2 3 4

1 3 3 2

2 2 2 2 1

3 1 2 3 2

4 1 1 1 4

1 2 3 4

1 3

2 3 2

3 1 4 4 2

4 6 4 1

GeneralisabilityGeneralisability

Validity

Validity

Efficacy

Efficacy

1 2 3 4

1 1 1 1

2 2 2 1

3 3 2 3 3

4 5 2 2 2

Exemplar heat map of 30 experiments testing an intervention in EAE

CAMARADES: Bringing evidence to translational medicine

Evidence based translational medicine

Experimental Studies

Systematic review and meta-analysis • how powerful is the treatment?• what is the quality of evidence?• what is the range of evidence?• is there evidence of a publication bias?• What are the conditions of maximum efficacy?

Multi Centre Animal Studies• confirm efficacy• robust and monitored conduct of

experiments• transparent analysis and reporting• deliberate heterogeneity

Clinical trial