research integrity: animal research - cebm.net · translational stroke research 7(5): 395-406....

CAMARADES: Bringing evidence to translational medicine

Research Integrity: animal research

Emily S Sena, PhDCentre for Clinical Brain Sciences, University of Edinburgh

@drEmilySena@camarades_


Disclosures

• BMJ Open Science (Editor-in-Chief)

– I receive an honorarium for this role

• I have applied and received (& will continue to) grant funding for this research


Hypotheses

• In the life sciences there are perverse incentives (publication, funding, promotion) to produce positive results with little attention paid to their validity

• In the use of animal disease models, pressure to reduce the number of animals (cost, time, ethics, feasibility) results in studies either being underpowered or of unknown power

• These factors combine to compromise the utility of animal models and contribute to translational failure


What we have seen

• Many examples of translational failure

• Replication difficulties

• Bayer: 53 of 67 findings did not replicate

• Amgen: 47 of 53 findings did not replicate

• Cancer Biology:• 3 of 5 findings did not replicate

What are the causes?

• ? Fraud

• ? False positive studies +/- dubious research practices

• ? Meta- (sectoral) problems like perverse incentives and publication bias

• ? True biological heterogeneity of observed effects


Reproducibility crisis


Bias is prevalent and important

RandomisationBlinded Outcome

Assessment

Stroke 36% 29%

MND 31% 20%

AD 15% 25%

PD 12% 15%

EAE 8% 15%

Glioma 14% 0%

Sena et al TiNS 2007

Efficacy

Randomisation


The ‘best’ institutions – RAE 1,173


• Overall efficacy was reduced from; – 32% (95% CI 30 to 34%) to 26% (95% CI 24 to 28%)

• 16% of experiments remain unpublished


Things are improving

McCann SK, Cramond F, Macleod MR, Sena ES (2016).

Translational stroke research 7(5): 395-406.

“The 2009 systematic review highlighted areas of weakness with respect

the lack of reporting on certain aspects of experimental design. While we

did not necessarily agree with all recommendations and also felt that not-reported

did not mean not done we did take on board that future studies did need to

more fully report details of experimental design. This change is reflected in

the positive outcome of the follow-up 2016 systematic review”

--- Professor Stuart Allan, University of Manchester


Research improvement activity

• Things done by stakeholders to increase the usefulness of research with which they are associated– The effectiveness must be assessed (RCTs, Observational)


What institutions can do….

• Be forward thinking/flexible

– Develop an active strategy to improve quality and value

– Unique dependent on local ambition and circumstance

• Learn from the healthcare improvement community

• Strategy may include

– Benchmarking/audit

– Prioritisation

– Tool development to measure performance

• Collaboration between meta-researchers & professional services


Potential strategy

• Non-punitive approach

– Work on everyone to make them a bit better

• ? Fraud

• ? False positive studies +/- dubious research practices

• ? Meta- (sectoral) problems like perverse incentives and publication bias

• ? True biological heterogeneity of observed effects


Research Improvement Activity

What performance do we

aspire to?

“95% of UoE manuscripts describing animal

research report randomisation where this would

be appropriate”

What is our current

performance?

Measure, [2009-10 = 8%]

What are we going to do

about it?

• Education sessions for PhD/post Doc/ PIs

• Highlighted component of AWERB review

• Identified factor in resource allocation (open

access publication funds, prioritisation of

research resources)

Did that make a difference? Measure

Is performance now good

enough?

Stick or twist


Define target performance

Measure performance

Seek to improve

performance

Measure performance

Did we succeed?

Consolidate into standard

practice


Take home…..

• Academic in vivo studies which do not report simple measures to avoid bias give larger estimates of treatment effects

• Most do not report simple measures to reduce bias

– Although improvements are happening

• Reporting/publication biases are important and prevalent

• You cannot pass responsibility to journals/funders alone

• Help is at hand but improvement strategies must be tested

• Meta-research approaches (& automation tools) can facilitate this work at scale

• Institutional/sectoral buy-in is required to effectively improve the integrity of our research

– I would advocate for a non-punitive approach


Thanks to...........

research integrity: animal research - cebm.net · translational stroke research 7(5): 395-406....

Documents